Pass arguments to runner in dotnet new templates

[heaths]

Templates for dotnet new were added for #1028 but don't use any passed arguments for the runner. With the global tool deprecated, this world provide a convenient way of customizing frameworks, reports, etc. from a console.

So instead of:

public static void Main(string[] args)
{
    var summary = BenchmarkRunner.Run<$(BenchmarkName)>();
}

The templates could use:

public static void Main(string[] args) 
{
    BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
}

[adamsitnik]

@heaths that's a very good point. Would you like to send a PR? The code is available here:

https://github.com/dotnet/BenchmarkDotNet/blob/master/templates/templates/BenchmarkDotNet.BenchmarkProjectTemplate.CSharp/Program.cs
https://github.com/dotnet/BenchmarkDotNet/blob/master/templates/templates/BenchmarkDotNet.BenchmarkProjectTemplate.FSharp/Program.fs
https://github.com/dotnet/BenchmarkDotNet/blob/master/templates/templates/BenchmarkDotNet.BenchmarkProjectTemplate.VB/Program.vb

[heaths]

Happy to, but in testing this out, I found it would otherwise force passing something like --filter *Benchmarks* (or whatever non-default benchmark name was used) so I'm trying to find a suitable replacement, i.e. no additional input required so the UX remains consistent. Using BenchmarkSwitcher.FromTypes doesn't change this behavior, such that you're prompted for which benchmarks to run if not otherwise filtered.

Still working on it, but any suggestions?

[heaths]

Somewhat oddly, BenchmarkSwitcher.RunAll doesn't take args. This almost seems to imply that Run(args) was meant mainly to filter benchmarks, while seemingly all possible options of the old global tool can be passed (I didn't compare completely, but both are long lists of switches and have a lot of recognizable overlap).

[heaths]

Seems the main culprit is:

BenchmarkDotNet/src/BenchmarkDotNet/Running/BenchmarkSwitcher.cs

Line 108 in b4e2b69

var benchmarksToFilter = options.UserProvidedFilters || !askUserForInput

If types were provided explicitly like in the following example that could be used in the template instead, perhaps, by default, the user shouldn't be prompted. Given the seeming intent of passing an explicit list of benchmark types, would you agree?

BenchmarkSwitcher.FromTypes(new[] { typeof(Benchmarks) }).Run(args);

[heaths]

To clarify, if that was the intent, I can go ahead and fix that problem in the BenchmarkSwitcher as well. I just wanted to make sure that was indeed the intent.

[tmds]

You could add an overload to Run that allows to set askUserForInput, and set it to false in the template.

Otherwise you'll break anyone that relies on getting this prompt.