v3 Roadmap

[bradwilson]

A quick note about this list... It represents a moment in time for planning for a final list of v3 features. Until items are checked off, there is no guarantee that the feature will make it into v3. We reserve the right to rearrange and modify this list at will as we work through the issues and decide whether we can and will implement them.

Architecture

• Move primary build system to Linux #2329
• Minimum framework versions for 3.0 #2330
• v3 test projects are programs instead of libraries #2331
• Support dynamic assemblies #2332
• Add Support for IAsyncDisposable to Dispose of Test Resources #2017
• Feature equivalence: runner switches and config files #2126
• Add test context object #621
• Attach DiagnosticMessageSink to TestContext #2427
• Replace Task with ValueTask #2547
• Reduce/eliminate first chance exceptions #1779
• Add support for dependency injection #1738
• Add outgoing metadata to TestContext #2457

Core Framework

• Support tuples in TheoryData #2333
• Add MatrixTheoryData to v3 #2100
• Add assembly-level support for BeforeAfterTestAttribute #2334
• Support for ValueTask #1770
• Skip messages are not output to the html result file #2004
• Incorrect constructor overload selected in DataAttribute #1711
• Add Timeout to TestState status #1777
• Add "core multiplier" versions of maximum thread count #2297
• Provide richer theory data API than IEnumerable<object?[]> #1758
• IAssemblyFixture should be part of xUnit #1947
• Theory MemberData to allow awaitable method #1698
• Is StringValues supported for InlineData? #2343
• ValueTuples as theory data #1948
• Add Explicit property to [Fact] and friends #2518
• [InlineData] should provide access to ITheoryDataRow enhancements #2545
• Enable controlled randomness #2371
• ANSI Escape code is being escaped. #2535
• Review to ensure we never double enumerate #2439
• Add ability to add warning messages to tests #2726
• Generic collection fixtures #2557
• Add support for CollectionAttribute<T> and CollectionDefinitionAttribute<T> #2629
• Make collection definitions better #2007
• Support ordering all test cases in a collection #1829
• Console / Trace / Debugger capture support #1730
• Create new super-type for FactAttribute that is the basis of extensibility #2335
• Feature Request - Test case parallelization when there is no class/collection fixture #1986
• Feature Request: Run Tests in Parallel with Shared Context #2055

Assertion Library

• Enable support for C# 8 nullable reference types #2336
• Consider providing Span<T> / Memory<T> based overloads of assertion methods #1966
• Add Assert.ThrowsAsync() overloads that accepts a ValueTask-returning function #1919
• Suggestion : Need a new overload of Assert.Equal(double,double,precision,MidpointRounding) #2074
• ignoreWhiteSpaceDifferences should include all Unicode white space characters #1931
• Assert.Equal for long arrays cuts of the actual difference #1924
• Assert.Equal for doubles with tolerance #1293
• Add float oriented Assert.Equal with precision #1984
• Add Assert.Equivalent #1604
• Add Assert.Multiple: Record Assertion Failures and Report At End #1920
• Improve output for Assert.Equal for collections of mis-matched sizes/underrun #2352
• Review to ensure we never double enumerate #2439
• Add Assert.IsNotAssignableFrom to match reports for xUnit2018 #2727
• Assert.Equal result unclear, with string with differing trailing spaces #2049
• Standardize all assert exception argument formatting #1932
• Add support for IAsyncEnumerable element assertion #2367
• Add extensibility point for custom argument formatting #1914
• Include full comparison values in Assert.Equal output (configurable?) #1805
• Assert.Equal crashes with stack overflow #2271

Runner Library

• Add runner support for v3 cross-process RPC #2337

Runners

• Runners which skip theory pre-enumeration by default should offer an override switch #1871
• Add default culture as a runner option #1934
• Allow multi-line skip messages #1832
• List all available traits / traits values by trait name #2084
• Include flowId and parseServiceMessagesInside in testStdOut message to TeamCity #2397
• Support TRX output file format #2188
• Set NO_COLOR environment variable when user adds -nocolor command line switch #2605
• Jenkins failes to cancel xunit.console run #1811
• Enable custom reporters #1874
• Test filtering expressions #1833

Extensibility

• Extensibility attributes which took name pairs of type/assembly can alternately take Type objects #2338

Bugs/Small Features

• Assert.Equals on two large arrays takes a long time #1907
• Dispose method on test class not being called if test is cancelled #2008
• Remove EventArgs constraint from Assert.Raises #2221
• TestDisplayName does not print enough digits to differentiate certain inputs #1822
• Confusing assertion message when comparing floats. #2016
• Extensions instances caching does not work, when TraitHelper.GetTraits is used #2232
• Skip attribute on [InlineData] is not respected in Visual Studio visualstudio.xunit#266
• The generic Theory method does not support decimal argument type. #2418
• Seal TheoryAttribute #2366
• Assert.Throws<ArgumentException>(paramName:null, Action) throws ArgumentNullException #2396
• Unicode escaping for method names is sometimes broken #2651
• being able to execute static test method inherited from base class #2704
• Add overload to Assert.Contains for ConcurrentDictionary #1857
• Misleading error message when class fixture constructor throws an exception #2342
• Use MemberData for methods with optional parameters #1985
• ClassData outputs do not get disposed if they implement IDisposable #2162
• Add README to NuGet packages #2357
• Generic type arguments are not encoded properly #1916
• XUnit "Timeout" parameter on Fact attribute does not work with synchronous test/code #2222
• Test logs get mixed across test cases #2377

Open Questions/Further Discussion Required

• Introduce Snapshot Testing Like Jest #1727
• Drop-in replacement for NUnit #1724
• Is the ordering in AssertEqualityComparer correct? #2612
• Feature: Runner option to treat assert failures as warnings #1845
• Two-phase parametrised tests #1719
• Should FactAttribute support SkipWhen/SkipUnless properties? #2339
• Theory-like attribute at class level? #2050
• Test fails comparing two identical list of tuples #1995
• AssemblyRunner with a dynamic assembly #1896
• Property Injection for fixture instead of ctor Injection #2134
• Dependency concerns between discoverer and executor #2163
• Should more equivalence interfaces be used in Assert.Equivalent? #2697

Miscellaneous

• All source uses tabs now instead of spaces (reasoning)
• Move VSTest plugin (xunit.runner.visualstudio) to its own project

Breaking Changes

• IAsyncLifetime now derives from IAsyncDisposable, which changes the return value for DisposeAsync from Task to ValueTask. For better symmetry, the return value for InitializeAsync is now also ValueTask.
• ITestFramework.GetExecutor now takes IReflectionAssemblyInfo instead of AssemblyName.
• TestInvoker.GetTaskFromResult was renamed to GetValueTaskFromResult and now returns a ValueTask.
• TestAssemblyConfiguration.PreEnumerateTheoriesOrDefault() has been removed, to force runners to choose what the default behavior they want is. The previous default (true) had performance problems, and was being awkwardly handled with runners which wanted a different default value.
• ITestFrameworkDiscoveryOptions.PreEnumerateTheoriesOrDefault() has changed the default to false for performance reasons. Runners which require/desire theory pre-enumeration must now explicitly request it.