NuGet.Frameworks dependency is causing weird test failures when trying to load MSBuild

[twsouthwick]

Description

I have a project (https://github.com/dotnet/upgrade-assistant) that loads MSBuild from an available source and lets that run how it will. Tests employing this fail when moving to .NET 6 because the test framework has already loaded an (old) version of the NuGet.Frameworks library.

System.IO.FileLoadException: Could not load file or assembly 'NuGet.Frameworks, Version=6.0.0.220, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Could not find or load a specific file. (0x80131621)

I've verified that before the tests start, NuGet.Frameworks is already loaded. This worked fine when it was a .NET 5 version of msbuild as the versions aligned. This can be fixed by adding the following line to the test project file:

<ItemGroup>
  <!-- The test framework brings in an old NuGet.Frameworks that ends up getting used rather than what MSBuild wants -->
  <PackageReference Include="NuGet.Frameworks" Version="6.0.0-rc.258" />
</ItemGroup>

I see a PR opened that is still in draft mode to address this (#2544). For anyone that is trying to test something that uses MSBuild, they'll probably hit this and end up spending too much time fiddling with things to figure out what is going on.

[twsouthwick]

It seems like #2544 could be replaced with a source generator to autogenerate the right values. Happy to take a stab at that if that sounds good to you.

Another option would be to move the usage of it to a separate AppDomain on framework and AssemblyLoadContext on core (this doesn't work if it needs to be functional on a .NET Standard build).

[twsouthwick]

I do think this highlights that the test framework should be very careful about what dependencies they are requiring. I don't know if you have a policy at this point, but any external dependency should be done in a separate appdomain/assemblyloadcontext (depending on runtime) so that it doesn't conflict with users' code that may have the same dependency

[nohwnd]

We depend on newtonsoft.json and nuget.frameworks. For .net framework tests we do have appdomains, but they often fail to unload, causing hangs in CI runs. For .NET AssemblyLoadContexts are not implemented, yet, unfortunately.

[JoeRobich]

The NuGet.Frameworks dependency in the TestPlatform.ObjectModel is a terrible experience for tooling that loads MSBuild.

In dotnet-format I must ensure that my test projects reference a matching version of NuGet.Frameworks as the global.json pinned SDK brings in otherwise I get version mismatch at runtime. This is because the Test SDK of course uses TestPlatform.ObjectModel which has the NuGet.Frameworks dependency.

In OmniSharp where we use TestPlatform for test discovery and running, we have a similar issue in our vnext branch which runs on .NET 6. If we are not version matched with the version of the SDK we are attempting to run O# on top of, then we get version mismatches at runtime. We will likely have to copy the code we require from the TestPlatform.ObjectModel into O# just so we can avoid the NuGet Reference.

In Roslyn we get issues against our MSBuildWorkspace because users are using it in their tests and don't realize that a NuGet.Frameworks dependency is being added that will conflict with the version of the SDK they are running on top of. (dotnet/roslyn#52954 (comment))

Does the benefit that this dependency provides outweigh the many sometimes hard to diagnose issues that developers must workaround?

[Evangelink]

See #2544

[pavelhorak]

see #4693 which replaces #2544

[nohwnd]

@JoeRobich if we drop this dependency in net9 (but not in net7 or net8) will it help?

[JoeRobich]

@JoeRobich if we drop this dependency in net9 (but not in net7 or net8) will it help?

@nohwnd It would. Even if we couldn't remove the NuGet dependency from our test projects right away. We could at least stop bumping the version with each new SDK.