smoke-tests: Minimal set of tests without the need of downloading

[mirespace]

Hi!

Describe the Problem

I would like to use the smoke test suite as part of the test build stage when packaging dotNet6 from source-build on Ubuntu (for the 22.04 series currently) but with a minimal set of tests that don't require downloading anything from the internet to pass, as our CI infra doesn't allow it.

I locally build the last tag version v6.0.104 and I can also use the tests:

VERBOSE=1 ./build.sh --run-smoke-test
Found bootstrap SDK 6.0.103, bootstrap Arcade 6.0.0-beta.22122.7, bootstrap SourceLink 1.1.0-beta-21480-02
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET

Copyright (C) Microsoft Corporation. All rights reserved.


/<<PKGBUILDDIR>>/.dotnet/sdk/6.0.103/MSBuild.dll /bl:/<<PKGBUILDDIR>>/artifacts/log/Debug/BuildTests_0418121549.binlog /clp:v=m /fileLoggerParameters:LogFile=/<<PKGBUILDDIR>>/artifacts/logs/BuildTests_0418121549.log /flp:v=detailed /t:RunSmokeTest /<<PKGBUILDDIR>>/build.proj
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.DotNet.Arcade.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
[SourceBuiltSdkResolver] Overriding Microsoft.DotNet.Arcade.Sdk with 'ARCADE'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  
  Welcome to .NET 6.0!
  ---------------------
  SDK Version: 6.0.103
  
  ----------------
  Installed an ASP.NET Core HTTPS development certificate.
  To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
  Learn about HTTPS: https://aka.ms/dotnet-https
  ----------------
  Write your first app: https://aka.ms/dotnet-hello-world
  Find out what's new: https://aka.ms/dotnet-whats-new
  Explore documentation: https://aka.ms/dotnet-docs
  Report issues and find source on GitHub: https://github.com/dotnet/core
  Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
  --------------------------------------------------------------------------------------
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
    Determining projects to restore...
    Restored /<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj (in 6.86 sec).
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
    Microsoft.DotNet.SourceBuild.SmokeTests -> /<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll
  Test run for /<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (.NETCoreApp,Version=v6.0)
  Microsoft (R) Test Execution Command Line Tool Version 17.0.0
  Copyright (c) Microsoft Corporation.  All rights reserved.
  
  Starting test execution, please wait...
  A total of 1 test files matched the specified pattern.
  [xUnit.net 00:00:00.40]     Microsoft.DotNet.SourceBuild.SmokeTests.SdkContentTests.CompareMsftToSb [SKIP]
    Skipped Microsoft.DotNet.SourceBuild.SmokeTests.SdkContentTests.CompareMsftToSb [1 ms]
  Results File: /<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/TestResults/_Camelot-TITAN_2022-04-18_12_16_00.trx
  
  Passed!  - Failed:     0, Passed:    41, Skipped:     1, Total:    42, Duration: 2 m 48 s - /<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (net6.0)

but unfortunately, our CI does not allow the use of external downloads and it breaks:

 VERBOSE=1 ./build.sh --run-smoke-test
Found bootstrap SDK 6.0.103, bootstrap Arcade 6.0.0-beta.22122.7, bootstrap SourceLink 1.1.0-beta-21480-02
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET

Copyright (C) Microsoft Corporation. All rights reserved.


/<<PKGBUILDDIR>>/.dotnet/sdk/6.0.103/MSBuild.dll /bl:/<<PKGBUILDDIR>>/artifacts/log/Debug/BuildTests_0418220959.binlog /clp:v=m /fileLoggerParameters:LogFile=/<<PKGBUILDDIR>>/artifacts/logs/BuildTests_0418220959.log /flp:v=detailed /t:RunSmokeTest /<<PKGBUILDDIR>>/build.proj
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.DotNet.Arcade.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
[SourceBuiltSdkResolver] Overriding Microsoft.DotNet.Arcade.Sdk with 'ARCADE'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  
  Welcome to .NET 6.0!
  ---------------------
  SDK Version: 6.0.103
  
  ----------------
  Installed an ASP.NET Core HTTPS development certificate.
  To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
  Learn about HTTPS: https://aka.ms/dotnet-https
  ----------------
  Write your first app: https://aka.ms/dotnet-hello-world
  Find out what's new: https://aka.ms/dotnet-whats-new
  Explore documentation: https://aka.ms/dotnet-docs
  Report issues and find source on GitHub: https://github.com/dotnet/core
  Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
  --------------------------------------------------------------------------------------
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.22122.7' at '/<<PKGBUILDDIR>>/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.22122.7'
    Determining projects to restore...
/<<PKGBUILDDIR>>/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [/<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.sln] [/<<PKGBUILDDIR>>/build.proj]
/<<PKGBUILDDIR>>/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : Name or service not known (api.nuget.org:443) [/<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.sln] [/<<PKGBUILDDIR>>/build.proj]
/<<PKGBUILDDIR>>/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : Name or service not known [/<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.sln] [/<<PKGBUILDDIR>>/build.proj]
/<<PKGBUILDDIR>>/build.proj(98,5): error MSB3073: The command "/<<PKGBUILDDIR>>/.dotnet/dotnet test /<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/ --logger:trx -c Release" exited with code 1.
make[1]: *** [debian/rules:145: override_dh_auto_test] Error 1
make[1]: Leaving directory '/<<PKGBUILDDIR>>'
make: *** [debian/rules:30: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

In fact, with the old smoke-test.sh, using the --minimal --excludeOnlineTests --excludeOmniSharpTests options gave the idea of no need to download configurations or packages (in practice, this was not the case and internet access was needed to pass this minimal set of tests).

Describe the Solution

Ideally, the tarball should store everything needed to run at least that minimum test suite along with the smoke test suite.

Thanks in advance for considering this,

Miriam

[MichaelSimons]

There are a couple issues at play here:

1. We want the smoke tests to test the product just as the end users would be using the product. That is to say many of the test scenarios depend on packages from NuGet.org. We can filter out tests that depend on online artifacts but that would significantly reduce the test coverage. Using this reduced set of tests for sign-off doesn't feel right.
2. The smoke tests themselves are written using xunit. xunit is not part of source-build therefore the smoke-tests cannot be built offline. This is an illustration of the previous point

How are distro-maintainers testing other source-built dev platforms like .NET (e.g. OpenJDK, Python, etc)? Don't other dev platforms face similar issues? Are they able to test online or bring a set of test prereqs/prebuilts?

cc @dotnet/distro-maintainers - Please share your thoughts/experiences on this.

[asbjornu]

As can be seen in the dotnet Homebrew formula, the test we execute there is very naïve. We just build and execute a console application taking a few arguments and assert that the same arguments are written to stdout. It's much better than nothing, but not anywhere near sufficient to be sure that everything actually works.

This kind of naïve testing does seem to be the norm in Homebrew, though. You can see the openjdk formula having the same kind of "hello world" test. I suppose there's an expectancy that upstream to do the required testing before stable releases are published. I'm not sure whether executing a more extensive test suite would be something the Homebrew maintainers would want or not.

[ayakael]

On Alpine's side, we use the Redhat's dotnet-regular-tests testing suite. It requires building of dotnet-bunny (repo), and a few tests do phone in online, but I don't think it would require much to integrate dotnet-bunny in source-build with an offline version of dotnet-regular-tests.

[tmds]

Smoke tests should be small tests that verify things are not completely broken.

dotnet-regular-tests has many regression tests.
These tests are mostly for problems that happened in the source-build build, but not for the Microsoft build.
That source-build has more issues is because:
a. Microsoft doesn't use it for their own builds.
b. It only works for the latest previews/rc, so it doesn't see much end-user usage(/testing).

dotnet-regular-tests also has some rid related tests. We shouldn't need these, source-build should be able to check it can work well when faced with unknown rids (#297).

[MichaelSimons]

[Triage] @mirespace - per our discussions, my impression is that this is not a requirement and that you have a solution in place to test. Is that correct? Can this issue be closed?