--blame-hang does not create dumps

[rkieslinger]

Description

On our Azure DevOps Server we have a pipeline, which executes some automated tests. Currently, there's a test, which always hangs. Here's our dotnet call:

C:\AzAgent\_work\_tool\dotnet\dotnet.exe test --logger trx --results-directory C:\AzAgent\_work\_temp C:\AzAgent\_work\42\b\Assembly1.Test.dll C:\AzAgent\_work\42\b\Assembly2.Test.dll C:\AzAgent\_work\42\b\Assembly3.Test.dll --filter TestCategory!=PerformanceMeasurement --settings C:\AzAgent\_work\42\b\.runsettings --blame-crash --blame-crash-dump-type full --blame-hang --blame-hang-dump-type full --blame-hang-timeout 60m --verbosity Normal

Expected behavior

Because we are using the options --blame-hang --blame-hang-dump-type full --blame-hang-timeout 60m we would expect a dump to be created in our TestResults directory. Unfortunately, there's just a Sequence.xml, but no dump file.

I found the following lines in the log:

The active test run was aborted. Reason: Test host process crashed
Data collector 'Blame' message: The specified inactivity time of 60 minutes has elapsed. Collecting hang dumps from testhost and its child processes.

While reading another issue posted here I noticed, they had another line in their log, which I couldn't find in my log:

Data collector 'Blame' message: Dumping 13360 - dotnet.

Here's our .runsettings:

<RunSettings>
  <RunConfiguration>
    <!-- Run tests parallel using all cores on the machine -->
    <MaxCpuCount>0</MaxCpuCount>
  </RunConfiguration>
  <!-- MSTest adapter -->
  <MSTest>
    <MapInconclusiveToFailed>True</MapInconclusiveToFailed>
    <MapNotRunnableToFailed>True</MapNotRunnableToFailed>
    <CaptureTraceOutput>True</CaptureTraceOutput>
    <DeleteDeploymentDirectoryAfterTestRunIsComplete>False</DeleteDeploymentDirectoryAfterTestRunIsComplete>
    <DeploymentEnabled>False</DeploymentEnabled>
  </MSTest>
  <TestRunParameters>
    <Parameter name="UseExtendedEFCoreLogging" value="true" />
    <Parameter name="DbExecutionStrategyAutoRetryCount" value="5" />
    <Parameter name="DbExecutionStrategySupportConnectionClosingExceptions" value="true" />
    <Parameter name="ServerName" value="v-sql2019d-ut" />
    <Parameter name="BuildDefinitionName" value="Current.Release" />
    <Parameter name="BuildNumber" value="162063" />
  </TestRunParameters>
</RunSettings>

Environment

• Windows Server 2022 Standard
• .NET 7.0
• Environment variables:
  • PROCDUMP_PATH: C:\ProcDump
  • VSTEST_DUMP_FORCEPROCDUMP: 1

[rkieslinger]

On another run, I got the following logs:

2023-03-29T12:52:57.5780801Z Blame: Attaching crash dump utility to process testhost (6076).
...
...
...
2023-03-29T13:52:59.3933766Z The active test run was aborted. Reason: Test host process crashed
2023-03-29T13:52:59.3991560Z Data collector 'Blame' message: The specified inactivity time of 60 minutes has elapsed. Collecting hang dumps from testhost and its child processes.
2023-03-29T13:52:59.3992450Z Data collector 'Blame' message: Data collector caught an exception of type 'System.ArgumentException': 'Process with an Id of 6076 is not running.'. More details: Blame: Creating hang dump failed with error...
2023-03-29T13:52:59.3993007Z Data collector 'Blame' message: Data collector caught an exception of type 'System.IO.FileNotFoundException': 'Collect dump was enabled but no dump file was generated.'. More details: Blame: Collecting hang dump failed with error...

What I don't understand is, what that log actually means. Is the Test host process crashed, or does it hang? It seems like the process crashed, but the system kinda thinks, it hangs and wants to create a dump later on, but the process isn't there anymore (maybe because it crashed?)

[nohwnd]

Are you running it together with --blame-crash? There is an issue where hang kills procdump and that kills the testhost process as well. Trying to solve that in #4378

edit: Your log says that on the top. I overlooked it.

[nohwnd]

I can tell you if you post whole .datacollector. logs, if that is a possibility for you.

[rkieslinger]

Thank you very much for fixing it!

Can you please tell me, how this fix gets released? Is it included in a new Visual Studio version? Is there a timetable?

[nohwnd]

This will be in 17.6.1 and 17.7.0 (and there will be servicing for net7.0.3xx but not the one that will be coming soon, we missed that window.).

[egraff]

@nohwnd will the fix also be made available to the current LTSC version of Visual Studio? Alternatively, is it part of the .NET 6 SDK, and if so, which version? 🙂

[nohwnd]

We plan to service 17.6.x once it is possible. We don't plan to service net6, but you can use net7 sdk and build net6 without any problem. Same with running tests, you can install any version of dotnet sdk that is newer than net6 and use that to run tests, and that way get vstest fixes early.