Microsoft.Data.SqlClient.SNI.x64.dll crashes when using multiple AppDomains in IIS Express and IIS Server #1418

MojoDK · 2021-12-02T11:22:25Z

Describe the bug

In my .NetFramework 4.8 web app, I "Start Wihout Debugging" and everything works fine, bu when I rebuild the project, I get the errror below. I have to "Start Without Debugging" again, to make the website work again (and then it crashes againg when rebuilding etc).

From event viewer (sorry it is in Danish):

Program.
Application Error
Event ID 1000

Navn på program med fejl: iisexpress.exe, version: 10.0.22377.1000, tidsstempel: 0x8db0c0a2
Navn på modul med fejl: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, tidsstempel: 0x61953095
Undtagelseskode: 0xc0000409
Forskydning med fejl 0x000000000001c80c
Proces-id 0xff14
Programmets starttidspunkt 0x01d7e767da588b02
Programsti: C:\Program Files\IIS Express\iisexpress.exe
Modulsti: C:\Users\mora\AppData\Local\Temp\Temporary ASP.NET Files\vs\1453ae1f\938c099f\assembly\dl3\7084a998\0061005c_e1dcd701\Microsoft.Data.SqlClient.SNI.x64.dll
Rapport-id: a6347776-fe45-45b4-a202-d971b9ac10a9
Fuldt navn på program med fejl:
Relativt program-id for program med fejl:

Exception message:
None 

Stack trace:
None

To reproduce

Include a complete code listing (or project/solution) that we can run to reproduce the issue.

Partial code listings, or multiple fragments of code, will slow down our response or cause us to push the issue back to you to provide code to reproduce the issue.

I can't reproduce with a new project.

Expected behavior

Not crashing iisexpress.exe and let me build and refresh the website.

Further technical details

Microsoft.Data.SqlClient version: 4.0.0.0
.NET target: Framework 4.8
SQL Server version: Sql Server 2019
Operating system: Windows 11 Pro x64

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

JRahnama · 2021-12-08T05:49:44Z

@MojoDK have you tried to manually copy paste the required dll and see if that works?

MojoDK · 2021-12-09T11:36:02Z

@JRahnama the "Microsoft.Data.SqlClient.SNI.x64.dll" is already in the bin folder - if I copy (overwrite) it with another "Microsoft.Data.SqlClient.SNI.x64.dll" file, the error is the same.

JRahnama · 2021-12-09T17:19:24Z

@MojoDK are you able to reproduce the issue with a simple newly created application? or can you provide a repro please?

leonmeijer · 2022-01-11T11:37:25Z

We're having the same issue. For us it happens in an application with multiple app domains. When the second app domain loads and begins doing a database function, the application crashes hard with no stack trace in Visual Studio. We have the same error codes in Event Viewer. Downgrading to v3 resolved it for now.

jaymonson · 2022-01-11T17:06:16Z

We are having the same issue. It is intermittent but at times can occur frequently. When it occurs we get the following crashstack:

Faulting application name: w3wp.exe, version: 10.0.17763.1, time stamp: 0xcfdb13d8
Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c
Faulting process id: 0x228c
Faulting application start time: 0x01d7f3989332762e
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\inetpub\wwwroot\C2SWeb\C2SWebAdministrator\bin\Microsoft.Data.SqlClient.SNI.x64.dll
Report Id: 3b4dba21-3703-43ee-9ad7-fdd3c0b15863
Faulting package full name:
Faulting package-relative application ID:

I have not yet experienced the crash while in Visual Studio with debugger attached. I have seen the
crash on a Server 2019 with SQL 2019 and also on a Server 2019 with SQL 2017.

JRahnama · 2022-01-11T17:39:20Z

@leonmeijer and @jaymonson are you on Windows 11 as well and the issue does not happen when downgrading to Microsoft.Data.SqlClient v3?

jaymonson · 2022-01-11T18:24:01Z

Hi, I have not tried it on Windows 11. I have seen it on 2 different Server 2019 systems. I have not yet tried the downgrade to v3 as that would not be an option for us as we try to stay with the latest/greatest/current versions as much as possible. But I can try the downgrade when I get a chance. Thanks, Jay From: Javad ***@***.***> Sent: Tuesday, January 11, 2022 11:40 AM To: dotnet/SqlClient ***@***.***> Cc: Jay Monson ***@***.***>; Mention ***@***.***> Subject: Re: [dotnet/SqlClient] Microsoft.Data.SqlClient.SNI.x64.dll crashes iisexpress (Issue #1418) [External email: Use caution! Do not open attachments or click on links from unknown senders or unexpected emails.] @leonmeijer<https://protect-us.mimecast.com/s/A8_qCADAYoIZZ0zyC8JsII?domain=github.com> and @jaymonson<https://protect-us.mimecast.com/s/TdOzCBBQYpuDDqOQCNwC37?domain=github.com> are you on Windows 11 as well and the issue does not happen when downgrading to Microsoft.Data.SqlClient v3? — Reply to this email directly, view it on GitHub<https://protect-us.mimecast.com/s/GxVlCzp9w5IGGDAlu4ZO0k?domain=github.com>, or unsubscribe<https://protect-us.mimecast.com/s/KrkNCDkPVrfoopk2cZ5AoD?domain=github.com>. Triage notifications on the go with GitHub Mobile for iOS<https://protect-us.mimecast.com/s/n8-nCERGVvullyNrcZYkBb?domain=apps.apple.com> or Android<https://protect-us.mimecast.com/s/GJy0CG6gVxsBBnQxu0Mf_g?domain=play.google.com>. You are receiving this because you were mentioned.Message ID: ***@***.***>

tdhintz · 2022-01-11T18:26:46Z

The @jaymonson issue occurs on a redirect. It didn't happen on the latest System.Data.SqlClient, but it does on Microsoft.Data.SqlClient. It feels like it is losing some sort of thread context.

leonmeijer · 2022-01-11T18:49:41Z

I have the issue with v4 on Server 2016, 2022, Windows 10 and 11. Tried with SQL Server 2016. Happens with and without Visual Studio. Does not happen with v3 Outlook voor Android<https://aka.ms/AAb9ysg> downloaden

…

________________________________ From: Hintz ***@***.***> Sent: Tuesday, January 11, 2022 7:26:58 PM To: dotnet/SqlClient ***@***.***> Cc: Leon Meijer ***@***.***>; Mention ***@***.***> Subject: Re: [dotnet/SqlClient] Microsoft.Data.SqlClient.SNI.x64.dll crashes iisexpress (Issue #1418) The @jaymonson<https://github.com/jaymonson> issue occurs on a redirect. It didn't happen on the latest System.Data.SqlClient, but it does on Microsoft.Data.SqlClient. It feels like it is losing some sort of thread context. — Reply to this email directly, view it on GitHub<#1418 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA6N3SY3BYFYWDBRJWOURYTUVRY7FANCNFSM5JG3OPHQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

tdhintz · 2022-01-11T18:57:35Z

For the record, our test environments are typically Encryption true, certificates trusted.

waf · 2022-01-14T08:28:19Z

Also seeing this, on Windows 10, .NET Framework with Encrypt=False. It's not just IISExpress; it's also crashing w3wp.exe (as @jaymonson's earlier log showed).

Happening for all members of our development team. When debugging from Visual Studio the first time after building it crashes; after crashing once it then starts OK on subsequent debug launches until the next build.

Event Log Entry

Faulting application name: w3wp.exe, version: 10.0.19041.1, time stamp: 0x58c67bf3
Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c
Faulting process id: 0x6f7c
Faulting application start time: 0x01d8091b05a21bbc
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Projects\web\lib\Microsoft.Data.SqlClient.SNI.x64.dll
Report Id: 6e33d06d-be0e-4496-b5dd-f82c5285eb01
Faulting package full name: 
Faulting package-relative application ID:

agclark27 · 2022-01-24T17:44:01Z

We're observing the same crashes of w3wp.exe with the same exception code and fault offset in Microsoft.Data.SqlClient.SNI.x64.dll. We checked all of our Splunk logs, and the issue wasn't present when we were on 3.x, but it immediately began appearing with 4.0.0 and remains present in 4.0.1.

JRahnama · 2022-01-24T17:49:38Z

@waf and @agclark27 is it happening with a fresh build application even? Can you make a minimal repro with the issue happening and send it to us?

agclark27 · 2022-01-24T21:02:01Z

Yes, it'll happen throughout the day, even with fresh builds, and we're resetting the IIS app pool overnight each day. There are no memory pressure issues on the servers at the time the crashes occur. I'll see if we can attach DebugDiag to these to get a stack trace when the process crashes.

JRahnama · 2022-01-24T21:46:52Z

Have you tested this issue?
This blog could be helpful as well.

leonmeijer · 2022-01-24T21:49:02Z

Have you tested this issue? This blog could be helpful as well.

Mind that I have this issue in a Console/Desktop application, not using wp3wp at all

leonmeijer · 2022-01-28T10:17:39Z

Since today I see the same issue occur on servers I manage, but then in the Microsoft Monitoring Agent.

Faulting application name: MonitoringHost.exe, version: 10.20.18053.0, time stamp: 0x5f8bacc8
Faulting module name: Microsoft.Data.SqlClient.SNI.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c
Faulting process id: 0x31e8
Faulting application start time: 0x01d80da91b030a5d
Faulting application path: C:\Program Files\Microsoft Monitoring Agent\Agent\MonitoringHost.exe
Faulting module path: C:\Program Files\Microsoft Monitoring Agent\Agent\Health Service State\Resources\1469\Microsoft.Data.SqlClient.SNI.dll
Report Id: a5d50e8d-7b52-46e3-b837-6723229c595f

JRahnama · 2022-01-28T22:07:44Z

@leonmeijer can you provide a stand alone repro please?

alphaleonis · 2022-02-23T13:45:34Z

@JRahnama I was able to create a repro of this problem, though not involving IISExpress it sounds similar to the scenario described by @leonmeijer, which is also the problem we we're seeing, and I suspect it has the same underlying cause.

Here is a small standalone repro that exhibits this problem when using AppDomains in .NET Framework 4.8:

Program.cs:

using System;
using Microsoft.Data.SqlClient;

namespace ConsoleApp1
{   
   public class Program
   {
      public const string ConnectionString = "Data Source=.;Initial Catalog=SinT.feature/sqlClient.ProtectionTest;Integrated Security=SSPI;Multiple Active Result Sets=True;Encrypt=False;Multi Subnet Failover=False";

      public class DbTest : MarshalByRefObject
      {
         public void Call()
         {
            using (SqlConnection conn = new SqlConnection(ConnectionString))
            {
               Console.WriteLine("Open connection");
               conn.Open();
            }
         }
      }

      static void Main(string[] args)
      {
         Console.WriteLine($"Start using {typeof(SqlConnection).Assembly.FullName}");
         AppDomain domain = AppDomain.CreateDomain("MyDomain");

         using (SqlConnection conn = new SqlConnection(ConnectionString))
         {
            Console.WriteLine("Open connection");
            conn.Open();

            DbTest dbTest = (DbTest)domain.CreateInstanceAndUnwrap(typeof(DbTest).Assembly.FullName, typeof(DbTest).FullName);
            try
            {
               dbTest.Call();
            }
            catch (Exception)
            {
               Console.WriteLine("Error");
            }

            Console.WriteLine("All calls done");
         }
      }
   }
}

ConsoleApp1.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net48</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
     <LangVersion>latest</LangVersion>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Data.SqlClient" Version="4.1.0" />
  </ItemGroup>

</Project>

And below is the output when running it, first with version 3.0.1 of Microsoft.Data.SqlClient, and then with version 4.1.0 (run on Windows 10 with latest updates in this scenario, but problem also observed on Windows Server 2019 and 2022)

PS> .\ConsoleApp1.exe
Start using Microsoft.Data.SqlClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5
Open connection
Open connection
All calls done

PS> $LASTEXITCODE
0

PS> .\ConsoleApp1.exe
Start using Microsoft.Data.SqlClient, Version=4.1.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5
Open connection
Open connection

PS> $LASTEXITCODE
-1073740791

The event log contains the following error entry after the second execution:

Faulting application name: ConsoleApp1.exe, version: 1.0.0.0, time stamp: 0xdd7eed65
Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c
Faulting process id: 0x2bf4
Faulting application start time: 0x01d828badaa0788b
Faulting application path: F:\ETWork\SqlClientTest\ConsoleApp1\ConsoleApp1\bin\Debug\net48\ConsoleApp1.exe
Faulting module path: F:\ETWork\SqlClientTest\ConsoleApp1\ConsoleApp1\bin\Debug\net48\Microsoft.Data.SqlClient.SNI.x64.dll
Report Id: 45bd024d-1012-4796-8dfc-3e0329cefe30
Faulting package full name: 
Faulting package-relative application ID:

If running from Visual Studio with Native Debugging enabled and break on exceptions, we can see the following exception text at the call to connection.Open() from the other AppDomain:

Unhandled exception at 0x00007FFEBBBEC80C (Microsoft.Data.SqlClient.SNI.x64.dll) in ConsoleApp1.exe: An invalid parameter was passed to a function that considers invalid parameters fatal.

JRahnama · 2022-03-04T21:32:56Z

@alphaleonis I tried your repro by adding [Serializable] to DbTest and it worked fine for v4.1.0 however when I try with v3.0.1 I get Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. I will try to solve that and test with that version as well.

IgorWolbers · 2022-03-21T13:05:53Z

We are having the same issue as described by @alphaleonis. We are loading multiple appdomains in a Windows Service and executing EF queries and after we updated our dependencies our app is now crashing if 2 app domains are simultaniously making queries to the database.

Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c

With the code that can reproduce the issue provided by @alphaleonis is there an update yet on this issue? This should be regarded as a high priority bug IMHO.

Edit

I reverted my package version of Microsoft.Data.SqlClient.SNI back to 2.1.1 and the problem does not occur. I have not tried 3.0.0.

alphaleonis · 2022-03-21T13:55:47Z

@JRahnama [Serializable] doesn't make a difference if the DbTest class derives from MarshalByRefObject. Of course, this sample works if we add [Serializable] and remove the inheritance from MarshalByRefObject, but then we're testing something else entirely. (The problem occurrs when we have SqlConnections opened in multiple AppDomains)

JasonAtATL · 2022-04-11T16:52:40Z

We recently upgraded our ASP.NET / .NET Framework 4.8 web app to use Microsoft.Data.SqlClient Version 4.1 and Microsoft.Data.SqlClient.SNI Version 4.0.

The new package worked fine locally and in our development environment, but when we deployed our app to our QA server the IIS application pool immediately crashes when starting up:

Faulting application name: w3wp.exe, version: 10.0.17763.1, time stamp: 0xcfdb13d8
Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c
Faulting process id: 0x292c
Faulting application start time: 0x01d84dbf4eb092c5
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: c:\inetpub\wwwroot\qa-edelweissplus\bin\Microsoft.Data.SqlClient.SNI.x64.dll
Report Id: 2ce9eb27-a486-4986-abaf-94fcf9eeac37
Faulting package full name: 
Faulting package-relative application ID:

We upgraded primarily because we lost SQL statement logging using Application Insights with Microsoft.Data.SqlClient 3.0 and upgrading to 4.0 seemed to fix it.

antymon4o · 2022-04-15T07:00:17Z

We are experiencing the same issue of crashing the process (w3wp) due to exception in Microsoft.Data.SqlClient.SNI. We use Microsoft.Data.SqlClient (4.1) and Microsoft.Data.SqlClient.SNI (4.0). The OS is Windows Server 2016 and Framework is 4.7.2.
This is the info from dumpproc

dumpproc version 1.231.282.20220209-154615, installer version 1.231.284.20220209-201902

time: 2022-04-13 08:16:55 UTC, processId: 18416, path: c:\windows\system32\inetsrv\w3wp.exe
Exception 0xc0000409: The system detected an overrun of a stack-based buffer in this application. This overrun could potentially allow a malicious user to gain control of this application.

Number of threads: 566
threadId 0x1880 - stack: 0000008844E00000-0000008844E80000, size: 512 kB (committed 84 kB) , CAUSED EXCEPTION
0x0000008844E767A0 0x00007FFDB467C80C Microsoft.Data.SqlClient.SNI.x6!SNIPacketSetData+0x17a0c
0x0000008844E76910 0x00007FFDB467CF04 Microsoft.Data.SqlClient.SNI.x6!SNIInitialize+0x84
0x0000008844E76980 0x00007FFD6EE2B402 !+0xba7cb402
0x0000008844E76988 0x00007FFD6EED0748 !+0xba870748

This issue has become very critical as it is happening in production environment and we had to rollback the deployment. This crash does not happen on any other SIT or UAT environments.
@DavoudEshtehari, It would be very helpful if someone could present a solution to this problem, because this behavior renders this package unreliable and unusable :(

JasonAtATL · 2022-04-29T12:41:34Z

Would it be possible to get an update on this issue?

We upgraded to Microsoft.Data.SqlClient 3.1 so we could take advantage of Active Directory Default authentication. Unfortunately, 3.1 broke Application Insights SQL telemetry as described here: #1258

The only way we can use Active Directory Default and restore AI SQL telemetry is to upgrade to 4.1, but it's crashing our w3wp.exe process.

I would also submit that the title of this issue should be change. This issue doesn't only affect IIS Express. It impacts web applications running under IIS as well.

alphaleonis · 2022-04-29T15:03:35Z

I would also submit that the title of this issue should be change. This issue doesn't only affect IIS Express. It impacts web applications running under IIS as well.

Or any other application using SqlClient from multiple AppDomains actually.

holc-tkomp · 2022-07-12T09:24:24Z

From my experience with an access violation error, it could be a mismatch between the version of MDS.SNI and MDS.

The thing is that I get this crash with MDS 4.1.0 w/ MDS.SNI 4.0.0, but it seems to work when using MDS 4.1.0 w/ MDS.SNI 3.0.0 😅

lcheunglci · 2022-08-05T20:54:42Z

The MDS v5.0.0 just went live, and I tried the 3 sample projects provided here: #1418 (comment) , #1418 (comment) , and
#1418 (comment) and it no longer crashed in the Microsoft.Data.SqlClient.SNI.dll.

agclark27 · 2022-08-06T03:23:24Z

@lcheunglci We're still seeing the crash in v5.0.0 with the sample projects provided. With #1418 (comment) as an example, it's still crashing on both x32 and x64 as shown below. The reproduction of the crash when using multiple appdomains is pretty consistent, but since most of the repros are console apps, you may not be seeing the crash unless you're looking in the event log or output, since it blows up the whole process. This is an issue not just for console apps but also for any use in IIS which makes regular use of appdomains with a .NET Framework app.

Faulting application name: SqlClientCrashTest.exe, version: 1.0.0.0, time stamp: 0x932c7422
Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 5.0.0.0, time stamp: 0x62e973e6
Exception code: 0xc0000409
Fault offset: 0x0000000000005b5d

lcheunglci · 2022-08-06T05:04:46Z

Sorry, I realized that I was testing a 5.0 dev SNI build, which worked. I just re-tried all 3 repro samples again using v5.0.0 from nuget, and it does continue to fail when multiple AppDomain instances are used. We'll have to further investigate and get back to you soon.

lcheunglci · 2022-08-17T17:06:00Z

I did some research and found the difference between my local environment and the environment on the pipeline that generates the SNI GA build. I found out that I was using an older version of the Windows 10 SDK 10.0.19041.0 and the pipeline vm has all the versions of the Window SDK installed including the latest i.e. 10.0.22000 and the SNI project references Windows 10 SDK without a specific version, so it uses the latest one installed. When I installed the latest Windows 10 SDK, I was able to reproduce the crash locally on my sample project and found out that the crash occurred inside of the registration of TraceLoggerProvider and compared the difference between the 2 versions of the Windows 10 SDK. It turns out a fatal error was introduced in the latest Windows 10 SDK version to prevent multiple registrations of the same TraceLoggingProvider (i.e. when using multiple AppDomain) due to a memory leak and memory corruption, so I've opened a PR to fix the multiple registrations issue. Hopefully, the fix will be included in the next 5.0 hotfix release and it might need to be back ported to the previous versions as well.

HamsterExAstris · 2022-08-17T17:30:31Z

Thanks for the information on the cause.

This would specifically only impact the SNI package/build, correct?

(I verified that v3.1.1 of MDS built 5 days ago works. If it only impacts the SNI package, then that would make sense, because that wasn't revved - it's still using v3.0.0 from June 2021.)

lcheunglci · 2022-08-17T18:16:13Z

Thanks for the information on the cause.

This would specifically only impact the SNI package/build, correct?

(I verified that v3.1.1 of MDS built 5 days ago works. If it only impacts the SNI package, then that would make sense, because that wasn't revved - it's still using v3.0.0 from June 2021.)

Yes, this only impacts the SNI nuget package, but from my knowledge, it usually released in sync with a MDS release.

HamsterExAstris · 2022-08-18T01:02:50Z

Yes, this only impacts the SNI nuget package, but from my knowledge, it usually released in sync with a MDS release.

Yes, SNI releases are (usually) synced to an MDS release, but not every MDS release has an SNI release. None of the 3.x or 4.x MDS patches have a corresponding SNI release, just 3.0.0, 4.0.0, and 5.0.0.

jehelles · 2022-08-22T09:35:27Z

I did some research and found the difference between my local environment and the environment on the pipeline that generates the SNI GA build. I found out that I was using an older version of the Windows 10 SDK 10.0.19041.0 and the pipeline vm has all the versions of the Window SDK installed including the latest i.e. 10.0.22000 and the SNI project references Windows 10 SDK without a specific version, so it uses the latest one installed. When I installed the latest Windows 10 SDK, I was able to reproduce the crash locally on my sample project and found out that the crash occurred inside of the registration of TraceLoggerProvider and compared the difference between the 2 versions of the Windows 10 SDK. It turns out a fatal error was introduced in the latest Windows 10 SDK version to prevent multiple registrations of the same TraceLoggingProvider (i.e. when using multiple AppDomain) due to a memory leak and memory corruption, so I've opened a PR to fix the multiple registrations issue. Hopefully, the fix will be included in the next 5.0 hotfix release and it might need to be back ported to the previous versions as well.

Please backport to 4.0 as well, as we cannot uptake a non-LTS release.

ProVega · 2022-08-30T02:33:55Z

Same repro on Windows 10 running IIS and Visual Studio 2022 17.3

Faulting application name: w3wp.exe, version: 10.0.19041.1, time stamp: 0x58c67bf3
Faulting module name: Microsoft.Data.SqlClient.SNI.x64.dll, version: 4.0.0.0, time stamp: 0x61953095
Exception code: 0xc0000409
Fault offset: 0x000000000001c80c
Faulting process id: 0x5fb4
Faulting application start time: 0x01d8bc175b0cab6c
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\730cb586\240d9e80\assembly\dl3\1d304453\000a58ab_f216d801\Microsoft.Data.SqlClient.SNI.x64.dll
Report Id: 90a210ef-cb1c-422d-893f-9d4acbf755b7
Faulting package full name:
Faulting package-relative application ID:

ProVega · 2022-08-30T02:36:15Z

I did some research and found the difference between my local environment and the environment on the pipeline that generates the SNI GA build. I found out that I was using an older version of the Windows 10 SDK 10.0.19041.0 and the pipeline vm has all the versions of the Window SDK installed including the latest i.e. 10.0.22000 and the SNI project references Windows 10 SDK without a specific version, so it uses the latest one installed. When I installed the latest Windows 10 SDK, I was able to reproduce the crash locally on my sample project and found out that the crash occurred inside of the registration of TraceLoggerProvider and compared the difference between the 2 versions of the Windows 10 SDK. It turns out a fatal error was introduced in the latest Windows 10 SDK version to prevent multiple registrations of the same TraceLoggingProvider (i.e. when using multiple AppDomain) due to a memory leak and memory corruption, so I've opened a PR to fix the multiple registrations issue. Hopefully, the fix will be included in the next 5.0 hotfix release and it might need to be back ported to the previous versions as well.

Any progress on this? On my local dev system (Windows 10) I am experiencing both the memory leak (There is insufficient system memory in resource pool default to run this query) AND the SNI crash.

davhdavh · 2022-08-30T05:10:43Z

Any news on this? Isn't it same bug as #1687?

lcheunglci · 2022-08-30T06:32:59Z

Any progress on this? On my local dev system (Windows 10) I am experiencing both the memory leak (There is insufficient system memory in resource pool default to run this query) AND the SNI crash.

I'm not sure if this will resolve the memory leak that you're encountering, but for the SNI crash fix is currently in review and once it's merged, it will be included in the next 5.0 hotfix release.

Any news on this? Isn't it same bug as #1687?

No, #1687 is a different issue and the fix from here won't resolve the unloading issue.

isandburn · 2022-09-13T20:17:46Z

Not to belabor the issue, but I think that I too have been experiencing this issue... below is the abbreviated stacktrace I am seeing when debugging the static crash data in VS 2022. @lcheunglci - Could you (or someone) please verify that this is in fact the same issue being addressed in the coming 5.0 hotfix? This also may help others identify is this is in fact their issue as well. Any/All help is greatly appreciated. EDIT: adding @DavoudEshtehari

Microsoft.Data.SqlClient.SNI.x64.dll!SNITraceLogging::registerProvider(void)	
Microsoft.Data.SqlClient.SNI.x64.dll!SNIInitializeEx�()	
Microsoft.Data.SqlClient.SNI.x64.dll!SNIInitialize�()	
00007fff00694ee2()	
00007fff006946bf()	
00007fff00694695()	
00007fff006940fc()	
clr.dll!CallDescrWorkerInternal�()	
clr.dll!CallDescrWorkerWithHandler()	
clr.dll!DispatchCallDebuggerWrapper(struct CallDescrData *,class ContextTransitionFrame *,int)	
clr.dll!DispatchCallSimple()	
clr.dll!MethodTable::RunClassInitEx()	
clr.dll!MethodTable::DoRunClassInitThrowing()	
clr.dll!MethodTable::CheckRunClassInitThrowing()	
clr.dll!JIT_GetSharedNonGCStaticBase_Helper()	
00007fff00693efc()	
clr.dll!CallDescrWorkerInternal�()	
clr.dll!CallDescrWorkerWithHandler()	
clr.dll!DispatchCallDebuggerWrapper(struct CallDescrData *,class ContextTransitionFrame *,int)	
clr.dll!DispatchCallSimple()	
clr.dll!MethodTable::RunClassInitEx()	
clr.dll!MethodTable::DoRunClassInitThrowing()	
clr.dll!MethodTable::CheckRunClassInitThrowing()	
clr.dll!JIT_GetSharedNonGCStaticBase_Helper()	

... skipping a bunch of tokenized-named functions for brevity...

clr.dll!CallDescrWorkerInternal�()	
clr.dll!CallDescrWorkerWithHandler()	
clr.dll!MethodDescCallSite::CallTargetWorker()	
clr.dll!QueueUserWorkItemManagedCallback(void *)	
clr.dll!ManagedThreadBase_DispatchInner()	
clr.dll!ManagedThreadBase_DispatchMiddle()	
clr.dll!ManagedThreadBase_DispatchOuter()	
clr.dll!ManagedThreadBase_DispatchInCorrectAD()	
clr.dll!Thread::DoADCallBack(struct ADID,void (*)(void *),void *,int)	
clr.dll!ManagedThreadBase_DispatchInner()	
clr.dll!ManagedThreadBase_DispatchMiddle()	
clr.dll!ManagedThreadBase_DispatchOuter()	
clr.dll!ManagedThreadBase_FullTransitionWithAD()	
clr.dll!ManagedPerAppDomainTPCount::DispatchWorkItem()	
clr.dll!ThreadpoolMgr::ExecuteWorkRequest()	
clr.dll!ThreadpoolMgr::WorkerThreadStart()	
clr.dll!Thread::intermediateThreadProc(void *)	
kernel32.dll!BaseThreadInitThunk�()	
ntdll.dll!RtlUserThreadStart�()

lcheunglci · 2022-09-13T20:44:41Z

Not to belabor the issue, but I think that I too have been experiencing this issue... below is the abbreviated stacktrace I am seeing when debugging the static crash data in VS 2022. @lcheunglci - Could you (or someone) please verify that this is in fact the same issue being addressed in the coming 5.0 hotfix? This also may help others identify is this is in fact their issue as well. Any/All help is greatly appreciated. EDIT: adding @DavoudEshtehari
Microsoft.Data.SqlClient.SNI.x64.dll!SNITraceLogging::registerProvider(void)	
Microsoft.Data.SqlClient.SNI.x64.dll!SNIInitializeEx�()	
Microsoft.Data.SqlClient.SNI.x64.dll!SNIInitialize�()	

Yes, the fix will resolve this issue. As the issue lies in the registerProvider SNI method when registering the same provider more than once.

DavoudEshtehari · 2022-09-15T18:32:08Z

The PR has been merged and will be in the next release.

isandburn · 2022-09-15T20:50:22Z

@DavoudEshtehari - that's great news, thank you. unfortunately i am not familiar with your release cadence - do you have any estimation on when i may expect next release?

DavoudEshtehari · 2022-09-15T21:03:38Z

Two recent hotfixes 4.0.2 and 4.1.1 ship this fix. And, the next version 5.1 will be released till end of this year.

Release Notes

gregpakes · 2022-09-21T22:57:02Z

@DavoudEshtehari - We are experiencing this issue and we are using 5.0.0.

When you said 5.1 will be released by the end of the year - did you mean that? I guess I'll have to use 4.1.1

DavoudEshtehari · 2022-09-22T00:09:43Z

@gregpakes Yes, 5.1 is on the plan to be released this year.

tdhintz · 2022-09-29T18:31:59Z

A patch for 5.0 is needed asap as well.

JRahnama · 2022-09-29T19:55:19Z

@tdhintz we are planning on a hotfix of 5.0.1. Date yet to be decided.

tdhintz · 2022-09-30T02:54:04Z

Can you give any background that would be useful in understanding the root cause? I have an Win server 2016 running IIS in an x64 app pool on an 8 core virtual. It isn't under much load, yet it seems to crash. The crash occurs about the same time a group of sessions are ended together.

DavoudEshtehari · 2022-10-11T20:22:57Z

I'm glad to announce that Microsoft.Data.SqlClient v5.0.1 has been released.

JRahnama added this to Needs triage in SqlClient Triage Board via automation Dec 3, 2021

JRahnama moved this from Needs triage to Under Investigation in SqlClient Triage Board Dec 3, 2021

DavoudEshtehari self-assigned this Mar 18, 2022

MojoDK changed the title ~~Microsoft.Data.SqlClient.SNI.x64.dll crashes iisexpress~~ Microsoft.Data.SqlClient.SNI.x64.dll crashes IIS Express and IIS Server (w3wp.exe) Apr 29, 2022

DavoudEshtehari closed this as completed Sep 15, 2022

SqlClient Triage Board automation moved this from Under Investigation to Closed Sep 15, 2022

gregpakes mentioned this issue Sep 21, 2022

DISK IO wait due to the millions of update statement HangfireIO/Hangfire#1962

Closed

HaoK mentioned this issue Oct 6, 2022

ANCM (In-proc) can cause IIS worker process to become unhealthy dotnet/aspnetcore#44211

Open

davhdavh mentioned this issue Oct 13, 2022

Occasional crash in IIS after compile dotnet/aspnetcore#44546

Closed

cvocvo mentioned this issue Mar 31, 2023

503 error with InProcess hosting, completely fine with OutOfProcess hosting dotnet/aspnetcore#47518

Closed

1 task

Microsoft.Data.SqlClient.SNI.x64.dll crashes when using multiple AppDomains in IIS Express and IIS Server #1418

Microsoft.Data.SqlClient.SNI.x64.dll crashes when using multiple AppDomains in IIS Express and IIS Server #1418

Comments

MojoDK commented Dec 2, 2021

Describe the bug

To reproduce

Expected behavior

Further technical details

JRahnama commented Dec 8, 2021

MojoDK commented Dec 9, 2021

JRahnama commented Dec 9, 2021

leonmeijer commented Jan 11, 2022 • edited

jaymonson commented Jan 11, 2022

JRahnama commented Jan 11, 2022

jaymonson commented Jan 11, 2022 via email

tdhintz commented Jan 11, 2022

leonmeijer commented Jan 11, 2022 via email

tdhintz commented Jan 11, 2022

waf commented Jan 14, 2022 • edited

agclark27 commented Jan 24, 2022

JRahnama commented Jan 24, 2022

agclark27 commented Jan 24, 2022

JRahnama commented Jan 24, 2022

leonmeijer commented Jan 24, 2022

leonmeijer commented Jan 28, 2022

JRahnama commented Jan 28, 2022

alphaleonis commented Feb 23, 2022 • edited

JRahnama commented Mar 4, 2022

IgorWolbers commented Mar 21, 2022 • edited

alphaleonis commented Mar 21, 2022 • edited

JasonAtATL commented Apr 11, 2022

antymon4o commented Apr 15, 2022

JasonAtATL commented Apr 29, 2022 • edited

alphaleonis commented Apr 29, 2022

holc-tkomp commented Jul 12, 2022

lcheunglci commented Aug 5, 2022

agclark27 commented Aug 6, 2022

lcheunglci commented Aug 6, 2022 • edited

lcheunglci commented Aug 17, 2022 • edited by David-Engel

HamsterExAstris commented Aug 17, 2022 • edited

lcheunglci commented Aug 17, 2022

HamsterExAstris commented Aug 18, 2022

jehelles commented Aug 22, 2022

ProVega commented Aug 30, 2022

ProVega commented Aug 30, 2022 • edited

davhdavh commented Aug 30, 2022

lcheunglci commented Aug 30, 2022

isandburn commented Sep 13, 2022 • edited

lcheunglci commented Sep 13, 2022

DavoudEshtehari commented Sep 15, 2022

isandburn commented Sep 15, 2022

DavoudEshtehari commented Sep 15, 2022 • edited

gregpakes commented Sep 21, 2022

DavoudEshtehari commented Sep 22, 2022

tdhintz commented Sep 29, 2022

JRahnama commented Sep 29, 2022

tdhintz commented Sep 30, 2022

DavoudEshtehari commented Oct 11, 2022

leonmeijer commented Jan 11, 2022 •

edited

waf commented Jan 14, 2022 •

edited

alphaleonis commented Feb 23, 2022 •

edited

IgorWolbers commented Mar 21, 2022 •

edited

alphaleonis commented Mar 21, 2022 •

edited

JasonAtATL commented Apr 29, 2022 •

edited

lcheunglci commented Aug 6, 2022 •

edited

lcheunglci commented Aug 17, 2022 •

edited by David-Engel

HamsterExAstris commented Aug 17, 2022 •

edited

ProVega commented Aug 30, 2022 •

edited

isandburn commented Sep 13, 2022 •

edited

DavoudEshtehari commented Sep 15, 2022 •

edited