Fix ContentStore locking exceptions in async code #17246
Conversation
|
As expected, the test without fix results in this exception: I've applied the same change in The only remaining usage of Umbraco-CMS/src/Umbraco.Web.Common/Cache/HttpContextRequestAppCache.cs Lines 215 to 263 in 7787af2
|
* Add ContentCache test * Use SemaphoreSlim as write lock * Apply lock imrpovements to SnapDictionary * Obsolete unused MonitorLock (cherry picked from commit c3db345)
|
Awesome, let's verify whether this not only fixes the test, but also the runtime exceptions!👍🏻 Looking more into |
* Add ContentCache test * Use SemaphoreSlim as write lock * Apply lock imrpovements to SnapDictionary * Obsolete unused MonitorLock (cherry picked from commit c3db345)
|
Hello, I wanted to touch base with you all, because, since this was applied automatically by Umbraco Cloud, or while running recurring Jobs that pulls from APIs and store locally. var results = "pull from API and parse with the Proper class"
foreach (var item in results)
{
IContent dbNode = null;
IPublishedContent? found = _publishedContentQuery.Content("<parentFOlderID>").Children().Where(x => x.Value<string>("externalId") == item.Id.Trim()).FirstOrDefault();
if(found != null){
dbNode = ContentService.GetById(found.Id);
} else {
dbNode = ContentService.Create(track.Title, int.Parse(parentId), "<nodeType>");
}
// Set Values
dbNode.SetValue("...." , item.Id);
// .....
try
{
ContentService.SaveAndPublish(dbNode);
}
catch (AggregateException ae)
{
Console.WriteLine("Caught aggregate exception behavior"+ae.Message);
}
}While using For moving content between folders, we are "workarounding" it by moving smaller groups of content. Is this the intended behavoir? Should I open a new issue? |
Looking at the stack trace, the error happens in a notification handler (as notifications are published in the |
|
We are experiencing the same issue as user We can probably get round it by making the jobs run at different times, but it'd be good to understand the root cause here. I can reproduce the issue locally by triggering the two jobs together. Adding ICoreScopeProvider as in the documentation has not helped. |
|
@lauren-g-2025 After reviewing our code and verifying that we were not customizing the handler, we end up "workarounding" it. So basically: But going back to the main problem it is definitively this change. I hope this helps. |
|
@sauron Can you share the stack traces of the inner exceptions of that The write lock is enlisted in the current scope here: Umbraco-CMS/src/Umbraco.Infrastructure/Scoping/ScopeContextualBase.cs Lines 47 to 58 in fd9c1a0 If there's no scope, it will call the Umbraco-CMS/src/Umbraco.PublishedCache.NuCache/ContentStore.cs Lines 317 to 318 in ed0b236 Also, using You might need to wrap the code in a @lauren-g-2025 Umbraco uses distributed write locks, so only a single write operation (like updating content) can be done at a time, so running 2 background jobs at the same isn't recommended for that reason alone. But otherwise, it seems like Hangfire executes jobs from the same thread, potentially causing issues with Umbraco contexts being shared between jobs, so the same |
Prerequisites
Description
After large Deploy operations (like an environment restore) has completed and the CMS tries to update the content cache (NuCache) as part of completing the Umbraco scope, the following exceptions were thrown:
This all originates from the
ContentStoreeither not 'seeing' it already has a write lock or getting a timeout when trying to get it (essentially deadlocking). Looking at the locking implementation in this class, it usesMonitor, which has the following requirement:But looking at the code and stack trace, the lock/monitor is entered and exited in different methods and they can potentially be called from different threads, which is especially likely if it contains async calls in-between them (Deploy heavily uses async and does more work in a single lock, further increasing the chances of exiting on a different thread).
I've added a test showcasing one of the exceptions and will push a fix that uses
SemaphoreSlim.