Fix deadlock during NRI plugin registration #79
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
What are you looking for to make this non-WIP? |
|
I think this should be fine. |
During NRI external plugin registration: * `acceptPluginConnections()` is called * adaptation lock from `nri/pkg/adaptation` is acquired * `syncFn` is invoked * `syncFn` acquires NRI lock in `pkg/nri/nri.go` During container lifecycle events such as `ContainerStart` * NRI lock is acquired in pkg/nri.go * adaptation lock is acquired in `StateChange()` in `nri/pkg/adaptation` As a result, the locking order during NRI plugin registration is: * adaptation lock -> NRI lock While the locking order during container starts is: * NRI lock -> adaptation lock Due the fact that the locking order is inverted and not consistent, it it possible to encounter a deadlock. To fix the issue, during NRI plugin registration, first acquire the NRI lock (done via `syncFn` call) and only after acquire the adaptation lock. This ensures that NRI plugin registration the locking order is adaption lock -> NRI lock, which is consistent with the locking order during container lifecycle events. Fixes containerd/containerd#10085 Signed-off-by: David Porter <porterdavid@google.com>
bobbypage
changed the title
I removed WIP, wanted to test it again a few times against the repro in containerd/containerd#10085 (comment) After a few runs of those steps, I can confirm that without this patch the deadlock always repros and with this patch I am no longer able to repro the deadlock. |
samuelkarp
approved these changes
Apr 18, 2024
estesp
approved these changes
Apr 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
During NRI external plugin registration:
nri/pkg/adaptationis acquiredsyncFnis invokedsyncFnacquires NRI lock inpkg/nri/nri.goDuring container lifecycle events such as
ContainerStartStateChange()innri/pkg/adaptationAs a result, the locking order during NRI plugin registration is:
While the locking order during container starts is:
Due the fact that the locking order is inverted and not consistent, it it possible to encounter a deadlock.
To fix the issue, during NRI plugin registration, first acquire the NRI lock (done via
syncFncall) and only after acquire the adaptation lock. This ensures that NRI plugin registration the locking order is adaption lock -> NRI lock, which is consistent with the locking order during container lifecycle events.Fixes containerd/containerd#10085