Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

btf: fix race due to concurrent read access #1357

Merged
merged 1 commit into from Feb 22, 2024

Conversation

lmb
Copy link
Collaborator

@lmb lmb commented Feb 22, 2024

Up until the introduction of lazy copying, reading from a Spec concurrently was safe. Now a read may trigger a copy and a write into the Spec, therefore causing a race on mutableTypes.

Fix this by introducing a mutex which protects access to the mutable state. We need to be a bit careful here: copying in mutableTypes.add happens piecemeal, so we need to take a lock for the whole duration of modifyGraph.

@lmb lmb requested a review from a team as a code owner February 22, 2024 11:32
@lmb
Copy link
Collaborator Author

lmb commented Feb 22, 2024

@eiffel-fl the locking ended up being more subtle than what you had in your PR. Can you give this a whirl?


if ok {
// Fast path: the type has been copied before.
return cpy
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come we didn't have this before? 🤔

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the same thing happens when entering the closure passed to modifyGraphPreorder.

Copy link
Collaborator

@ti-mo ti-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up! We should release 0.13.1 to address this.

@lmb Looks like CI doesn't like it.

@lmb lmb force-pushed the btf-spec-concurrency branch 2 times, most recently from accdd88 to 4456db1 Compare February 22, 2024 11:53
Up until the introduction of lazy copying, reading from a Spec
concurrently was safe. Now a read may trigger a copy and a write
into the Spec, therefore causing a race on mutableTypes.

Fix this by introducing a mutex which protects access to the
mutable state. We need to be a bit careful here: copying in
mutableTypes.add happens piecemeal, so we need to take a lock
for the whole duration of modifyGraph.

Signed-off-by: Lorenz Bauer <lmb@isovalent.com>
@lmb lmb merged commit 26065e7 into cilium:main Feb 22, 2024
15 checks passed
@lmb lmb deleted the btf-spec-concurrency branch February 22, 2024 12:14
@eiffel-fl
Copy link
Contributor

Hi!

@eiffel-fl the locking ended up being more subtle than what you had in your PR. Can you give this a whirl?

We are not on the same timezone, hence I can only test it now but everything seems OK locally:

$ sudo go test ./...
go: downloading github.com/hashicorp/go-multierror v1.1.1
go: downloading github.com/hashicorp/errwrap v1.1.0
ok      github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer      1.309s

I also updated our PR to bump directly to 0.13.2 and the unit tests passed cleanly:
https://github.com/inspektor-gadget/inspektor-gadget/actions/runs/8014473871/job/21893122556?pr=2513

So, thank you for quickly fixing this :D!

Best regards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants