Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rfc: Add locks to deal avoid fatal reads and writes to map. #1353

Closed
wants to merge 1 commit into from
Closed

rfc: Add locks to deal avoid fatal reads and writes to map. #1353

wants to merge 1 commit into from

Conversation

eiffel-fl
Copy link
Contributor

Hi.

In inspektor-gadget/inspektor-gadget#2513, we wanted to bump cilium/ebpf to v0.13.0.
Sadly, our CI caught some problems while running unit tests.
Basically, we run several tests in parallel where we create a lot of eBPF related structures.
I took a look at it and found that the fatal concurrent operations occur there:

ebpf/btf/btf.go

Lines 78 to 94 in ff37506

return modifyGraphPreorder(typ, func(t Type) (Type, bool) {
cpy, ok := mt.copies[t]
if ok {
// This has been copied previously, no need to continue.
return cpy, false
}
cpy = t.copy()
mt.copies[t] = cpy
if id, ok := typeIDs[t]; ok {
mt.copiedTypeIDs[cpy] = id
}
// This is a new copy, keep copying children.
return cpy, true
})

Here is the output I get when I ran the tests locally:

=== RUN   TestCapabilitiesTracerCreate
=== PAUSE TestCapabilitiesTracerCreate
=== RUN   TestCapabilitiesTracer
=== PAUSE TestCapabilitiesTracer
=== RUN   TestCapabilitiesTracerMultipleMntNsIDsFilter
=== PAUSE TestCapabilitiesTracerMultipleMntNsIDsFilter
=== CONT  TestCapabilitiesTracerCreate
=== CONT  TestCapabilitiesTracerMultipleMntNsIDsFilter
=== CONT  TestCapabilitiesTracer
=== RUN   TestCapabilitiesTracer/captures_events_with_matching_filter_CAP_NET_BIND_SERVICE
=== PAUSE TestCapabilitiesTracer/captures_events_with_matching_filter_CAP_NET_BIND_SERVICE
=== RUN   TestCapabilitiesTracer/captures_all_events_with_no_filters_configured
=== PAUSE TestCapabilitiesTracer/captures_all_events_with_no_filters_configured
=== RUN   TestCapabilitiesTracer/captures_no_events_with_no_matching_filter
=== PAUSE TestCapabilitiesTracer/captures_no_events_with_no_matching_filter
=== RUN   TestCapabilitiesTracer/captures_events_with_matching_filter_CAP_CHOWN
=== PAUSE TestCapabilitiesTracer/captures_events_with_matching_filter_CAP_CHOWN
=== CONT  TestCapabilitiesTracer/captures_events_with_matching_filter_CAP_NET_BIND_SERVICE
=== CONT  TestCapabilitiesTracer/captures_no_events_with_no_matching_filter
=== CONT  TestCapabilitiesTracer/captures_all_events_with_no_filters_configured
=== CONT  TestCapabilitiesTracer/captures_events_with_matching_filter_CAP_CHOWN
fatal error: concurrent map read and map write

goroutine 37 [running]:
github.com/cilium/ebpf/btf.(*mutableTypes).typeByID.(*mutableTypes).add.func1({0xb409e0, 0xc000c27e90})
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/btf.go:79 +0x45
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0018d20b0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:112 +0x92
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc001891230?}, 0xc001ad05b8)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc00189e280)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc00189e280?}, 0xc001ad04e8?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc00189a710)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc001891200?}, 0xc001ad05b8)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder({0xb409e0, 0xc000c27bc0}, 0xc001ad0740)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:122 +0x145
github.com/cilium/ebpf/btf.(*mutableTypes).add(...)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/btf.go:78
github.com/cilium/ebpf/btf.(*mutableTypes).typeByID(0x1000000414555?, 0x601a20?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/btf.go:139 +0x85
github.com/cilium/ebpf/btf.(*Spec).TypeByID(0xc001955d40, 0xb409e0?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/btf.go:652 +0x25
github.com/cilium/ebpf/btf.coreCalculateFixups({0xc00189e270, 0x2, 0xc00031287c?}, 0x4?, {0xc0007e0f98, 0x1, 0x18?}, {0xb43c80, 0xef7360})
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/core.go:258 +0xe9
github.com/cilium/ebpf/btf.CORERelocate({0xc0018a6b00, 0xc, 0xc001ad0c70?}, 0x5f913f?, {0xb43c80?, 0xef7360?}, 0xc001ad0cd0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/core.go:231 +0x6a5
github.com/cilium/ebpf.applyRelocations({0xc0018bf000?, 0xc0018bf000?, 0x0?}, 0xc0018be000?, {0xb43c80, 0xef7360}, 0xc001ad10d0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/linker.go:141 +0x2dd
github.com/cilium/ebpf.newProgramWithOptions(0xc001ad1338, {0xb3e100?, 0xc0018ae140?, 0x0, 0x0?})
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/prog.go:246 +0x439
github.com/cilium/ebpf.(*collectionLoader).loadProgram(0xc001892000, {0x7e01ac, 0xe})
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/collection.go:531 +0x416
github.com/cilium/ebpf.(*CollectionSpec).LoadAndAssign.func1({0xb48a20?, 0x877c60}, {0x7e01ac, 0xe})
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/collection.go:295 +0x1f9
github.com/cilium/ebpf.assignValues({0x810e40, 0xc000340008}, 0xc001ad1870)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/collection.go:911 +0x402
github.com/cilium/ebpf.(*CollectionSpec).LoadAndAssign(0xc0005291a0, {0x810e40, 0xc000340008}, 0x16?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/collection.go:307 +0x1b1
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets.LoadeBPFSpec(0xc00032a0a0, 0xc0005291a0, 0xc0007a7cf0, {0x810e40, 0xc000340008})
	/home/francis/Codes/kinvolk/inspektor-gadget/pkg/gadgets/helpers.go:264 +0x28c
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer.(*Tracer).install(0xc000340000)
	/home/francis/Codes/kinvolk/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer/tracer.go:150 +0x1bd
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer.NewTracer(0xc00030e0c0, {0x0?, 0x0}, 0xc00040e040)
	/home/francis/Codes/kinvolk/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer/tracer.go:110 +0xa8
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer_test.createTracer(0xc000202d00, 0xc000202d00?, 0x27?)
	/home/francis/Codes/kinvolk/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer/tracer_test.go:427 +0x3b
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer_test.TestCapabilitiesTracer.func22(0x0?)
	/home/francis/Codes/kinvolk/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer/tracer_test.go:342 +0xda
testing.tRunner(0xc000202d00, 0xc00021c810)
	/usr/lib/go-1.21/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 7
	/usr/lib/go-1.21/src/testing/testing.go:1648 +0x3ad

goroutine 6 [runnable]:
github.com/cilium/ebpf/btf.(*mutableTypes).typeByID.(*mutableTypes).add.func1({0xb409e0, 0xc001712c60})
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/btf.go:88 +0xd2
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002680da0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:112 +0x92
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002680da0?}, 0xc0028876b0?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc00268b190)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0026ca030?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0026ae718)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0026c5fb0?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002680d80)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002680d80?}, 0xc002887920?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0026b0670)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb40a10?, 0xc0026c5f80?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:161 +0x334
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002680d70)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002680d70?}, 0xc002887ac0?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0026bcc10)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0026c4ae0?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002680940)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb414c0?, 0xc002680940?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:153 +0x172
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002680930)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002680930?}, 0xc002887d30?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc00267cd90)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0026c4ab0?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002680920)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002680920?}, 0xc002887ed0?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc00269ca60)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0026c4a80?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0026c4a60)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb41580?, 0xc0026c4a20?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:146 +0x390
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0026ac530)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0026c49f0?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002680910)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002680910?}, 0xc002888210?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0026c6538)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0026c4210?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0026807a0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc0026807a0?}, 0xc00288e600?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc00267c828)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0026c4030?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002680740)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002680740?}, 0xc002888550?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc00269c6a0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0026c4000?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002671fc0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb40980?, 0xc002671fb0?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:139 +0xd3
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc00269c678)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc002671f80?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002680730)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002680730?}, 0xc00288e600?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0026aa408)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc002670a50?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0026801d0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
...195 frames elided...
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0019a24e8)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc002801140?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002810140)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002810140?}, 0xc001ad4600?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc001786998)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc002800ff0?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002836128)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc002800e40?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002810110)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002810110?}, 0x8300000000408c4d?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002830150)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb40a10?, 0xc002800e10?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:161 +0x334
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc002810100)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc002810100?}, 0xc001ad4600?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc001250ae0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc002800c30?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc00282a4c0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc002800c00?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0028100f0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc0028100f0?}, 0xc001ad4600?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0028343d0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc002800900?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0028100d0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc0028100d0?}, 0xc001ad4600?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0028323a8)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc002800660?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc0028100c0)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409b0?, 0xc0028100c0?}, 0xc001ad4600?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:136 +0x92
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc00280a448)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0028005a0?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder.func1(0xc000b1ed60)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:117 +0x126
github.com/cilium/ebpf/btf.walkType({0xb409e0?, 0xc0028003c0?}, 0xc00288e600)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:142 +0x3ed
github.com/cilium/ebpf/btf.modifyGraphPreorder({0xb409e0, 0xc000cedfb0}, 0xc001ad4788)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/traversal.go:122 +0x145
github.com/cilium/ebpf/btf.(*mutableTypes).add(...)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/btf.go:78
github.com/cilium/ebpf/btf.(*mutableTypes).typeByID(0x60c420?, 0x8?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/btf.go:139 +0x85
github.com/cilium/ebpf/btf.(*Spec).TypeByID(0xc001955d40, 0x6a143108?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/btf.go:652 +0x25
github.com/cilium/ebpf/btf.coreCalculateFixups({0xc000418018, 0x1, 0xc0000153f0?}, 0xb?, {0xc0007e0980, 0x1, 0x200000003?}, {0xb43c80, 0xef7360})
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/core.go:258 +0xe9
github.com/cilium/ebpf/btf.CORERelocate({0xc002802040, 0x4, 0x862500?}, 0xc002806288?, {0xb43c80?, 0xef7360?}, 0xc001ad4d18)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/btf/core.go:231 +0x6a5
github.com/cilium/ebpf.applyRelocations({0xc00280e500?, 0xc?, 0x22000000866060?}, 0xc000156e90?, {0xb43c80, 0xef7360}, 0xc001ad5118)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/linker.go:141 +0x2dd
github.com/cilium/ebpf.newProgramWithOptions(0xc00288f380, {0xb3e100?, 0xc00282a0a0?, 0x0, 0x0?})
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/prog.go:246 +0x439
github.com/cilium/ebpf.(*collectionLoader).loadProgram(0xc00193ae20, {0x7e14cb, 0x10})
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/collection.go:531 +0x416
github.com/cilium/ebpf.(*CollectionSpec).LoadAndAssign.func1({0xb48a20?, 0x877c60}, {0x7e14cb, 0x10})
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/collection.go:295 +0x1f9
github.com/cilium/ebpf.assignValues({0x810e40, 0xc0000ba378}, 0xc00288f8b8)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/collection.go:911 +0x402
github.com/cilium/ebpf.(*CollectionSpec).LoadAndAssign(0xc00072c960, {0x810e40, 0xc0000ba378}, 0x16?)
	/home/francis/go/pkg/mod/github.com/cilium/ebpf@v0.13.0/collection.go:307 +0x1b1
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets.LoadeBPFSpec(0x0, 0xc00072c960, 0xc0001c7d38, {0x810e40, 0xc0000ba378})
	/home/francis/Codes/kinvolk/inspektor-gadget/pkg/gadgets/helpers.go:264 +0x28c
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer.(*Tracer).install(0xc0000ba370)
	/home/francis/Codes/kinvolk/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer/tracer.go:150 +0x1bd
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer.NewTracer(0xc000032a90, {0x0?, 0x0}, 0xad0308)
	/home/francis/Codes/kinvolk/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer/tracer.go:110 +0xa8
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer_test.createTracer(0xc0001ce4e0, 0xc0001ce4e0?, 0x0?)
	/home/francis/Codes/kinvolk/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer/tracer_test.go:427 +0x3b
github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer_test.TestCapabilitiesTracerCreate(0xc0001ce4e0)
	/home/francis/Codes/kinvolk/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer/tracer_test.go:44 +0x4d
testing.tRunner(0xc0001ce4e0, 0xad01c0)
	/usr/lib/go-1.21/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/usr/lib/go-1.21/src/testing/testing.go:1648 +0x3ad

FAIL	github.com/inspektor-gadget/inspektor-gadget/pkg/gadgets/trace/capabilities/tracer	0.167s
FAIL

So, in this patch, I added some locks to prevent having concurrent operations on the map.
I am doubtful this is a good fix, particularly because the goal of 4ad6b8c was to make everything faster and by adding locks I for sure decrease the speed.
But at least, the problem is known and we can discuss it.

Best regards.

@eiffel-fl eiffel-fl mentioned this pull request Feb 20, 2024
Merged
@lmb
Copy link
Collaborator

lmb commented Feb 20, 2024

Damn! I guess this happens because you read an ELF once, and then do multiple calls to NewCollection or similar?

@eiffel-fl
Copy link
Contributor Author

eiffel-fl commented Feb 21, 2024 •
edited

Damn! I guess this happens because you read an ELF once, and then do multiple calls to NewCollection or similar?

It only occurs in our unit tests, because we run them as t.Parallel().
But yes, we create several tracers as once, which call newProgramWithOptions() reading BTF from /sys/kernel/btf/vmlinux.

@eiffel-fl
wip: Add locks to deal avoid fatal reads and writes to map.
1e88fa8 
Signed-off-by: Francis Laniel <flaniel@linux.microsoft.com>
@lmb
Copy link
Collaborator

lmb commented Feb 22, 2024 •
edited

See #1357

@lmb
Copy link
Collaborator

lmb commented Feb 22, 2024

Thanks for raising this!

@lmb lmb closed this Feb 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dylandreimerink dylandreimerink Awaiting requested review from dylandreimerink dylandreimerink will be requested when the pull request is marked ready for review dylandreimerink is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@eiffel-fl @lmb