cmd/go/internal/list: segmentation fault on darwin

[podtserkovskiy]

Go version

go1.22.2 darwin/amd64

Output of go env in your module/workspace:

GO111MODULE='off'
GOARCH='amd64'
GOBIN=''
GOCACHE='/paragon/pods/248162430/home/execution/3/8baeb09e22e747828c4b95d8f75cf17c/cachedir/4673ec2911b04f06'
GOENV='/Users/macmini/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT='nocoverageredesign'
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/macmini/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/macmini/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/paragon/pods/248162430/home/execution/3/8baeb09e22e747828c4b95d8f75cf17c/work/third-party/go/1.22.2/darwin_amd64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/paragon/pods/248162430/home/execution/3/8baeb09e22e747828c4b95d8f75cf17c/work/third-party/go/1.22.2/darwin_amd64/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.22.2'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='0'
GOMOD=''
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/paragon/pods/248162430/home/execution/3/8baeb09e22e747828c4b95d8f75cf17c/tmp/go-build422398310=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Ran go list -e '-json=GoFiles,CgoFiles,HFiles,CFiles,CXXFiles,SFiles,EmbedFiles' -tags custom_tag

This is a flaky issue, rerun helps. However it's happening regularly.

What did you see happen?

unexpected fault address 0x100001550
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100001550 pc=0x5deaffe]

goroutine 20 gp=0xc0000948c0 m=2 mp=0xc00006e908 [running]:
runtime.throw({0x630e78e?, 0x0?})
	runtime/panic.go:1023 +0x5c fp=0xc00014f9a8 sp=0xc00014f978 pc=0x5db32fc
runtime.sigpanic()
	runtime/signal_unix.go:895 +0x285 fp=0xc00014fa08 sp=0xc00014f9a8 pc=0x5dcbd05
gogo()
	runtime/asm_amd64.s:422 +0x3e fp=0xc00014fa10 sp=0xc00014fa08 pc=0x5deaffe
created by cmd/go/internal/load.(*preload).preloadImports in goroutine 1
	cmd/go/internal/load/pkg.go:1114 +0x265

goroutine 1 gp=0xc0000061c0 m=nil [runnable]:
syscall.syscall(0xc000508000?, 0x88?, 0x76?, 0x1?)
	runtime/sys_darwin.go:23 +0x70 fp=0xc00003f9b0 sp=0xc00003f920 pc=0x5dea850
syscall.Open({0xc000508000?, 0x2?}, 0x1000000, 0x0)
	syscall/zsyscall_darwin_amd64.go:1162 +0x74 fp=0xc00003fa00 sp=0xc00003f9b0 pc=0x5e02c34
os.open(...)
	os/file_open_unix.go:15
os.openFileNolog({0xc000508000, 0x88}, 0x0, 0x0)
	os/file_unix.go:272 +0xe5 fp=0xc00003fa50 sp=0xc00003fa00 pc=0x5e6a945
os.OpenFile({0xc000508000, 0x88}, 0x0, 0x0)
	os/file.go:374 +0x3e fp=0xc00003fa80 sp=0xc00003fa50 pc=0x5e68a1e
cmd/go/internal/fsys.openFile({0xc000508000, 0x88}, 0x0, 0x0)
	cmd/go/internal/fsys/fsys.go:435 +0x345 fp=0xc00003fb18 sp=0xc00003fa80 pc=0x5f303e5
cmd/go/internal/fsys.Open({0xc000508000, 0x88})
	cmd/go/internal/fsys/fsys.go:403 +0x45 fp=0xc00003fb48 sp=0xc00003fb18 pc=0x5f30065
cmd/go/internal/cfg.defaultContext.func1({0xc000508000?, 0x90?})
	cmd/go/internal/cfg/cfg.go:171 +0x18 fp=0xc00003fb68 sp=0xc00003fb48 pc=0x5f37318
go/build.(*Context).openFile(0x68b12c0?, {0xc000508000?, 0x64183b0?})
	go/build/build.go:211 +0x63 fp=0xc00003fb98 sp=0xc00003fb68 pc=0x5efd603
go/build.(*Context).matchFile(0x68b12c0, {0xc00016e880, 0x79}, {0xc0001151ea, 0xe}, 0xc000156c00, 0xc0000bbbe8, 0xc00014af40)
	go/build/build.go:1466 +0x4b6 fp=0xc00003fc40 sp=0xc00003fb98 pc=0x5f06656
go/build.(*Context).Import(0x68b12c0, {0xc0000bd007, 0x3}, {0xc0000d0cc0, 0xb1}, 0x4)
	go/build/build.go:877 +0x1585 fp=0xc000040340 sp=0xc00003fc40 pc=0x5effa25
cmd/go/internal/load.loadPackageData.func2()
	cmd/go/internal/load/pkg.go:992 +0x19e fp=0xc000040538 sp=0xc000040340 pc=0x6205f3e
cmd/go/internal/par.(*ErrCache[...]).Do.func1()
	cmd/go/internal/par/work.go:119 +0x13 fp=0xc000040548 sp=0xc000040538 pc=0x61695f3
cmd/go/internal/par.(*Cache[...]).Do(0x6532ba0, {0xc0000bd007, 0x3}, 0xc0000405d0)
	cmd/go/internal/par/work.go:160 +0xfb fp=0xc0000405a8 sp=0xc000040548 pc=0x612cf3b
cmd/go/internal/par.(*ErrCache[...]).Do(0x3?, {0xc0000bd007?, 0xb2?}, 0xc0000d0cc0?)
	cmd/go/internal/par/work.go:118 +0x47 fp=0xc0000405f8 sp=0xc0000405a8 pc=0x612cd07
cmd/go/internal/load.loadPackageData({0x652dd90, 0x6912da0}, {0xc0000bd007, 0x3}, {0xc0000d0b40, 0xb2}, {0xc0000d0cc0, 0xb1}, {0x0, 0x0}, ...)
	cmd/go/internal/load/pkg.go:920 +0x4f1 fp=0xc000040848 sp=0xc0000405f8 pc=0x6205bf1
cmd/go/internal/load.loadImport({0x652dd90, 0x6912da0}, {0x0, 0x0, 0x0, 0x1, 0x1, 0x0}, 0x0, {0xc0000bd007, ...}, ...)
	cmd/go/internal/load/pkg.go:741 +0x19d fp=0xc000040a38 sp=0xc000040848 pc=0x620433d
cmd/go/internal/load.LoadImport(...)
	cmd/go/internal/load/pkg.go:714
cmd/go/internal/load.(*Package).load(0xc00016c008, {0x652dd90, 0x6912da0}, {0x0, 0x0, 0x0, 0x1, 0x1, 0x0}, {0x64183b0, ...}, ...)
	cmd/go/internal/load/pkg.go:2017 +0x1a93 fp=0xc000040db8 sp=0xc000040a38 pc=0x620bff3
cmd/go/internal/load.loadImport({0x652dd90, 0x6912da0}, {0x0, 0x0, 0x0, 0x1, 0x1, 0x0}, 0xc0000ae830, {0x64183b0, ...}, ...)
	cmd/go/internal/load/pkg.go:794 +0x546 fp=0xc000040fa8 sp=0xc000040db8 pc=0x62046e6
cmd/go/internal/load.PackagesAndErrors({0x652dd90?, 0x6912da0?}, {0x0, 0x0, 0x0, 0x1, 0x1, 0x0}, {0xc0000d2060, 0x1, ...})
	cmd/go/internal/load/pkg.go:2882 +0xa2b fp=0xc000041248 sp=0xc000040fa8 pc=0x62140cb
cmd/go/internal/list.runList({0x652dd90, 0x6912da0}, 0x68a4900?, {0xc0000d2060, 0x1, 0x1})
	cmd/go/internal/list/list.go:613 +0xc76 fp=0xc000041d30 sp=0xc000041248 pc=0x62d9256
main.invoke(0x68a4900, {0xc0000d2010, 0x6, 0x6})
	cmd/go/main.go:257 +0x5c4 fp=0xc000041e58 sp=0xc000041d30 pc=0x630cbc4
main.main()
	cmd/go/main.go:175 +0x6d5 fp=0xc000041f50 sp=0xc000041e58 pc=0x630c315
runtime.main()
	runtime/proc.go:271 +0x29d fp=0xc000041fe0 sp=0xc000041f50 pc=0x5db5d5d
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000041fe8 sp=0xc000041fe0 pc=0x5dedd21

goroutine 2 gp=0xc000006700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc000068fa8 sp=0xc000068f88 pc=0x5db618e
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.forcegchelper()
	runtime/proc.go:326 +0xb3 fp=0xc000068fe0 sp=0xc000068fa8 pc=0x5db6013
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000068fe8 sp=0xc000068fe0 pc=0x5dedd21
created by runtime.init.6 in goroutine 1
	runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000006c40 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:402 +0xce fp=0xc000069780 sp=0xc000069760 pc=0x5db618e
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.bgsweep(0xc00002a150)
	runtime/mgcsweep.go:278 +0x94 fp=0xc0000697c8 sp=0xc000069780 pc=0x5da2034
runtime.gcenable.gowrap1()
	runtime/mgc.go:203 +0x25 fp=0xc0000697e0 sp=0xc0000697c8 pc=0x5d96985
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc0000697e8 sp=0xc0000697e0 pc=0x5dedd21
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000006e00 m=nil [GC scavenge wait]:
runtime.gopark(0xc00002a150?, 0x6418348?, 0x1?, 0x0?, 0xc000006e00?)
	runtime/proc.go:402 +0xce fp=0xc000069f78 sp=0xc000069f58 pc=0x5db618e
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.(*scavengerState).park(0x68b1140)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc000069fa8 sp=0xc000069f78 pc=0x5d9fa29
runtime.bgscavenge(0xc00002a150)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc000069fc8 sp=0xc000069fa8 pc=0x5d9ffbc
runtime.gcenable.gowrap2()
	runtime/mgc.go:204 +0x25 fp=0xc000069fe0 sp=0xc000069fc8 pc=0x5d96925
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x5dedd21
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:204 +0xa5

goroutine 17 gp=0xc000094380 m=nil [finalizer wait]:
runtime.gopark(0xc000068660?, 0x5d9eefc?, 0x60?, 0x42?, 0x550011?)
	runtime/proc.go:402 +0xce fp=0xc000068620 sp=0xc000068600 pc=0x5db618e
runtime.runfinq()
	runtime/mfinal.go:194 +0x107 fp=0xc0000687e0 sp=0xc000068620 pc=0x5d959c7
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc0000687e8 sp=0xc0000687e0 pc=0x5dedd21
created by runtime.createfing in goroutine 1
	runtime/mfinal.go:164 +0x3d

goroutine 19 gp=0xc000094700 m=nil [sync.Mutex.Lock]:
runtime.gopark(0x0?, 0xc000204b68?, 0x0?, 0xa0?, 0xc000204b70?)
	runtime/proc.go:402 +0xce fp=0xc000204b28 sp=0xc000204b08 pc=0x5db618e
runtime.goparkunlock(...)
	runtime/proc.go:408
runtime.semacquire1(0xc0001569f8, 0x0, 0x3, 0x1, 0x15)
	runtime/sema.go:160 +0x225 fp=0xc000204b90 sp=0xc000204b28 pc=0x5dc8e65
sync.runtime_SemacquireMutex(0xc000204c08?, 0xe6?, 0x6488b60?)
	runtime/sema.go:77 +0x25 fp=0xc000204bc8 sp=0xc000204b90 pc=0x5de9c25
sync.(*Mutex).lockSlow(0xc0001569f4)
	sync/mutex.go:171 +0x15d fp=0xc000204c18 sp=0xc000204bc8 pc=0x5df969d
sync.(*Mutex).Lock(...)
	sync/mutex.go:90
cmd/go/internal/par.(*Cache[...]).Do(0x6532ba0, {0xc0000bd007, 0x3}, 0xc000204ca0)
	cmd/go/internal/par/work.go:158 +0xde fp=0xc000204c78 sp=0xc000204c18 pc=0x612cf1e
cmd/go/internal/par.(*ErrCache[...]).Do(0x3?, {0xc0000bd007?, 0xb2?}, 0xc0000d0cc0?)
	cmd/go/internal/par/work.go:118 +0x47 fp=0xc000204cc8 sp=0xc000204c78 pc=0x612cd07
cmd/go/internal/load.loadPackageData({0x652dd90, 0x6912da0}, {0xc0000bd007, 0x3}, {0xc0000d0b40, 0xb2}, {0xc0000d0cc0, 0xb1}, {0x0, 0x0}, ...)
	cmd/go/internal/load/pkg.go:920 +0x4f1 fp=0xc000204f18 sp=0xc000204cc8 pc=0x6205bf1
cmd/go/internal/load.(*preload).preloadImports.func1({0xc0000bd007?, 0x0?})
	cmd/go/internal/load/pkg.go:1115 +0x8b fp=0xc000204fc0 sp=0xc000204f18 pc=0x620736b
cmd/go/internal/load.(*preload).preloadImports.gowrap1()
	cmd/go/internal/load/pkg.go:1120 +0x28 fp=0xc000204fe0 sp=0xc000204fc0 pc=0x62072a8
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc000204fe8 sp=0xc000204fe0 pc=0x5dedd21
created by cmd/go/internal/load.(*preload).preloadImports in goroutine 1
	cmd/go/internal/load/pkg.go:1114 +0x265

What did you expect to see?

Normal json output

[thanm]

Thanks for the report. For this to be actionable we'll need some way to reproduce the problem -- could you please share which go repo or package you are running this on. You mention that this problem is intermittent/flaky, does it happen on other machines?

[podtserkovskiy]

could you please share which go repo or package you are running this on.

Unfortunately I can't share a specific example at the moment, because this code is not open sourced.

You mention that this problem is intermittent/flaky, does it happen on other machines?

Yes, I've seen it on several different machines.

I can keep looking into our build logs to find a pattern and create some example.

[podtserkovskiy]

Repro code: https://github.com/podtserkovskiy/repro-golang-go-issue-67113
Repro command:

CGO_ENABLED=0 GO111MODULE=off GOARCH=amd64 GOEXPERIMENT=nocoverageredesign GOOS=darwin go list -e '-json=GoFiles,CgoFiles,HFiles,CFiles,CXXFiles,SFiles,EmbedFiles,CgoCFLAGS,CgoCPPFLAGS' -tags .

GOHOSTARCH='amd64'
GOHOSTOS='darwin'

I guess this may be kind of a race condition issue, but it seems like src/make.bash doesn't offer an option to build go command with -race.