new panic in after upgrading from google.golang.org/grpc from 1.55.0 to 1.56.0

[ricebin]

What version of gRPC are you using?

grpc-1.56.0

What version of Go are you using (go version)?

1.20

What operating system (Linux, Windows, …) and version?

linux

What did you do?

If possible, provide a recipe for reproducing the error.

unable to repo consistently
only occur for some requests

What did you expect to see?

What did you see instead?

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8cd8ef]

goroutine 118 [running]:
google.golang.org/grpc.(*addrConn).updateAddrs(0xc0006fcdc0, {0xc001177c20?, 0x2, 0x2})
/workspace/vendor/google.golang.org/grpc/clientconn.go:1036 +0x2ef
google.golang.org/grpc.(*acBalancerWrapper).UpdateAddresses(...)
/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:394
google.golang.org/grpc.(*ccBalancerWrapper).UpdateAddresses(0xc0009332c0?, {0x1a65f90?, 0xc0005c27e0}, {0xc001177c20, 0x2, 0x2})
/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:351 +0x6d
google.golang.org/grpc/internal/balancer/gracefulswitch.(*balancerWrapper).UpdateAddresses(0xc0006674c0, {0x1a65f90, 0xc0005c27e0}, {0xc001177c20, 0x2, 0x2})
/workspace/vendor/google.golang.org/grpc/internal/balancer/gracefulswitch/gracefulswitch.go:379 +0xef
google.golang.org/grpc/balancer/grpclb.(*lbBalancer).refreshSubConns(0xc000638240, {0xc001177c20?, 0x2, 0x2}, 0x20?, 0x3d?)
/workspace/vendor/google.golang.org/grpc/balancer/grpclb/grpclb_remote_balancer.go:151 +0x38e
google.golang.org/grpc/balancer/grpclb.(*lbBalancer).UpdateClientConnState(0xc000638240, {{{0xc001177b90, 0x2, 0x2}, 0x0, 0x0}, {0x1a556c0, 0xc000842780}})
/workspace/vendor/google.golang.org/grpc/balancer/grpclb/grpclb.go:501 +0x7d5
google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState(0xc000cd1f44?, {{{0xc001177b90, 0x2, 0x2}, 0x0, 0x0}, {0x1a556c0, 0xc000842780}})
/workspace/vendor/google.golang.org/grpc/internal/balancer/gracefulswitch/gracefulswitch.go:164 +0x82
google.golang.org/grpc.(*ccBalancerWrapper).updateClientConnState.func1({0xc000842870?, 0xc000cd1f40?})
/workspace/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:116 +0xfa
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00083c6c0, {0x1a676d0, 0xc000853630})
/workspace/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:92 +0x194

[arhea]

We are seeing this as well using the Google Go SDK. After upgrading to 1.56 we are seeing this panics intermittently. Adding replace to our go.mod fixes the issue.

replace google.golang.org/grpc => google.golang.org/grpc v1.55.0

[atollena]

#6354 fixes this issue and has been merged, so presumably it is present in all future releases, including 1.56.1 if there is one. It would help to know which combination of balancer and resolver you're using, the PR description mentions fixing grpclb that has a behavior that "may not be proper" but I'd be curious if it can happen with other combinations of resolvers/balancers.

Note that there's another source of panic, also triggered in google go sdk, since 1.55. See #6373.

[philiplinell]

We are also seeing the same error. This is the direct dependencies (minus some private ones)

require (
	cloud.google.com/go/bigtable v1.18.1
	cloud.google.com/go/pubsub v1.31.0
	github.com/davecgh/go-spew v1.1.1
	github.com/gofiber/adaptor/v2 v2.2.1
	github.com/gofiber/fiber/v2 v2.46.0
	github.com/google/go-cmp v0.5.9
	github.com/google/uuid v1.3.0
	github.com/stretchr/testify v1.8.4
	go.opentelemetry.io/otel v1.16.0
	go.opentelemetry.io/otel/sdk v1.16.0
	go.opentelemetry.io/otel/trace v1.16.0
	golang.org/x/exp v0.0.0-20221002003631-540bb7301a08
	golang.org/x/net v0.11.0
	google.golang.org/api v0.128.0
	google.golang.org/grpc v1.56.0
	google.golang.org/protobuf v1.30.0
)

We have reverted back to 1.55.0 for the time being, that solves the issue.

We are running our services in Google Cloud Run.

[zasweq]

Hello all, sorry about this. We introduced a bug that we fixed, but forgot to backport to 1.56.0 release. Yesterday, I released a patch release containing fix (1.56.1): https://github.com/grpc/grpc-go/releases. Upgrading to this release should fix this particular panic :).