Data race in otelgrpc.StatsHandler

[rosstimothy]

Description

WARNING: DATA RACE
Read at 0x00c007ab7e30 by goroutine 58927:
  go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*config).handleRPC()
      /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.0/stats_handler.go:198 +0x176a
  go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*serverHandler).HandleRPC()
      /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.0/stats_handler.go:88 +0x5d
  google.golang.org/grpc.(*Server).processStreamingRPC.func1()
      /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1546 +0x8f9
  runtime.deferreturn()
      /opt/go/src/runtime/panic.go:477 +0x30
  google.golang.org/grpc.(*Server).handleStream()
      /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:1741 +0x1292
  google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /go/pkg/mod/google.golang.org/grpc@v1.59.0/server.go:986 +0x11d

Previous write at 0x00c007ab7e30 by goroutine 58935:
  sync/atomic.AddInt64()
      /opt/go/src/runtime/race_amd64.s:289 +0xb
  sync/atomic.AddInt64()
      <autogenerated>:1 +0x15
  go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*serverHandler).HandleRPC()
      /go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.0/stats_handler.go:88 +0x5d
  google.golang.org/grpc.(*serverStream).RecvMsg()
      /go/pkg/mod/google.golang.org/grpc@v1.59.0/stream.go:1733 +0x5e2
  github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcServerStreamWrapper).RecvMsg()
      /__w/teleport/teleport/api/utils/grpc/interceptors/errors.go:39 +0x55

The problem seems to stem from inconsistent usage of the members of gRPCContext. Writes to the fields are performed via atomic.AddInt64, however reads of the variables do not use atomic.LoadInt64.

Environment

• OS: all
• Architecture: all
• Go Version: 1.21.4
• otelgrpc version: v0.46.0

Expected behavior

Converting from the now deprecated interceptors to the new stats handler should produce the same traces without any data races occurring.

[slonka]

We're seeing panics that might be caused by the same root problem.

[strideynet]

closed by #4577 ?

[MrAlias]

closed by #4577 ?

Assuming yes, based on lack of response.