Untangle genproto / cloud.google.com/go dependencies

[liggitt]

#113225 uncovered a coming snarl of dependencies between cloud.google.com/go and google.golang.org/genproto

Because several generic libraries use the cloud.google.com/go/compute/metadata package, the following chain makes it almost impossible to avoid adding all cloud.google.com/go submodules into the dependency graph for all downstream consumers:

• {golang.org/x/oauth2, google.golang.org/api, google.golang.org/grpc} --> cloud.google.com/go/compute --> cloud.google.com/go --> google.golang.org/genproto --> cloud.google.com/go/*

It's possible we can avoid the updates that expanded these cross-references in the short-term with judicious upstream changes back to older genproto versions, but we should get ahead of this if we want to avoid being forced to pull in these dependency graph increases in the future for a critical bugfix.

Recommendations

1. Isolate cloud.google.com/go/compute/metadata

The cloud.google.com/go/compute/metadata package (https://github.com/googleapis/google-cloud-go/tree/main/compute/metadata) only uses stdlib dependencies, but lives inside cloud.google.com/go/compute, which has many other references (https://github.com/googleapis/google-cloud-go/blob/main/compute/go.mod)

1. Isolate cloud.google.com/go/compute/metadata to its own zero-dependency module

   • compute/metadata: make its own module googleapis/google-cloud-go#6311 / https://pkg.go.dev/cloud.google.com/go/compute/metadata
   • drop backreference to cloud.google.com/go/compute from https://github.com/googleapis/google-cloud-go/blob/main/compute/metadata/go.mod#L5 to reach zero-dependencies - chore: re-drop weak refs to parent modules and tag googleapis/google-cloud-go#9545
   • tag a release of cloud.google.com/go/compute/metadata

2. Sweep dependencies using cloud.google.com/go/compute/metadata and update them to use the zero-dependency alternative

   • golang.org/x/oauth2
     • committed
     • tagged
   • google.golang.org/api
     • committed
     • tagged
   • google.golang.org/grpc
     • committed
     • tagged
   • github.com/google/cadvisor (transitive via golang.org/x/oauth2, google.golang.org/api, google.golang.org/grpc)
     • committed
     • tagged
   • go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc (transitive via golang.org/x/oauth2, google.golang.org/grpc)
     • committed
     • tagged

2. Prune use of cloud genproto packages in non-cloud.google.com modules

1. Update github.com/googleapis/gax-go/v2 test to not require the google.golang.org/genproto/googleapis/cloud/location package
   • committed - chore(v2): switch ProtoJSONStream test to use non-cloud API googleapis/gax-go#237
   • tagged - https://github.com/googleapis/gax-go/releases/tag/v2.7.0
   • Use a non-cloud type for nested unit test googleapis/gax-go#270
   • tag release (>=v2.9.1)

3. Drop use of google.golang.org/genproto/protobuf/field_mask

Update to use canonical google.golang.org/protobuf/types/known/fieldmaskpb package instead

• google.golang.org/genproto
  • Eliminate use of alias field_mask package googleapis/go-genproto#1008
• cloud.google.com/go
  • chore(all): Use fieldmask types directly instead of field_mask genproto alias googleapis/google-cloud-go#7962
• github.com/grpc-ecosystem/grpc-gateway/v2
  • Switch from aliased field_mask package to canonical package grpc-ecosystem/grpc-gateway#3317
• github.com/grpc-ecosystem/grpc-gateway
  • Mark grpc-gateway v1 as an unwanted dependency #118119
  • etcd depends on two different versions of the same package - grpc-gateway etcd-io/etcd#14499 (used by go.etcd.io/etcd/api/v3 / go.etcd.io/etcd/server/v3)
    • Upgrade grpc-gateway from v1 to v2 etcd-io/etcd#16595
    • etcd client release (3.6)
    • update kubernetes to use etcd client 3.6+

4. Isolate generic genproto packages

• make the types in the following generic packages available without dragging in all of cloud.google.com/go

  • google.golang.org/genproto/googleapis/api/...
  • google.golang.org/genproto/googleapis/bytestream
  • google.golang.org/genproto/googleapis/rpc/...
  • Submodules created in Isolate submodules, take 2 googleapis/go-genproto#1012
  • Backreferences dropped in chore(all): Drop submodule backreferences googleapis/go-genproto#1087, chore(all): Update to google.golang.org/genproto/googleapis/rpc without backreference googleapis/go-genproto#1093

• Update the following dependencies to use the isolated generic rpc/api packages:

  • github.com/containerd/ttrpc
  • github.com/envoyproxy/go-control-plane
  • github.com/google/cel-go
  • github.com/google/gnostic
  • github.com/googleapis/gax-go/v2
  • github.com/grpc-ecosystem/go-grpc-middleware
  • github.com/grpc-ecosystem/grpc-gateway/v2
  • github.com/grpc-ecosystem/grpc-gateway
  • go.etcd.io/etcd/api/v3
  • go.etcd.io/etcd/server/v3
  • go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
  • go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
  • go.opentelemetry.io/otel/exporters/otlp/otlptrace
  • google.golang.org/api
  • google.golang.org/grpc

/priority important-longterm
/area code-organization
/sig architecture
/cc @dims
fyi @codyoss @bertinatto

[codyoss]

I am in the process of doing item one rn: googleapis/google-cloud-go#6311

[codyoss]

For item three I might say this can be simplified to the following:

• google.golang.org/genproto/googleapis/api/*
• google.golang.org/genproto/googleapis/rpc/*
• google.golang.org/genproto/googleapis/type/* (new entry)
• google.golang.org/genproto/googleapis/bytestream

google.golang.org/genproto/protobuf/field_mask is already an alias to another project that should be used directly.
It may be easiest to create a new module somewhere to house all of these super general stubs. Then genproto just alias to that repo.

[codyoss]

cc @noahdietz @quartzmo

[liggitt]

opened googleapis/gax-go#237 for item 2

[aramase]

opened google/cadvisor#3194 for github.com/google/cadvisor . I've marked it as WIP so we can update the transitive deps in 1 PR.