Rework grpc cancelation propagation #8957
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #8923
When running with the agent we switch grpc context storage to use opentelemetry context. As described in #4169 (comment) this strategy works too well, context propagation sometimes happens when originally context was not propagated. This has the side effect of also propagating deadlines to calls that previously didn't have them. This is corrected in #5543 Unfortunately this has the side effect that now deadlines and cancellations aren't sometimes propagated when they are with the original context store implementation. #8924 introduces a flag that allows users to decide whether cancelation and deadlines should propagate. Such flags aren't ideal because users need to figure out what works for them. This pr explores an alternative strategy where, instead of completely replacing the original context storage, we'll track propagation through both otel context and the original context storage. When we discover that context was propagated through otel, but not with the original context storage, we'll fork the grpc context to break cancellation propagation. Hopefully this way deadlines and cancellation are propagated only when they are propagated without the agent. This pr can coexist with the flag introduced in #8924