Project: Metrics Rate Limits

[stephanie-anderson]

Description

In order to protect our infrastructure and also customers money, Relay is now able to propagate the limits from processing to PoP Relays and further to our SDKs. We need to make sure that all SDKs can stop sending metrics for some orgs (e.g. above quota, abusing Sentry).

The develop specs for this can be found here:
https://develop.sentry.dev/sdk/rate-limiting/

The tracking issue for the Ingest team is:
https://github.com/getsentry/team-ingest/issues/303

A quick TL;DR: How to Implement

Add a new data category to the existing rate limiter code, called metric_bucket.
A new parameter called namespaces might be returned for the metric_bucket category.
If the returned namespaces contain custom, back off transmitting metrics from the SDK.
If the returned namespaces exclusively contain namespaces other than custom, do not back off transmitting metrics from the SDK.
If the namespaces are empty (2700:metric_bucket:organization:quota_exceeded:) or omitted (2700:metric_bucket:organization:quota_exceeded) in the response header, back off transmitting metrics from the SDK.
While we do not support setting metric namespaces in the SDK yet, Relay could return a X-Sentry-Rate-Limits header that contains a different metric namespace only, e.g. transaction.
In case your SDK supports client outcomes, report rate limited metrics as the metric_bucket category.

Get inspired by Python SDK PR #1 or PHP PR #1 and PHP PR #2.

Why should we be doing this?

We need to protect our infrastructure and respect communicated rate limits.

Why now?

We didn't plan this for the metrics beta, but metrics is adopted pretty well.

Slack-Channel

#discuss-ingest

Stakeholder(s)

Ingest team (esp. Oleksandr & Jan)

Team(s)

Mobile, Web Frontend, Web Backend

Implementation Status

Web Backend SDKs

Beta Give feedback

1. Metrics Rate Limits sentry-php#1723
   Type: Feature +1
   
2. Metrics Rate Limits sentry-python#2921
   +0
3. Metrics Rate Limits sentry-java#3292
   Platform: Android Platform: Java +2
   
4. Metrics Rate Limits sentry-ruby#2282
   metrics +1
   
5. Metrics Rate Limits sentry-dotnet#3248
   +0
   
6. Unity: should get it via Metrics Rate Limits sentry-dotnet#3248
7. Unreal doesn't support Metrics yet
8. Elixir doesn't support Metrics yet
9. Go doesn't support Metrics yet

Web Frontend SDKs

Beta Give feedback

1. Metrics Rate Limits sentry-javascript#11336
   Feature: Metrics +1
   

Mobile SDKs

Beta Give feedback

1. Android: should get it via Metrics Rate Limits sentry-java#3292
2. Metrics Rate Limits sentry-dart#1957
   Platform: Dart +1
3. Metrics Rate Limits sentry-cocoa#3805
   Platform: Cocoa Platform: iOS +2
   
4. React Native https://github.com/getsentry/sentry-react-native/releases/tag/5.22.0

[philipphofmann]

A new parameter called namespaces might be returned for the metric_bucket category.
If the returned namespaces contain custom, back off transmitting metrics from the SDK.
If the namespace is omitted in the response header, back off transmitting metrics from the SDK.

I had to double-check the code to ensure I got this right. @cleptric, please correct me if I'm wrong. When the SDK receives a rate limits header with the category metric_bucket and a namespace other than custom e.g. transaction, the SDK must not apply a rate limit for metrics.

[cleptric]

If the returned namespaces exclusively contain namespaces other than custom, do not back off transmitting metrics from the SDK.

Updated the description.

[krystofwoldrich]

React Native SDK https://github.com/getsentry/sentry-react-native/releases/tag/5.22.0 includes Rate Limits impl from sentry-cocoa and sentry-android.