feat(celery): Send queue name to Sentry #2984

szokeasaurusrex · 2024-04-16T16:05:51Z

Send the queue name to Sentry for Celery tasks using the default exchange. The queue name is sent as span data with the key messaging.destination.name.

Closes GH-2961

Struggling to get this test to work due to complex test setup

szokeasaurusrex · 2024-04-17T12:41:18Z

Due to the current way we test Celery, it appears there is no straightforward way to test this change in our test suite. The problem is that our tests run with Celery's eager mode, which means the tasks are run synchronously in the same process, without any queueing. Since there is no queue, we never set messaging.destination.name.

We should improve our Celery tests, and add a test for this change, in a separate PR.

szokeasaurusrex · 2024-04-17T12:42:46Z

sentry_sdk/integrations/celery/__init__.py

    def _inner(*args, **kwargs):
        # type: (*Any, **Any) -> Any
        try:
-            return f(*args, **kwargs)
+            with sentry_sdk.start_span(
+                op=OP.QUEUE_TASK_CELERY, description=task.name


Is there a better op we could use here? queue.task.celery seemed like the most fitting op; however, since the parent transaction also has op set to queue.task.celery, this leads to the span having the same op as its parent, which could be somewhat confusing

Yes, could be confusing. Lets see what @cleptric spec on develop docs will look like and then see if we should add a new op for this.

Will be queue.process if I get the context here right.
We'll put celery on messaging.system.

So this would be a new span op @cleptric? I do not see queue.process listed anywhere on our list of span ops.

We'll use these values, prefixed with queue.

https://opentelemetry.io/docs/specs/semconv/attributes-registry/messaging/

sentrivana · 2024-04-17T12:45:10Z

Due to the current way we test Celery, it appears there is no straightforward way to test this change in our test suite. The problem is that our tests run with Celery's eager mode, which means the tasks are run synchronously in the same process, without any queueing. Since there is no queue, we never set messaging.destination.name.

We should improve our Celery tests, and add a test for this change, in a separate PR.

Just a sidenote, but this is also basically the reason why we can't test #2377 properly at the moment. So as soon as we can run the tests in non-eager mode we also unblock at least one other feature.

antonpirker

The one assert in the tests I do not get.

And because of the "always-eager" there is no way we can tests for messaging.destination.name right? Do you have any other PR plan on how we could run tests with a queue?

tests/integrations/celery/test_celery.py

sentry_sdk/integrations/celery/__init__.py

antonpirker · 2024-04-22T12:45:51Z

sentry_sdk/integrations/celery/__init__.py

    def _inner(*args, **kwargs):
        # type: (*Any, **Any) -> Any
        try:
-            return f(*args, **kwargs)
+            with sentry_sdk.start_span(
+                op=OP.QUEUE_TASK_CELERY, description=task.name


Yes, could be confusing. Lets see what @cleptric spec on develop docs will look like and then see if we should add a new op for this.

szokeasaurusrex · 2024-04-22T13:19:47Z

And because of the "always-eager" there is no way we can tests for messaging.destination.name right? Do you have any other PR plan on how we could run tests with a queue?

I could not find an obvious way to test messaging.destination.name, since with the "always-eager" mode, the queueing information does not get set properly. We would need to find how to have a real queue in the tests. Or, perhaps, we could work around this with some mocking logic

Struggling to get this test to work due to complex test setup

szokeasaurusrex · 2024-04-25T12:51:08Z

@antonpirker Turns out that we can test for messaging.destination.name by mocking out the task request. I just added some commits to test the changes.

antonpirker · 2024-04-30T06:56:51Z

Failing tests should be fixed by this pr: #3030

Send the queue name to Sentry for Celery tasks using the default exchange. The queue name is sent as span data with the key `messaging.destination.name`. Ref GH-2961

The previous test only checked that the queue was set on the span when the queue had the default name ("celery"). This test adds a check to ensure the queue also gets set on the span when it has a non-default value.

This test checks that `messaging.destination.name` is only set for the default exchange. Other exchanges should not set this value, since the routing key may be different from the queue name.

edwardgou-sentry

worked for me in testing

szokeasaurusrex self-assigned this Apr 16, 2024

szokeasaurusrex force-pushed the szokeasaurusrex/celery-queue-consumer-default-exchange branch 2 times, most recently from b02c1eb to 759bd53 Compare April 17, 2024 08:52

szokeasaurusrex added a commit that referenced this pull request Apr 17, 2024

WIP: Attempt to test #2984

74f3988

Struggling to get this test to work due to complex test setup

szokeasaurusrex marked this pull request as ready for review April 17, 2024 12:41

szokeasaurusrex requested review from antonpirker and edwardgou-sentry April 17, 2024 12:41

szokeasaurusrex commented Apr 17, 2024

View reviewed changes

szokeasaurusrex changed the title ~~feat(celery): Send queue name to sentry_sdk~~ feat(celery): Send queue name to Sentry Apr 17, 2024

szokeasaurusrex force-pushed the szokeasaurusrex/celery-queue-consumer-default-exchange branch from 759bd53 to 0e3c823 Compare April 17, 2024 12:48

szokeasaurusrex mentioned this pull request Apr 19, 2024

Add queue or topic to Celery transactions/spans #2961

Open

antonpirker requested changes Apr 22, 2024

View reviewed changes

szokeasaurusrex requested a review from antonpirker April 22, 2024 13:19

szokeasaurusrex force-pushed the szokeasaurusrex/celery-queue-consumer-default-exchange branch 2 times, most recently from 8028917 to 5a99aad Compare April 24, 2024 13:55

Base automatically changed from sentry-sdk-2.0 to master April 25, 2024 09:13

szokeasaurusrex added a commit that referenced this pull request Apr 25, 2024

WIP: Attempt to test #2984

da438e6

Struggling to get this test to work due to complex test setup

szokeasaurusrex force-pushed the szokeasaurusrex/celery-queue-consumer-default-exchange branch from 2fea931 to 3bf4c12 Compare April 26, 2024 10:18

antonpirker approved these changes Apr 30, 2024

View reviewed changes

szokeasaurusrex added 5 commits April 30, 2024 17:16

feat(celery): Send queue name to Sentry

a70685e

Send the queue name to Sentry for Celery tasks using the default exchange. The queue name is sent as span data with the key `messaging.destination.name`. Ref GH-2961

feat(celery): Change inner span op to "queue.process"

e849ddf

test(celery): Test that celery queue set on span

1b6a927

test(celery): Add test for "custom" queue set on span

e70aa66

The previous test only checked that the queue was set on the span when the queue had the default name ("celery"). This test adds a check to ensure the queue also gets set on the span when it has a non-default value.

test(celery): Test that Celery queue only set for default exchange

67e2531

This test checks that `messaging.destination.name` is only set for the default exchange. Other exchanges should not set this value, since the routing key may be different from the queue name.

szokeasaurusrex force-pushed the szokeasaurusrex/celery-queue-consumer-default-exchange branch from f1671f5 to 67e2531 Compare April 30, 2024 15:16

This was referenced Apr 30, 2024

feat(celery): Send Celery retry count #2971

Open

Set task ID on span (Celery integration) #3015

Open

feat(celery): Set "messaging.system" on span #3024

Open

edwardgou-sentry approved these changes May 1, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(celery): Send queue name to Sentry #2984

feat(celery): Send queue name to Sentry #2984

szokeasaurusrex commented Apr 16, 2024 •

edited

szokeasaurusrex commented Apr 17, 2024

szokeasaurusrex Apr 17, 2024

antonpirker Apr 22, 2024

cleptric Apr 24, 2024

szokeasaurusrex Apr 24, 2024

cleptric Apr 24, 2024

sentrivana commented Apr 17, 2024 •

edited

antonpirker left a comment

antonpirker Apr 22, 2024

szokeasaurusrex commented Apr 22, 2024

szokeasaurusrex commented Apr 25, 2024

antonpirker commented Apr 30, 2024

edwardgou-sentry left a comment

feat(celery): Send queue name to Sentry #2984

Are you sure you want to change the base?

feat(celery): Send queue name to Sentry #2984

Conversation

szokeasaurusrex commented Apr 16, 2024 • edited

szokeasaurusrex commented Apr 17, 2024

szokeasaurusrex Apr 17, 2024

Choose a reason for hiding this comment

antonpirker Apr 22, 2024

Choose a reason for hiding this comment

cleptric Apr 24, 2024

Choose a reason for hiding this comment

szokeasaurusrex Apr 24, 2024

Choose a reason for hiding this comment

cleptric Apr 24, 2024

Choose a reason for hiding this comment

sentrivana commented Apr 17, 2024 • edited

antonpirker left a comment

Choose a reason for hiding this comment

antonpirker Apr 22, 2024

Choose a reason for hiding this comment

szokeasaurusrex commented Apr 22, 2024

szokeasaurusrex commented Apr 25, 2024

antonpirker commented Apr 30, 2024

edwardgou-sentry left a comment

Choose a reason for hiding this comment

szokeasaurusrex commented Apr 16, 2024 •

edited

sentrivana commented Apr 17, 2024 •

edited