feat: OpenTelemetry trace/spanID integration for Python handlers #889

gkevinzheng · 2024-04-30T19:57:08Z

Changes made:

Added opentelemetry-api as a dependency for the library.
Changed trace/http gathering function from get_request_data to get_request_and_trace_data
get_request_and_trace_data extracts and returns OpenTelemetry span context information, if a valid span exists.
Added unit tests for get_request_and_trace_data, as well as for both CloudLoggingHandler and StructuredLogHandler
Added a system test for Open Telemetry integration using the SDK
Added opentelemetry-sdk as a system test external dependency.

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…pis/python-logging into otel-span-support-python

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

…pis/python-logging into otel-span-support-python

cindy-peng · 2024-05-02T12:44:23Z

tests/system/test_system.py

+            cloud_logger.warning(LOG_MESSAGE)
+
+            entries = _list_entries(logger)
+            self.assertEqual(len(entries), 1)


Do we need to consider instrumentation source entry here? http://go/cdpe-ops-logentry-changes-source

Whether or not the instrumentation source entry gets added depends on whether or not the global variable google.cloud.logging_v2._instrumentation_emitted is true (see

python-logging/google/cloud/logging_v2/__init__.py

Line 40 in 6264107

_instrumentation_emitted = False

). When I was running the system tests locally I found that if I ran the entire test suite this case would pass, but if I ran just this test case the instrumentation source entry would be there and it would fail.

cindy-peng · 2024-05-02T12:47:37Z

google/cloud/logging_v2/handlers/_helpers.py

+    ) = get_request_data()
+
+    # otel_trace_id existing means the other return values are non-null
+    if otel_trace_id:


If there is no http request data, do we reuse the http_request from last request for otel?

If there is no http request data I have it set to null value.

aabmass

Didn't look too close at tests but LGTM from an OTel perspective

aabmass · 2024-05-07T19:30:20Z

setup.py

@@ -44,6 +44,7 @@
    "google-cloud-audit-log >= 0.1.0, < 1.0.0dev",
    "google-cloud-core >= 2.0.0, <3.0.0dev",
    "grpc-google-iam-v1 >=0.12.4, <1.0.0dev",
+    "opentelemetry-api >= 1.22.0",


I would consider using an older version here. The context APIs you are I believe were available in even the first 1.x release

aabmass · 2024-05-07T20:17:53Z

google/cloud/logging_v2/handlers/_helpers.py

+def get_request_and_trace_data():
+    """Helper to get http_request and trace data from supported web
+    frameworks (currently supported: Flask and Django), as well as OpenTelemetry. Attempts
+    to parse trace/spanID from OpenTelemetry first, before going to Traceparent then XCTC.


nit this is a bit misleading

Suggested change

to parse trace/spanID from OpenTelemetry first, before going to Traceparent then XCTC.

to retrieve trace/spanID from OpenTelemetry first, before going to Traceparent then XCTC.

aabmass · 2024-05-07T20:20:44Z

google/cloud/logging_v2/handlers/_helpers.py

@@ -191,9 +193,31 @@ def _parse_xcloud_trace(header):
    return trace_id, span_id, trace_sampled


+def _parse_current_open_telemetry_span():


nit i wouldn't use "parse" here

aabmass · 2024-05-07T20:28:43Z

tests/system/test_system.py

+        processor = BatchSpanProcessor(ConsoleSpanExporter())
+        provider.add_span_processor(processor)


You can omit these lines if you don't actually want any console output

aabmass · 2024-05-07T20:29:05Z

tests/system/test_system.py

+        processor = BatchSpanProcessor(ConsoleSpanExporter())
+        provider.add_span_processor(processor)
+
+        tracer = trace.get_tracer("test_system", tracer_provider=provider)


nit this is a bit simpler

Suggested change

tracer = trace.get_tracer("test_system", tracer_provider=provider)

tracer = provider.get_tracer("test_system")

aabmass · 2024-05-07T20:38:21Z

tests/unit/handlers/__init__.py

+    with mock.patch("opentelemetry.trace.get_current_span", return_value=span) as m:
+        yield m


You don't actually need to do any mocking by just setting the span in the real context implementation, see this example https://opentelemetry.io/docs/languages/python/cookbook/#manually-setting-span-context:~:text=%23%20Or%20you%20can,(token)

Suggested change

with mock.patch("opentelemetry.trace.get_current_span", return_value=span) as m:

yield m

ctx = trace.set_span_in_context(span)

token = context.attach(ctx)

try:

yield

finally:

context.detach(token)

I am guessing that's why you are using import opentelemetry.trace instead of from opentelemetry import trace

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

daniel-sanche · 2024-05-09T18:29:37Z

google/cloud/logging_v2/handlers/_helpers.py

@@ -211,3 +235,37 @@ def get_request_data():
            return http_request, trace_id, span_id, trace_sampled

    return None, None, None, False
+
+
+def get_request_and_trace_data():


Does this need to be a new function? The docstrings of get_request_data says "Helper to get http_request and trace data from supported web frameworks". It seems to me like this logic should just be merged into the existing one

I was concerned that by changing the function name that it could potentially be a breaking change. It should be OK to change the function name because it's in a private module, right?

Yes, the module is private, so it should be safe

Does the function need to be renamed though? Why not add new functionality to the existing function?

daniel-sanche · 2024-05-09T18:36:38Z

tests/system/test_system.py

@@ -662,6 +665,38 @@ def test_log_root_handler(self):
        self.assertEqual(len(entries), 1)
        self.assertEqual(entries[0].payload, expected_payload)

+    def test_log_handler_otel_integration(self):


Is it possible to also add a system test without the otel sdk imported?

I'm not exactly sure what that should look like, but I want to make sure we have coverage of the situation where otel isn't used at all

Is it possible to create another file for the newly added system tests, so that the existing TCs aren't importing otel?

Yeah, you can add an extra file. Or just import within the test functions. Or use a test class. Whatever seems cleanest

Does Otel behave any differently based on having the module installed? Or is import state the important part?

I actually managed to resolve this by creating a decorator that deletes the otel SDK imports after the test case gets run. I don't think it's perfect but it feels good enough.

Ok great! Do you have any test cases that test _retrieve_current_open_telemetry_span without otel included?

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

daniel-sanche

Consider moving the logic in get_request_and_trace_data into get_request_data. I dont think a new function is required

Other than that, LGTM

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

feat: OpenTelemetry trace/spanID integration for Python handlers

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

GPG key ID: B5690EEEBB952194

Verified
Learn about vigilant mode

Loading
Loading status checks…

c673fae

gkevinzheng requested review from a team as code owners April 30, 2024 19:57

gkevinzheng requested a review from daniel-sanche April 30, 2024 19:57

product-auto-label bot added the size: m label Apr 30, 2024

blunderbuss-gcf bot assigned daniel-sanche Apr 30, 2024

product-auto-label bot added the api: logging label Apr 30, 2024

gkevinzheng and others added 3 commits April 30, 2024 15:57

Merge branch 'main' into otel-span-support-python

Loading
Loading status checks…

9958630

🦉 Updates from OwlBot post-processor

48c21b6

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'otel-span-support-python' of https://github.com/googlea…

Loading
Loading status checks…

12c9884

…pis/python-logging into otel-span-support-python

product-auto-label bot added size: l and removed size: m labels Apr 30, 2024

gcf-owl-bot bot added 2 commits April 30, 2024 20:00

🦉 Updates from OwlBot post-processor

5869017

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Merge branch 'otel-span-support-python' of https://github.com/googlea…

Loading
Loading status checks…

2df699e

…pis/python-logging into otel-span-support-python

gkevinzheng marked this pull request as draft April 30, 2024 20:47

gkevinzheng added 3 commits May 1, 2024 18:27

Added more tests for OTel Python integration

Loading
Loading status checks…

ba90fd0

linting

Loading
Loading status checks…

4793848

more linting

Loading
Loading status checks…

bfc84dc

gkevinzheng marked this pull request as ready for review May 1, 2024 20:04

cindy-peng reviewed May 2, 2024

View reviewed changes

cindy-peng approved these changes May 7, 2024

View reviewed changes

aabmass approved these changes May 7, 2024

View reviewed changes

gkevinzheng and others added 3 commits May 8, 2024 19:37

renamed _parse_current_open_telemetry_span and fixed otel testcases

Loading
Loading status checks…

4189108

🦉 Updates from OwlBot post-processor

Loading
Loading status checks…

64766ed

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

linting + removed print statements

Loading
Loading status checks…

196749d

daniel-sanche reviewed May 9, 2024

View reviewed changes

gkevinzheng and others added 2 commits May 10, 2024 20:50

added opentelemetry sdk module cleanup to system test

Loading
Loading status checks…

3dab0c3

🦉 Updates from OwlBot post-processor

Loading
Loading status checks…

785c837

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

daniel-sanche approved these changes May 15, 2024

View reviewed changes

gkevinzheng and others added 2 commits May 16, 2024 15:01

Refactored get_request_and_trace_data back into get_request_data

Loading
Loading status checks…

ff22f7f

🦉 Updates from OwlBot post-processor

Loading
Loading status checks…

8ae8493

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

gkevinzheng added the kokoro:force-run label May 21, 2024

yoshi-kokoro removed the kokoro:force-run label May 21, 2024

Merge branch 'main' into otel-span-support-python

Loading
Loading status checks…

7a99cbd

gkevinzheng enabled auto-merge (squash) May 22, 2024 15:27

gkevinzheng merged commit 78168a3 into main May 22, 2024
17 checks passed

gkevinzheng deleted the otel-span-support-python branch May 22, 2024 15:58

release-please bot mentioned this pull request May 22, 2024

chore(main): release 3.11.0 #876

Merged

aradkdj mentioned this pull request Sep 19, 2024

feat: support OpenTelemetry's LoggingInstrumentor #696

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: OpenTelemetry trace/spanID integration for Python handlers #889

feat: OpenTelemetry trace/spanID integration for Python handlers #889

gkevinzheng commented Apr 30, 2024 •

edited

Loading

cindy-peng May 2, 2024

gkevinzheng May 2, 2024

cindy-peng May 2, 2024

gkevinzheng May 2, 2024

aabmass left a comment

aabmass May 7, 2024

aabmass May 7, 2024

aabmass May 7, 2024

aabmass May 7, 2024

aabmass May 7, 2024

aabmass May 7, 2024

daniel-sanche May 9, 2024

gkevinzheng May 9, 2024

daniel-sanche May 13, 2024

daniel-sanche May 9, 2024

gkevinzheng May 9, 2024

daniel-sanche May 13, 2024

gkevinzheng May 14, 2024

daniel-sanche May 14, 2024

daniel-sanche left a comment

	to parse trace/spanID from OpenTelemetry first, before going to Traceparent then XCTC.
	to retrieve trace/spanID from OpenTelemetry first, before going to Traceparent then XCTC.

		@@ -191,9 +193,31 @@ def _parse_xcloud_trace(header):
		return trace_id, span_id, trace_sampled


		def _parse_current_open_telemetry_span():

		processor = BatchSpanProcessor(ConsoleSpanExporter())
		provider.add_span_processor(processor)

	tracer = trace.get_tracer("test_system", tracer_provider=provider)
	tracer = provider.get_tracer("test_system")

		with mock.patch("opentelemetry.trace.get_current_span", return_value=span) as m:
		yield m

-    with mock.patch("opentelemetry.trace.get_current_span", return_value=span) as m:
-        yield m
+    ctx = trace.set_span_in_context(span)
+    token = context.attach(ctx)
+    try:
+        yield
+    finally:
+        context.detach(token)

feat: OpenTelemetry trace/spanID integration for Python handlers #889

feat: OpenTelemetry trace/spanID integration for Python handlers #889

Conversation

gkevinzheng commented Apr 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aabmass left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-sanche left a comment

Choose a reason for hiding this comment

gkevinzheng commented Apr 30, 2024 •

edited

Loading