Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix multi-reader observable counter double-counting bug #4742

Merged
merged 3 commits into from Dec 8, 2023

Conversation

dashpole
Copy link
Contributor

@dashpole dashpole commented Dec 4, 2023

Fixes #4741

Using multiple readers causes TestObservableExample/Cumulative and TestObservableExample/Delta to fail without this fix.

The issue is that the "observable" added to the callback for all pipelines includes instruments from all readers. So when callbacks are invoked during collect for one reader, it "observes" all of the instruments for all readers.

For observable instruments, instead of using lookup to fetch all instruments (for all readers), and then using registerCallbacks to register them will all pipelines, it now iterates through each pipeline and fetches only instruments for that pipeline (inserter), and registers callbacks on those instruments with the pipeline.

@dashpole dashpole changed the title Reproduce #4741 in a unit test Fix multi-reader observable counter double-counting bug Dec 6, 2023
Copy link

codecov bot commented Dec 6, 2023

Codecov Report

Merging #4742 (c3d7689) into main (214d5e0) will increase coverage by 0.0%.
The diff coverage is 93.2%.

Additional details and impacted files

Impacted file tree graph

@@          Coverage Diff          @@
##            main   #4742   +/-   ##
=====================================
  Coverage   82.0%   82.0%           
=====================================
  Files        225     225           
  Lines      18239   18242    +3     
=====================================
+ Hits       14961   14976   +15     
+ Misses      2989    2981    -8     
+ Partials     289     285    -4     
Files Coverage Δ
sdk/metric/instrument.go 93.5% <100.0%> (+0.2%) ⬆️
sdk/metric/pipeline.go 86.2% <100.0%> (-0.2%) ⬇️
sdk/metric/meter.go 92.5% <91.8%> (+3.5%) ⬆️

@dashpole dashpole force-pushed the fix_multireader branch 2 times, most recently from d582bf2 to 6c4286a Compare December 6, 2023 21:29
sdk/metric/meter.go Outdated Show resolved Hide resolved
sdk/metric/meter.go Outdated Show resolved Hide resolved
sdk/metric/meter_test.go Show resolved Hide resolved
Copy link

@StarpTech StarpTech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dashpole dashpole added pkg:SDK Related to an SDK package area:metrics Part of OpenTelemetry Metrics labels Dec 8, 2023
@MrAlias MrAlias merged commit b5afa70 into open-telemetry:main Dec 8, 2023
25 checks passed
@StarpTech
Copy link

@MrAlias when can we expect a release?

@dashpole dashpole deleted the fix_multireader branch December 12, 2023 19:08
@MrAlias MrAlias added this to the v1.22.0 milestone Jan 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:metrics Part of OpenTelemetry Metrics bug Something isn't working pkg:SDK Related to an SDK package
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using multiple concurrent readers makes async metrics produce wrong results
4 participants