Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Stackdriver's shutdown behavior #4358

Merged
merged 4 commits into from
Jan 31, 2024
Merged

Conversation

dimovelev
Copy link
Contributor

@dimovelev dimovelev commented Nov 17, 2023

Fixes #4353

@pivotal-cla
Copy link

@dimovelev Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

@pivotal-cla
Copy link

@dimovelev Thank you for signing the Contributor License Agreement!

@jonatan-ivanov
Copy link
Member

Thanks for the PR!
Would you mind giving us a little bit more details?
What issue you are trying to solve, how to reproduce it and why is this fixing that issue?
Also, is it possible to add tests?

@jonatan-ivanov jonatan-ivanov added the waiting for feedback We need additional information before we can continue label Nov 17, 2023
@jonatan-ivanov
Copy link
Member

jonatan-ivanov commented Nov 17, 2023

Oh, I see #4353 in the title, let me edit it with the description.

@jonatan-ivanov jonatan-ivanov removed the waiting for feedback We need additional information before we can continue label Nov 17, 2023
@jonatan-ivanov jonatan-ivanov changed the title Fix shutdown behaviour so that metrics are pushed and no exception thrown. Fixes 4353 Fix Stackdriver's shutdown behavior Nov 17, 2023
@dimovelev
Copy link
Contributor Author

@jonatan-ivanov any plans to merge that?

@mieseprem
Copy link

@jonatan-ivanov any plans to merge that?

And: does this only suppress the error message, or does it also solve the problem? Or more importantly: will metrics continue to be written ;)
I mean it's only a small fix, but I cannot estimate the impact.

@@ -138,11 +138,21 @@ public void start(ThreadFactory threadFactory) {

@Override
public void stop() {
if (client != null)
client.shutdownNow();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this from here will potentially be problematic (a resource leak) if the registry is stopped and started multiple times. If it is only stopped as part of close then I think this fix works. We should have some kind of tests for this, ideally.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • added a shutdown and close when start is called
  • I could not find any emulator etc. so that automated integration tests could be done against it. I have tested it manually against my GCP environment though.

Comment on lines 140 to 143
@Override
public void stop() {
if (client != null)
client.shutdownNow();
super.stop();
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it necessary to have an overridden method that just calls its super()?
If we remove this method the stop() from parent is called anyway.

Copy link
Member

@shakuzen shakuzen Jan 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking a look. Indeed we don't need to override if we aren't doing anything but calling the super here. I'll take care of it in a polish commit.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, it looks like removing the override breaks binary compatibility even if it is source compatible, which @dimovelev found out when they tried to remove it and added it back.

Copy link
Member

@shakuzen shakuzen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution and follow up to feedback. I wish we had a good way to test this, but I guess we'll have to rely on users to try it out and report back.

@@ -125,6 +125,7 @@ public void start(ThreadFactory threadFactory) {
logger.error("unable to start stackdriver, service settings are not available");
}
else {
shutdownClientIfNecessary(true);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is necessary if nothing is done with the client in stop. The question is: should something be done in stop or only in close? We haven't been very clear about what should happen where (no JavaDocs). I'll open a separate issue to clear that up.

It's a bit wasteful of resources to not shutdown the client when a user calls stop on the registry, but in terms of publishing it shouldn't be an issue because the scheduled call to publish won't happen after stop is called until start is called again. Looking at other implementations, we do the same in WavefrontMeterRegistry - nothing special in stop and close the client on close after super.close().

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is necessary if nothing is done with the client in stop.

I remembered now the issue I was pointing out in my previous review. We don't want to replace the client without shutting down the old one and making it eligible for garbage collection. Therefore having this line here should prevent that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As promised, I opened #4632 to track clarifying stop semantics.

@shakuzen shakuzen changed the base branch from main to 1.10.x January 31, 2024 11:18
@shakuzen shakuzen merged commit f53d61a into micrometer-metrics:1.10.x Jan 31, 2024
6 checks passed
shakuzen added a commit that referenced this pull request Jan 31, 2024
No need to override the `stop` method and reduces the indentation on shutdownClientIfNecessary.
@shakuzen
Copy link
Member

I've merged to the 1.10.x maintenance branch and I will merge forward. Please try out snapshots to make sure there are no issues with the fix. I did a little polish in 3326c10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Stackdriver exception trying to publish metrics during shutdown
6 participants