Memory Leak when using Multi Replay

[markusdlugi]

Context

In one of our applications we observed a memory leak when using Multi.createBy().replaying() due to subscriptions not being removed properly. After removing the usage, the memory leak vanished.

Description

We were using the Multi replay functionality in one of our applications to achieve that a REST call is only performed once and the result then re-used for all subsequent subscribers. The code looked something like this:

Multi.createBy().replaying().upTo(1).ofMulti(performRestCallAndStuff()).collect().first();

The resulting Uni would be consumed each time a message was received via SQS, i.e., each message resulted in a new subscription.

During some load tests where thousands of messages were sent to the application, we observed that the application's memory usage would increase continuously until the application ultimately crashed with OOM. After some debugging, we identified the Multi replay as the root cause. After removing it and replacing it with a different caching mechanism, the application's memory usage is fine again.

Additional details

We debugged further and found the culprit to be the subscriptions list in the ReplayOperator class. We observed during debugging that the size of the list keeps on increasing and no subscriptions are ever removed. We believe this is caused by ReplayOperator#subscribe calling subscriber.onSubscribe(replaySubscription) before subscriptions.add(replaySubscription). The former would result in cancel() to be called on the ReplaySubscription (and thus also subscriptions.remove(this)) even before the subscription was added to the list. So in the end, the subscription was not removed and stayed in the list forever.

• Quarkus version: 3.6.4
• Mutiny version: 2.5.1

[jponge]

I'll have a look, probably next week.

However since you've done a good amount of investigation, would you be willing to prepare a pull-request?

[markusdlugi]

@jponge, I created a PR, would be great if you could have a look. As expected, the fix itself was not too troublesome, but a proper test was a little more challenging :)

[jponge]

Thank you, I'll get back to you in the coming days 😃

…

On Tue, Jan 9, 2024 at 6:44 PM Markus Dlugi ***@***.***> wrote: @jponge <https://github.com/jponge>, I created a PR, would be great if you could have a look. As expected, the fix itself was not too troublesome, but a proper test was a little more challenging :) — Reply to this email directly, view it on GitHub <#1482 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAGK2JJHQ3LE7A2O7NKDZDYNV6YLAVCNFSM6AAAAABBTCI5Y2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBTGUYTANBVGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[markusdlugi]

@jponge do you plan to create a new release for this fix so it can be incorporated in the next Quarkus version?

[jponge]

There will be a release "soon", I'd like to push another bug fix along with this one.

…

On Mon, Jan 15, 2024 at 4:55 PM Markus Dlugi ***@***.***> wrote: @jponge <https://github.com/jponge> do you plan to create a new release for this fix so it can be incorporated in the next Quarkus version? — Reply to this email directly, view it on GitHub <#1482 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAGK2NDEBQMUHBPCJIEGULYOVGQBAVCNFSM6AAAAABBTCI5Y2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJSGQZDKMJTHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jponge]

@markusdlugi 2.5.4 has been released. It will eventually land in Quarkus, but meanwhile you can manually add an explicit dependency to Mutiny 2.5.4 in your Quarkus projects.