Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory issues / many instances of MaxListenersExceededWarning when using Undici via Elasticsearch #3131

Closed
daveyarwood opened this issue Apr 16, 2024 · 8 comments
Labels
bug Something isn't working

Comments

@daveyarwood
Copy link

Bug Description

This is a follow-up to #1711. @KhafraDev asked that I file a separate issue because Elasticsearch does not appear to be using fetch, which is what was fixed in that issue (via #2823).

I've been running into what I think is the same or similar issue to what was reported in #1711 - lots of MaxListenersExceededWarnings being logged, and my service eventually runs out of memory and dies. When I run with --trace-warnings, the stacktrace points to Undici:

(node:14480) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 101 abort listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit
    at _addListener (node:events:588:17)
    at EventEmitter.addListener (node:events:606:10)
    at addAbortListener (/Users/daveyarwood/code/spark/node_modules/undici/lib/core/util.js:443:10)
    at addSignal (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/abort-signal.js:36:3)
    at new RequestHandler (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:66:5)
    at Pool.request (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:171:25)
    at /Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:164:15
    at new Promise (<anonymous>)
    at Pool.request (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:163:12)

This is with Undici 6.13.0, which is a transitive dependency of @elastic/transport 8.5.0.

Reproducible By

I would be happy to provide a minimal repro case if I had any idea how to do that! :)

Please let me know if you have any ideas of things I can test in my setup.

Expected Behavior

  • No MaxListenersExceededWarning messages should be logged.
  • In a deployed environment, the memory graphs should not show available memory declining significantly, and the machine shouldn't run out of memory and die.

Environment

Local

2021 Macbook Pro M1 Max
macOS Sonoma 14.11
Node 20.9.0

I'm seeing the MaxListenersExceededWarnings when I run my service locally.

Deployed

My service is also running on a Linux server, in a Docker container with Node 20. I'm also seeing the MaxListenersExceededWarnings logged there, and the memory graph shows a significant decline in memory usage from the time the instance spins up to when it eventually runs out of memory and dies.

Additional context

I don't fully understand what @KhafraDev meant by Elasticsearch not using fetch. I looked through the @elastic/transport code base (on the v8.5.0 tag, which is the version I'm using) and I found UndiciConnection.ts, which appears to be where it's creating an Undici pool and calling request on it, but beyond that, I'm not sure what to look for.

@daveyarwood daveyarwood added the bug Something isn't working label Apr 16, 2024
@daveyarwood
Copy link
Author

daveyarwood commented Apr 16, 2024

It's quite possible that there is some other reason that my service is running out of memory and dying. But at any rate, it would be good for us to get to the bottom of the MaxListenersExceededWarnings, which (as far as I can tell, from my limited understanding) appear to suggest a memory leak somewhere in Undici.

@ronag
Copy link
Member

ronag commented Apr 16, 2024

I believe I open end an issue over at elasticsearch a long time ago. They don't properly cleanup signals.

@ronag
Copy link
Member

ronag commented Apr 16, 2024

elastic/elasticsearch-js#1716

@Uzlopak
Copy link
Contributor

Uzlopak commented Apr 17, 2024

Just one remark: when running the wpt suite for fetch, i see the MaxListenersExceededWarning emitted too.

@JoshMock
Copy link

elastic/elasticsearch-js#1716

Thanks for pointing this out @ronag. Seems I didn't address your comment there when I initially thought the problem was resolved in elasticsearch 8.8.1.

@sibelius
Copy link

is this fixed?

@JoshMock
Copy link

is this fixed?

Not in the Elasticsearch client, yet. I have to run down what @ronag suggested and establish the best way to fix that without introducing any breaking changes.

Updates will be shared on elastic/elastic-transport-js#63.

@daveyarwood
Copy link
Author

@elastic/elastic-transport 8.5.2 has just been released - updating to that version fixed my issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants