Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable default geoip database downloading for Elasticsearch container #7247

Merged
merged 1 commit into from Jun 28, 2023

Conversation

lhotari
Copy link
Contributor

@lhotari lhotari commented Jun 28, 2023

Motivation

By default, Elasticsearch container will download the geoip lite database.
It's better to disable this behavior by default.
There's also an elastic issue about this default behavior: elastic/elasticsearch#90673. Fixed since v8.7.0 with elastic/elasticsearch#92335 .

Modifications

geoip downloading can be disabled by passing -Dingest.geoip.downloader.enabled.default=false in JVM options or passing ingest.geoip.downloader.enabled=false in the environment.

The problem with passing ingest.geoip.downloader.enabled=false in the environment is that it's not compatible with OpenSearch. OpenSearch will fail to start with a configuration validation error org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [ingest.geoip.downloader.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings. Some users might be using ElasticsearchContainer class together with the OpenSearch docker image.

@lhotari lhotari requested a review from a team as a code owner June 28, 2023 17:33
@eddumelendez
Copy link
Member

JFTR, OpenSearch provides its own module

@lhotari
Copy link
Contributor Author

lhotari commented Jun 28, 2023

The elastic issue description elastic/elasticsearch#90673 is interesting:

In Elasticsearch 7.14 a feature was added which would automatically download the MaxMind GeoIP database. Specifically, it downloads the database from Google Cloud after fetching this JSON object. This is enabled by default.

We discovered that this feature was responsible for around $20,000 on our AWS bill since we upgraded last March and based on a rough approximation of traffic we were charged for it probably cost Elastic about $5,500 💸 Investigation was slow and required enabling AWS VPC logs to assess the NAT gateway traffic source.

@lhotari
Copy link
Contributor Author

lhotari commented Jun 28, 2023

I guess this geoip download doesn't impact the default image tag that ElasticsearchContainer sets, 7.9.2. The feature was introduced in 7.14.0.
7.14.0 release notes includes:

  • Enable GeoIP downloader by default #74323

@eddumelendez eddumelendez added this to the next milestone Jun 28, 2023
@eddumelendez eddumelendez merged commit b8c4a65 into testcontainers:main Jun 28, 2023
85 checks passed
@eddumelendez
Copy link
Member

Thanks again, @lhotari ! The context is really useful, let's keep compatibility in case OpenSearch is used along with ElasticsearchContainer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants