-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce execution_hint
for Cardinality aggregation
#17312
Merged
Merged
+222
−22
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>
Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>
Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>
Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>
Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>
Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>
Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>
Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>
Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>
Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com>
❌ Gradle check result for 0fafb49: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Asim Mahmood <asim.seng@gmail.com>
server/src/main/java/org/opensearch/search/aggregations/metrics/CardinalityAggregator.java
Show resolved
Hide resolved
...r/src/main/java/org/opensearch/search/aggregations/metrics/CardinalityAggregatorFactory.java
Show resolved
Hide resolved
Signed-off-by: Asim Mahmood <asim.seng@gmail.com>
rishabhmaurya
approved these changes
Feb 15, 2025
sandeshkr419
approved these changes
Feb 17, 2025
cwperks
approved these changes
Feb 18, 2025
msfroh
approved these changes
Feb 20, 2025
This was referenced Feb 20, 2025
The backport to
To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-17312-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e3a6ccadc942c64e83bd224031bc4d1c6ab14623
# Push it to GitHub
git push --set-upstream origin backport/backport-17312-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x Then, create a pull request where the |
3 tasks
asimmahmood1
added a commit
to asimmahmood1/OpenSearch
that referenced
this pull request
Feb 21, 2025
…oject#17312) - backport to 2.19 --------- Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com> Signed-off-by: Asim Mahmood <asim.seng@gmail.com> Signed-off-by: Asim M <asim.seng@gmail.com> Co-authored-by: Siddharth Rayabharam <maitreya2954@gmail.com> Co-authored-by: Craig Perkins <craig5008@gmail.com> (cherry picked from commit e3a6cca)
3 tasks
andrross
pushed a commit
that referenced
this pull request
Feb 21, 2025
Loading
Loading status checks…
…kport to 2.19 (#17420) * Introduce `execution_hint` for Cardinality aggregation (#17312) - backport to 2.19 --------- Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com> Signed-off-by: Asim Mahmood <asim.seng@gmail.com> Signed-off-by: Asim M <asim.seng@gmail.com> Co-authored-by: Siddharth Rayabharam <maitreya2954@gmail.com> Co-authored-by: Craig Perkins <craig5008@gmail.com> (cherry picked from commit e3a6cca) * Update changelog Signed-off-by: Asim Mahmood <asim.seng@gmail.com> --------- Signed-off-by: Asim Mahmood <asim.seng@gmail.com>
cwperks
pushed a commit
that referenced
this pull request
Feb 27, 2025
Loading
Loading status checks…
…kport to 2.x (#17419) * Introduce `execution_hint` for Cardinality aggregation (#17312) --------- Signed-off-by: Siddharth Rayabharam <maitreya2954@gmail.com> Signed-off-by: Asim Mahmood <asim.seng@gmail.com> Signed-off-by: Asim M <asim.seng@gmail.com> Co-authored-by: Siddharth Rayabharam <maitreya2954@gmail.com> Co-authored-by: Craig Perkins <craig5008@gmail.com> (cherry picked from commit e3a6cca) * Update changlog Signed-off-by: Asim Mahmood <asim.seng@gmail.com> * Update min version to 2_19_1 * since 2.19 version has been merged: #17420 Signed-off-by: Asim Mahmood <asim.seng@gmail.com> --------- Signed-off-by: Asim Mahmood <asim.seng@gmail.com>
3 tasks
asimmahmood1
added a commit
to asimmahmood1/opensearch-benchmark-workloads
that referenced
this pull request
Mar 13, 2025
…execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality Signed-off-by: Asim Mahmood <asim.seng@gmail.com>
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
backport 2.x
Backport to 2.x branch
backport-failed
enhancement
Enhancement or improvement to existing feature or request
good first issue
Good for newcomers
Search:Aggregations
Search:Performance
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Introduce
execution_hint
for Cardinality aggregation - Revive PR #15764This is intended as short term solution as a workaround to users that see large latency regression when upgrading to OS 2.x. The main risk here is ordinals take up more memory, i.e. depends on size of cardinality. So for very high cardinality, there is a risk of OOM at shard node. This is marked as power user.
I plan to follow up with #15269 which is more useful setting. This will allow the user select a higher threshold for using ordinals, e.g. a shard with large heap can use ordinals more often, but still fall back to the slower direct as needed.
Related Issues
Resolves #16837.
Check List
Testing
Tested using big5 dataset, see #16837 (comment)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.