Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: improve skip row group using statistics condition #14056

Merged
merged 1 commit into from
Jan 29, 2024

Conversation

taki-mekhalfa
Copy link
Contributor

Fixes: 13427

Old code skipped row groups only if values passed to is_in are ALL under min or ALL above max. Imroved to skip if ALL values are either under min or above max

@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars labels Jan 28, 2024

Verified

This commit was signed with the committer’s verified signature. The key has expired.
levrik Levin Rickert
Old code skipped row groups only if values passed to `is_in` are _ALL_ under min or _ALL_ above max.
Improved to skip if _ALL_ values are either under min or above max
@taki-mekhalfa taki-mekhalfa changed the title fix(python,rust): improve skip row group using statistics condition perf(python,rust): improve skip row group using statistics condition Jan 28, 2024
@github-actions github-actions bot added the performance Performance issues or improvements label Jan 28, 2024
@ritchie46
Copy link
Member

Small change, great consequences. Thank you very much. :)

@ritchie46 ritchie46 changed the title perf(python,rust): improve skip row group using statistics condition perf: improve skip row group using statistics condition Jan 29, 2024
@ritchie46 ritchie46 merged commit 4c45dac into pola-rs:main Jan 29, 2024
@bchalk101
Copy link
Contributor

Thnx @taki-mekhalfa 😄 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fix performance Performance issues or improvements python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Not skipping parquets correctly when using is_in
3 participants