Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add drop_nans method to DataFrame and LazyFrame #20029

Merged
merged 2 commits into from
Nov 27, 2024

Conversation

alexander-beedie
Copy link
Collaborator

@alexander-beedie alexander-beedie commented Nov 27, 2024

API consistency: we already have drop_nans exposed on both Series and Expr, but were missing it at the frame-level.

This is a useful one as there are many vectors by which we might be exposed to NaN values, especially when consuming the results of upstream numerical computation, so it's beneficial to provide a first-class method (while the first thought for NaN may be Pandas, in quant finance it's equally likely to be compute).

Slightly optimised over drop_nulls as it only has to consider floating point cols.

Example

import polars.selectors as cs
import polars as pl

df = pl.DataFrame(
  {
    "a": [1.0, float("nan"), 3.0, 4.0],
    "b": ["hello", "foo", "world", "bar"],
    "c": [-90.5, 25.0, 0.0, float("nan")],
  },
  schema_overrides={"a": pl.Float32, "c": pl.Float64},
)
# shape: (4, 3)
# ┌─────┬───────┬───────┐
# │ a   ┆ b     ┆ c     │
# │ --- ┆ ---   ┆ ---   │
# │ f32 ┆ str   ┆ f64   │
# ╞═════╪═══════╪═══════╡
# │ 1.0 ┆ hello ┆ -90.5 │
# │ NaN ┆ foo   ┆ 25.0  │
# │ 3.0 ┆ world ┆ 0.0   │
# │ 4.0 ┆ bar   ┆ NaN   │
# └─────┴───────┴───────┘

Default:

  • Drop rows when NaN value found in any float column
    df.drop_nans()
    # ┌─────┬───────┬───────┐
    # │ a   ┆ b     ┆ c     │
    # │ --- ┆ ---   ┆ ---   │
    # │ f32 ┆ str   ┆ f64   │
    # ╞═════╪═══════╪═══════╡
    # │ 1.0 ┆ hello ┆ -90.5 │
    # │ 3.0 ┆ world ┆ 0.0   │
    # └─────┴───────┴───────┘

Subset:

  • Drop rows when NaN value found in one or more specific columns
    df.drop_nans(subset=["a"])
    # shape: (3, 3)
    # ┌─────┬───────┬───────┐
    # │ a   ┆ b     ┆ c     │
    # │ --- ┆ ---   ┆ ---   │
    # │ f32 ┆ str   ┆ f64   │
    # ╞═════╪═══════╪═══════╡
    # │ 1.0 ┆ hello ┆ -90.5 │
    # │ 3.0 ┆ world ┆ 0.0   │
    # │ 4.0 ┆ bar   ┆ NaN   │
    # └─────┴───────┴───────┘
  • Drop rows when NaN value found in any columns matching a selector
    df.drop_nans(subset=cs.ends_with("c"))
    # shape: (3, 3)
    # ┌─────┬───────┬───────┐
    # │ a   ┆ b     ┆ c     │
    # │ --- ┆ ---   ┆ ---   │
    # │ f32 ┆ str   ┆ f64   │
    # ╞═════╪═══════╪═══════╡
    # │ 1.0 ┆ hello ┆ -90.5 │
    # │ NaN ┆ foo   ┆ 25.0  │
    # │ 3.0 ┆ world ┆ 0.0   │
    # └─────┴───────┴───────┘

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars labels Nov 27, 2024
@alexander-beedie alexander-beedie changed the title feat(python): Add missing drop_nans method to DataFrame and LazyFrame feat: Add missing drop_nans method to DataFrame and LazyFrame Nov 27, 2024
@github-actions github-actions bot added the rust Related to Rust Polars label Nov 27, 2024
@ritchie46
Copy link
Member

Promise that we never discuss drop_(inf)inite, then. ;)

@alexander-beedie
Copy link
Collaborator Author

alexander-beedie commented Nov 27, 2024

Promise that we never discuss drop_(inf)inite, then. ;)

As long as we never expose that on Series and Expr (which we haven't!) you're safe 🤣

@alexander-beedie alexander-beedie changed the title feat: Add missing drop_nans method to DataFrame and LazyFrame feat: Add drop_nans method to DataFrame and LazyFrame Nov 27, 2024
Copy link

codecov bot commented Nov 27, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.50%. Comparing base (7c97625) to head (7ece8d7).
Report is 3 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #20029   +/-   ##
=======================================
  Coverage   79.49%   79.50%           
=======================================
  Files        1556     1556           
  Lines      216527   216560   +33     
  Branches     2456     2457    +1     
=======================================
+ Hits       172136   172167   +31     
- Misses      43833    43835    +2     
  Partials      558      558           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Verified

This commit was signed with the committer’s verified signature. The key has expired.
arjantijms Arjan Tijms
@ritchie46
Copy link
Member

docs entries. :)

@alexander-beedie
Copy link
Collaborator Author

docs entries. :)

🤦

@ritchie46 ritchie46 merged commit 0b218d8 into pola-rs:main Nov 27, 2024
27 checks passed
@alexander-beedie alexander-beedie deleted the frame-level-drop-nans branch November 28, 2024 05:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants