Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BUG: mode not sorting values for arrow backed strings #55621

Merged
merged 9 commits into from Oct 25, 2023
Merged

BUG: mode not sorting values for arrow backed strings #55621

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
6815bbd
BUG: mode not sorting values for arrow backed strings
phofl Oct 21, 2023
699b0bd
Fix tests
phofl Oct 21, 2023
b047558
Change to pa_installed variable
phofl Oct 22, 2023
69826ec
Merge remote-tracking branch 'upstream/main' into string_array_mode
phofl Oct 22, 2023
50f5ddb
Merge branch 'main' into string_array_mode
phofl Oct 23, 2023
78ada02
Update pyarrow.py
phofl Oct 24, 2023
024e9b0
Fix
phofl Oct 24, 2023
fe1bc37
Fix
phofl Oct 24, 2023
fdb022a
Merge branch 'main' into string_array_mode
phofl Oct 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v2.1.2.rst
View file
Open in desktop
Expand Up @@ -30,6 +30,7 @@ Bug fixes
- Fixed bug in :meth:`Index.insert` raising when inserting ``None`` into :class:`Index` with ``dtype="string[pyarrow_numpy]"`` (:issue:`55365`)
- Fixed bug in :meth:`Series.all` and :meth:`Series.any` not treating missing values correctly for ``dtype="string[pyarrow_numpy]"`` (:issue:`55367`)
- Fixed bug in :meth:`Series.floordiv` for :class:`ArrowDtype` (:issue:`55561`)
- Fixed bug in :meth:`Series.mode` not sorting values for arrow backed string dtype (:issue:`55621`)
- Fixed bug in :meth:`Series.rank` for ``string[pyarrow_numpy]`` dtype (:issue:`55362`)
- Fixed bug in :meth:`Series.str.extractall` for :class:`ArrowDtype` dtype being converted to object (:issue:`53846`)
- Silence ``Period[B]`` warnings introduced by :issue:`53446` during normal plotting activity (:issue:`55138`)
Expand Down
1 change: 1 addition & 0 deletions pandas/core/arrays/arrow/array.py
View file
Open in desktop
Expand Up @@ -1921,6 +1921,7 @@ def _mode(self, dropna: bool = True) -> Self:
if pa.types.is_temporal(pa_type):
most_common = most_common.cast(pa_type)

most_common = most_common.take(pc.array_sort_indices(most_common))
return type(self)(most_common)

def _maybe_convert_setitem_value(self, value):
Expand Down
4 changes: 2 additions & 2 deletions pandas/tests/extension/test_arrow.py
View file
Open in desktop
Expand Up @@ -1416,7 +1416,7 @@ def test_quantile(data, interpolation, quantile, request):

@pytest.mark.parametrize(
"take_idx, exp_idx",
[[[0, 0, 2, 2, 4, 4], [0, 4]], [[0, 0, 0, 2, 4, 4], [0]]],
[[[0, 0, 2, 2, 4, 4], [4, 0]], [[0, 0, 0, 2, 4, 4], [0]]],
ids=["multi_mode", "single_mode"],
)
def test_mode_dropna_true(data_for_grouping, take_idx, exp_idx):
Expand All @@ -1434,7 +1434,7 @@ def test_mode_dropna_false_mode_na(data):
expected = pd.Series([None], dtype=data.dtype)
tm.assert_series_equal(result, expected)

expected = pd.Series([None, data[0]], dtype=data.dtype)
expected = pd.Series([data[0], None], dtype=data.dtype)
result = expected.mode(dropna=False)
tm.assert_series_equal(result, expected)

Expand Down
15 changes: 13 additions & 2 deletions pandas/tests/groupby/test_groupby.py
View file
Open in desktop
Expand Up @@ -5,6 +5,7 @@
import numpy as np
import pytest

from pandas.compat import pa_version_under7p0
from pandas.errors import (
PerformanceWarning,
SpecificationError,
Expand Down Expand Up @@ -2763,13 +2764,23 @@ def test_rolling_wrong_param_min_period():
test_df.groupby("name")["val"].rolling(window=2, min_period=1).sum()


def test_by_column_values_with_same_starting_value():
@pytest.mark.parametrize(
"dtype",
[
object,
pytest.param(
"string[pyarrow_numpy]",
marks=pytest.mark.skipif(pa_version_under7p0, reason="arrow not installed"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use an importorskip in the test when dtype is pyarrow_numpy? (avoids having to change pa_version_under7p0 every time we bump pyarrow)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thoughts about creating a pyarrow_installed variable?

I like this pattern a bit better than the if inside the test

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thoughts about creating a pyarrow_installed variable?

Sure this would be good too

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

),
],
)
def test_by_column_values_with_same_starting_value(dtype):
# GH29635
df = DataFrame(
{
"Name": ["Thomas", "Thomas", "Thomas John"],
"Credit": [1200, 1300, 900],
"Mood": ["sad", "happy", "happy"],
"Mood": Series(["sad", "happy", "happy"], dtype=dtype),
}
)
aggregate_details = {"Mood": Series.mode, "Credit": "sum"}
Expand Down