Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TST: Make read_csv tests pyarrow 13 compatable on 2.1.x #55855

Merged
merged 3 commits into from
Nov 7, 2023
Merged

TST: Make read_csv tests pyarrow 13 compatable on 2.1.x #55855

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
18f5a7d
TST: Make read_csv tests pyarrow 13 compatable on 2.1.x
mroeschke Nov 6, 2023
6ba8e2f
Merge remote-tracking branch 'upstream/2.1.x' into tst/pyarrow_failures
mroeschke Nov 7, 2023
a2f0029
Skip on windows for ARROW_TIMEZONE_DATABASE
mroeschke Nov 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 6 additions & 2 deletions pandas/tests/io/parser/common/test_common_basic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,9 @@ def test_read_csv_local(all_parsers, csv1):

fname = prefix + str(os.path.abspath(csv1))
result = parser.read_csv(fname, index_col=0, parse_dates=True)

# TODO: make unit check more specific
if parser.engine == "pyarrow":
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we have a pyarrow version compat instead of blanket doing this? I think we should just take whatever precision pyarrow provides?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should just take whatever precision pyarrow provides?

Yeah agreed. We have the same TODO comment on the main branch where ideally we should do a version compat check (IMO it's worth investing the change there as this is just to get the 2.1.x branch green again)

result.index = result.index.as_unit("ns")
expected = DataFrame(
[
[0.980269, 3.685731, -0.364216805298, -1.159738],
Expand Down Expand Up @@ -177,7 +179,9 @@ def test_read_csv_low_memory_no_rows_with_index(all_parsers):
def test_read_csv_dataframe(all_parsers, csv1):
parser = all_parsers
result = parser.read_csv(csv1, index_col=0, parse_dates=True)

# TODO: make unit check more specific
if parser.engine == "pyarrow":
result.index = result.index.as_unit("ns")
expected = DataFrame(
[
[0.980269, 3.685731, -0.364216805298, -1.159738],
Expand Down
8 changes: 7 additions & 1 deletion pandas/tests/io/parser/test_parse_dates.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -979,12 +979,15 @@ def test_parse_dates_custom_euro_format(all_parsers, kwargs):
)


def test_parse_tz_aware(all_parsers, request):
def test_parse_tz_aware(all_parsers):
# See gh-1693
parser = all_parsers
data = "Date,x\n2012-06-13T01:39:00Z,0.5"

result = parser.read_csv(StringIO(data), index_col=0, parse_dates=True)
# TODO: make unit check more specific
if parser.engine == "pyarrow":
result.index = result.index.as_unit("ns")
expected = DataFrame(
{"x": [0.5]}, index=Index([Timestamp("2012-06-13 01:39:00+00:00")], name="Date")
)
Expand Down Expand Up @@ -2231,6 +2234,9 @@ def test_parse_dates_arrow_engine(all_parsers):
2000-01-01 00:00:01,1"""

result = parser.read_csv(StringIO(data), parse_dates=["a"])
# TODO: make unit check more specific
if parser.engine == "pyarrow":
result["a"] = result["a"].dt.as_unit("ns")
expected = DataFrame(
{
"a": [
Expand Down