Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Fix near-minimum timestamp handling #57314

Merged
merged 11 commits into from
Feb 22, 2024
1 change: 1 addition & 0 deletions doc/source/whatsnew/v2.2.1.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Fixed regressions
~~~~~~~~~~~~~~~~~
- Fixed memory leak in :func:`read_csv` (:issue:`57039`)
- Fixed performance regression in :meth:`Series.combine_first` (:issue:`55845`)
- Fixed regression causing overflow for near-minimum timestamps (:issue:`57150`)
- Fixed regression in :func:`concat` changing long-standing behavior that always sorted the non-concatenation axis when the axis was a :class:`DatetimeIndex` (:issue:`57006`)
- Fixed regression in :func:`merge_ordered` raising ``TypeError`` for ``fill_method="ffill"`` and ``how="left"`` (:issue:`57010`)
- Fixed regression in :func:`pandas.testing.assert_series_equal` defaulting to ``check_exact=True`` when checking the :class:`Index` (:issue:`57067`)
Expand Down
18 changes: 14 additions & 4 deletions pandas/_libs/src/vendored/numpy/datetime/np_datetime.c
Original file line number Diff line number Diff line change
Expand Up @@ -482,10 +482,20 @@ npy_datetime npy_datetimestruct_to_datetime(NPY_DATETIMEUNIT base,

if (base == NPY_FR_ns) {
int64_t nanoseconds;
PD_CHECK_OVERFLOW(
scaleMicrosecondsToNanoseconds(microseconds, &nanoseconds));
PD_CHECK_OVERFLOW(
checked_int64_add(nanoseconds, dts->ps / 1000, &nanoseconds));

// for near-minimum timestamps, scaling microseconds to nanoseconds
// overflows but adding nanoseconds puts the timestamp back in a valid range
const int64_t min_nanoseconds = NPY_MIN_INT64 + 1;
if (microseconds == min_nanoseconds / 1000 - 1) {
WillAyd marked this conversation as resolved.
Show resolved Hide resolved
// calculate final nanoseconds from minimum without scaling microseconds
PD_CHECK_OVERFLOW(checked_int64_add(
WillAyd marked this conversation as resolved.
Show resolved Hide resolved
min_nanoseconds, (dts->ps - _NS_MIN_DTS.ps) / 1000, &nanoseconds));
WillAyd marked this conversation as resolved.
Show resolved Hide resolved
} else {
PD_CHECK_OVERFLOW(
scaleMicrosecondsToNanoseconds(microseconds, &nanoseconds));
PD_CHECK_OVERFLOW(
checked_int64_add(nanoseconds, dts->ps / 1000, &nanoseconds));
}

return nanoseconds;
}
Expand Down
17 changes: 17 additions & 0 deletions pandas/tests/tslibs/test_array_to_datetime.py
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,23 @@ def test_to_datetime_barely_out_of_bounds():
tslib.array_to_datetime(arr)


@pytest.mark.parametrize(
"timestamp",
[
# Close enough to bounds that scaling micros to nanos overflows
# but adding nanos would result in an in-bounds datetime.
"1677-09-21T00:12:43.145224193",
"1677-09-21T00:12:43.145224999",
# this always worked
"1677-09-21T00:12:43.145225000",
],
)
def test_to_datetime_barely_inside_bounds(timestamp):
# see gh-57150
result, _ = tslib.array_to_datetime(np.array([timestamp], dtype=object))
tm.assert_numpy_array_equal(result, np.array([timestamp], dtype="M8[ns]"))


class SubDatetime(datetime):
pass

Expand Down