Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Fix near-minimum timestamp handling #57314

Merged
merged 11 commits into from Feb 22, 2024
19 changes: 16 additions & 3 deletions pandas/_libs/src/vendored/numpy/datetime/np_datetime.c
Expand Up @@ -483,14 +483,27 @@ npy_datetime npy_datetimestruct_to_datetime(NPY_DATETIMEUNIT base,
if (base == NPY_FR_ns) {
int64_t nanoseconds;

// for near-minimum timestamps, scaling microseconds to nanoseconds
// overflows but adding nanoseconds puts the timestamp back in a valid range
// Minimum valid timestamp in nanoseconds (1677-09-21 00:12:43.145224193).
const int64_t min_nanoseconds = NPY_MIN_INT64 + 1;

// For near-minimum timestamps (1677-09-21 00:12:43.145224193 through
// 1677-09-21 00:12:43.145224999), scaling microseconds to nanoseconds
// overflows (1677-09-21 00:12:43.145224 -> 1677-09-21 00:12:43.145224000),
// but adding nanoseconds can put the timestamp back in a valid range for
// nanosecond parts >= 193.

// (min_nanoseconds / 1000 - 1) * 1000 would overflow, so do not scale.
// This happens if microseconds corresponds to 1677-09-21 00:12:43.145224.
if (microseconds == min_nanoseconds / 1000 - 1) {
WillAyd marked this conversation as resolved.
Show resolved Hide resolved
// calculate final nanoseconds from minimum without scaling microseconds
// Instead, use minimum nanosecond timestamp as base and offset it with
WillAyd marked this conversation as resolved.
Show resolved Hide resolved
// nanosecond delta between dts and the minimum (_NS_MIN_DTS.ps = 193000).
// If dts->ps >= _NS_MIN_DTS.ps, timestamp is at/above the minimum.
// If dts->ps < _NS_MIN_DTS.ps, timestamp is below minimum and overflows.
PD_CHECK_OVERFLOW(checked_int64_add(
WillAyd marked this conversation as resolved.
Show resolved Hide resolved
min_nanoseconds, (dts->ps - _NS_MIN_DTS.ps) / 1000, &nanoseconds));
WillAyd marked this conversation as resolved.
Show resolved Hide resolved
} else {
// microseconds does not correspond to near-minimum timestamp, use default
// scaling and addition approach, handling any other overflows.
PD_CHECK_OVERFLOW(
scaleMicrosecondsToNanoseconds(microseconds, &nanoseconds));
PD_CHECK_OVERFLOW(
Expand Down