Windows: base implementation on `GetTimeZoneInformationForYear` #1017

pitdicker · 2023-04-17T13:41:08Z

Issues caused by the current use of Windows API's:

TzSpecificLocalTimeToSystemTime doesn't let us choose how to handle ambiguous cases during a DST transition. It always returns the earliest option.
using GetLocalTime with TzSpecificLocalTimeToSystemTime can give wrong results during a DST transition because the issue mentioned above.
TzSpecificLocalTimeToSystemTime always returns a result, also for non-existing datetimes during a DST transition.
The Windows API restricts years to a range of 1601..30828.

The alternative in this PR is to calculate the offset ourselves using the information provided by GetTimeZoneInformationForYear.This is similar to what we do on Unix.

The tricky bits to determine the correct offset between DST transitions is copied from the Unix implementation. Initially I wanted to share the code between the two systems, but that didn't make the Unix implementation any clearer.

Part of the old Windows implementation have been moved to a test, so our implementation can be compared to TzSpecificLocalTimeToSystemTime. I have set my computer timezone to ca. 20 'interesting' timezones (determined by looking at data of the entries enumerated by EnumDynamicTimeZoneInformation) and found no differences.

I ended up writing this PR because of a yak shaving. In order to remove some unwraps from my code, I need some progress on #716. To show we need a fourth enum variant for LocalResult, I started implementing a proof of concept for DateTime<Local>. And it turned out the Windows implementation had no place to hook into...

Fixes #1150, #651.

djc · 2023-04-17T13:47:37Z

Did you see #997 already?

esheppa · 2023-04-17T13:52:20Z

It looks like these two PRs should be somewhat orthogonal, but this is also related to #750 which is great as by extending this we should be able to extract the name of the timezone as well

djc · 2023-04-17T13:55:31Z

The tricky bits to determine the correct offset between DST transitions is copied from the Unix implementation. Initially I wanted to share the code between the two systems, but that didn't make the Unix implementation any clearer.

Can you add that here as a separate commit? It does sound like something we'd want.

pitdicker · 2023-04-17T14:02:52Z

Did you see #997 already?

Somewhat, I forgot about it. That workaround would no longer be necessary. But the PR before by @nekevss was very useful for feeling my way in the Windows implementation.

Can you add that here as a separate commit? It does sound like something we'd want.

I can dig up the commit, but it will probably be tomorrow because it needs some cleaning up.

esheppa · 2023-04-17T14:04:01Z

We could also review and merge this as is, before enhancing to get the timezone name via another PR if that is more suitable @djc?

pitdicker · 2023-04-17T18:37:37Z

I added the commits to share part of the implementation with Unix. It touches more than I like, because Windows and Unix need to pass around the same type for the std/dst offsets: FixedOffset instead of LocalTimeType.

djc

Thanks!

src/offset/local/tz_info/timezone.rs

src/offset/local/mod.rs

pitdicker · 2023-04-19T05:37:50Z

Just noticed issue #651. That issue would be fixed by this PR, added a test.

src/offset/local/windows.rs

src/datetime/tests.rs

pitdicker · 2023-04-24T05:08:44Z

I noticed Utc::now uses the standard library to get the current date, while for Local::now we call a Windows API ourselves. Because on both Unix and Windows we get an UTC timestamp from the OS and then convert it to local time, we may just as well always use Utc::now inside Local::now and be consistent.

On Windows the standard library uses QueryPerformanceCounter, which is guaranteed to provide sub-microsecond values. What we used for Local::now returned a SYSTEMTIME with at most millisecond precision.

pitdicker · 2023-04-24T05:09:46Z

The benchmark results have improved a bit.
Before:

     Running benches\chrono.rs (target\release\deps\chrono-3f8222dc2c00b8c6.exe)
bench_get_local_time    time:   [1.4577 µs 1.5100 µs 1.5519 µs]

After:

     Running benches\chrono.rs (target\release\deps\chrono-3f8222dc2c00b8c6.exe)
bench_get_local_time    time:   [1.2091 µs 1.2737 µs 1.3304 µs]
                        change: [-17.784% -13.853% -9.6609%] (p = 0.00 < 0.05)
                        Performance has improved.

pitdicker · 2023-04-27T16:54:44Z

src/offset/local/mod.rs

+
+#[cfg(any(unix, windows))]
+impl TzInfo {
+    fn lookup_with_dst_transitions(&self, dt: &NaiveDateTime) -> LocalResult<FixedOffset> {


It took me hours to convince myself the logic in this function was correct in the Unix version, so I simplified it a bit here.

If the first transition is to DST, the year starts in STD, transitions to DST, and ends with STD.
If the first transition is to STD, the opposite.

For every transition we take in local time the earliest clock value, and latest clock value (transition_min and transition_max).

If the transition crates a gap, all times exclusive do not exist (*_transition_min..*_transition_max).

If the transition crates an overlap, all times inclusive do not exist [*_transition_min..*_transition_max].

The time in between the transitions should have the remaining comparison operator (i.e. > if the transition uses <=).

The result to return in LocalResult::Single is straightforward, dst_offset or std_offset as appropriate.

The correct order in LocalResult::Ambiguous is the offset right before the transition, then the offset right after.

Hope that helps for the review.

This is a delicate function. I'd feel better if there was a fn test_lookup_with_dst_transitions that exercises the logic of every branch within, and of any corner cases.

I'm guessing the test case below verify_against_tz_specific_local_time_to_system_time() might also do that. I think it'd be an improvement to have a 1:1 test case just for this function.

Added a test 👍

Can you lay your explanation here down in a comment?

If this is created to replace existing logic for the Unix time zone logic, I think we should switch the Unix implementation in the same commit so that it's easier for me to compare the new code with the old code.

jtmoon79 · 2023-04-30T03:55:43Z

src/offset/local/mod.rs

+
+#[cfg(any(unix, windows))]
+impl TzInfo {
+    fn lookup_with_dst_transitions(&self, dt: &NaiveDateTime) -> LocalResult<FixedOffset> {


This is a delicate function. I'd feel better if there was a fn test_lookup_with_dst_transitions that exercises the logic of every branch within, and of any corner cases.

I'm guessing the test case below verify_against_tz_specific_local_time_to_system_time() might also do that. I think it'd be an improvement to have a 1:1 test case just for this function.

src/offset/local/tz_info/rule.rs

src/offset/local/windows.rs

pitdicker · 2023-04-30T08:31:35Z

@jtmoon79 Thank you for the review again!

I am a slow learning when it comes to adding tests 🤣. This will take some work.

pitdicker · 2023-05-04T07:33:51Z

Now that Windows supports for greater year ranges, it might overflow when near the limits of NaiveDateTime.
When adding a test for creating a DateTime<Local> with NaiveDateTime::{MIN, MAX} I hit multiple fun issues.

It panicked on overflow instead of returning LocalResult::None.
The Debug implementation of DateTime can panic (not something I want to fix in this PR).
FixedOffset has the same issue.

This can be fixed inside the provided implementation of TimeZone::from_local_datetime, which I did.

But Local doesn't use that, the code is duplicated.
Local also does other things in a roundabout way, creating a DateTime just to extract the offset in offset_from_local_datetime for example. I would like to clean this up, but that would complicate this PR even more. I'll wait until after this is merged.

pitdicker · 2023-05-05T11:16:21Z

Local also does other things in a roundabout way, creating a DateTime just to extract the offset in offset_from_local_datetime for example. I would like to clean this up, but that would complicate this PR even more. I'll wait until after this is merged.

Thought better of this. I have removed the commit that fixes handling overflow when near the limits of NaiveDateTime.

I have added a commit to this PR removing all the roundabout stuff in the local module, including in stub.rs. In my opinion it is quite a bit more readable.

Now that all inner modules return offsets, any fix for overflow handling at the TimeZone trait level will automatically be picked up without merge conflicts.

ChrisDenton · 2024-01-27T19:08:45Z

Thank you so much for working on this. The lasted update does simplify things and I find it much easier to follow.

jtmoon79 · 2024-01-27T19:50:31Z

src/offset/local/mod.rs

+        ];
+        compare_lookup(&transitions, 2023, 3, 26, 2, 0, 0, LocalResult::Single(std));
+        compare_lookup(&transitions, 2023, 10, 29, 3, 0, 0, LocalResult::Single(std));
+    }


Is it worthwhile adding edge cases to this test function test_lookup_with_dst_transitions? i.e. should this also test transitions at or near DateTime::MIN and DateTime::MAX (or whatever the minimum and maximum should on Windows)? At first glance, that looks important to me.

jtmoon79 · 2024-01-27T19:53:21Z

src/offset/local/win_bindings.rs

 ::windows_targets::link!("kernel32.dll" "system" fn SystemTimeToFileTime(lpsystemtime : *const SYSTEMTIME, lpfiletime : *mut FILETIME) -> BOOL);
 ::windows_targets::link!("kernel32.dll" "system" fn SystemTimeToTzSpecificLocalTime(lptimezoneinformation : *const TIME_ZONE_INFORMATION, lpuniversaltime : *const SYSTEMTIME, lplocaltime : *mut SYSTEMTIME) -> BOOL);
 ::windows_targets::link!("kernel32.dll" "system" fn TzSpecificLocalTimeToSystemTime(lptimezoneinformation : *const TIME_ZONE_INFORMATION, lplocaltime : *const SYSTEMTIME, lpuniversaltime : *mut SYSTEMTIME) -> BOOL);
 pub type BOOL = i32;
+pub type BOOLEAN = u8;
+#[repr(C)]
+pub struct DYNAMIC_TIME_ZONE_INFORMATION {


This struct seems to have dropped out of the sky.

I prefer to give the reader some help finding the source material for magic values and structures, e.g.

/// struct from Windows Win32 API /// <https://learn.microsoft.com/en-us/windows/win32/api/timezoneapi/ns-timezoneapi-dynamic_time_zone_information>

This struct seems to have dropped out of the sky.

Quite close, the entire file gets generated by windows-bindgen.

jtmoon79 · 2024-01-27T19:54:48Z

src/offset/local/windows.rs


+// We don't use `SystemTimeToTzSpecificLocalTime` because it doesn't support the same range of dates


This is a great comment.

jtmoon79 · 2024-01-27T19:57:04Z

src/offset/local/windows.rs

-    let local_secs = system_time_as_unix_seconds(&local_time)?;
-    let offset = (local_secs - utc_secs) as i32;
-    Ok(FixedOffset::east_opt(offset).unwrap())
+// The basis for Windows timezone and DST support has been in place since Windows 2000. It does not


Great info @ChrisDenton . I suggest placing your reply into the code as a code comment including the links.

pitdicker · 2024-01-27T21:11:09Z

I think the logic is pretty much what I would want it to be. The logic in lookup_with_dst_transitions is reusable on Unix, I have another branch to do so (but want to keep this one focused on just Windows).

If anyone wants to help out with adding some tests I would deeply appreciate it. But otherwise I'll get to it.
I think it would be nice to test lookup_with_dst_transitions with just one transition date, and with three.
And to add a pretty artificial test for lookup_with_dst_transitions with transitions right around midnight at NaiveDateTime::MIN|MAX.

pitdicker · 2024-02-04T08:00:10Z

Added tests for lookup_with_dst_transitions with transitions right around midnight at NaiveDateTime::MIN|MAX, and with a single transitions. Why did I put this off, only an hour work? Anyway ready for review.

djc · 2024-02-07T09:49:40Z

src/offset/local/windows.rs

+// The basis for Windows timezone and DST support has been in place since Windows 2000. It does not
+// allow for complex rules like the IANA timezone database:
+// - A timezone has the same base offset the whole year.
+// - There some to be either zero or two DST transitions (but we support having just one).


"There some to be"?

djc · 2024-02-07T09:51:09Z

src/offset/local/windows.rs

-    let local_secs = system_time_as_unix_seconds(&local_time)?;
-    let offset = (local_secs - utc_secs) as i32;
-    Ok(FixedOffset::east_opt(offset).unwrap())
+fn tz_info_for_year(year: i32) -> Option<TzInfo> {


Let's make this a method? TzInfo::for_year()?

djc · 2024-02-07T09:51:58Z

src/offset/local/windows.rs

-        wSecond: dt.second() as u16,
-        // Valid values: 0-999
-        wMilliseconds: 0,
+fn systemtime_to_naive_dt(st: SYSTEMTIME, year: i32) -> Option<NaiveDateTime> {


I think I preferred the old name? (And don't see a strong reason to change it.)

We didn't have anything like this function before. I don't mind changing the name to something else.

Ah, sorry -- I was confused because the diff shows this right next to system_time_from_naive_date_time() and I missed the from/to difference. I think system_time is more idiomatic than systemtime, and would probably prefer to spell out date_time over dt for this function name.

src/offset/local/windows.rs

src/offset/local/mod.rs

djc · 2024-02-07T09:53:46Z

src/datetime/tests.rs

+#[test]
+#[cfg(feature = "clock")]
+fn test_datetime_before_windows_api_limits() {
+    // dt corresponds to `FILETIME = 147221225472` from issue 651.


Can you refer to the issue by making it a link?

djc · 2024-02-10T09:57:46Z

src/offset/local/windows.rs

-        // Valid values: 0-999
-        wMilliseconds: 0,
+impl TzInfo {
+    fn tz_info_for_year(year: i32) -> Option<TzInfo> {


As suggested previously, let's call this for_year() -- no need to duplicate the type name in the method name.

Sorry, only now I get your intention. Yes, that is better.

And add test against previous implementation

pitdicker · 2024-02-10T18:38:45Z

🎉

Happy to see this completed. Thank you @djc, @ChrisDenton and others!

ChrisDenton · 2024-02-12T09:18:40Z

Oh wow! Thanks so much for working on this @pitdicker, you're a star!

pitdicker force-pushed the win_timezone branch from 88d07ec to d44a068 Compare April 17, 2023 13:50

pitdicker force-pushed the win_timezone branch from d44a068 to c61e32b Compare April 17, 2023 14:05

pitdicker force-pushed the win_timezone branch from 3356a22 to d80d7e2 Compare April 17, 2023 18:39

djc reviewed Apr 18, 2023

View reviewed changes

src/offset/local/tz_info/timezone.rs Outdated Show resolved Hide resolved

src/offset/local/mod.rs Outdated Show resolved Hide resolved

pitdicker force-pushed the win_timezone branch 3 times, most recently from 2b1bb99 to d3be65b Compare April 19, 2023 05:27

nekevss reviewed Apr 20, 2023

View reviewed changes

src/offset/local/windows.rs Outdated Show resolved Hide resolved

nekevss reviewed Apr 20, 2023

View reviewed changes

src/datetime/tests.rs Show resolved Hide resolved

pitdicker mentioned this pull request Apr 25, 2023

Fix warnings when running tests on Windows #1038

Merged

pitdicker commented Apr 27, 2023

View reviewed changes

jtmoon79 suggested changes Apr 30, 2023

View reviewed changes

pitdicker force-pushed the win_timezone branch 3 times, most recently from f078179 to 87b45ba Compare May 4, 2023 07:25

pitdicker force-pushed the win_timezone branch from 87b45ba to f31f73d Compare May 5, 2023 11:10

pitdicker force-pushed the win_timezone branch from f31f73d to 1aeadf1 Compare May 5, 2023 11:18

pitdicker mentioned this pull request May 6, 2023

Fix DateTime out-of-range panics #1048

Closed

pitdicker force-pushed the win_timezone branch from d6e8c7a to 47f240c Compare January 27, 2024 18:51

jtmoon79 reviewed Jan 27, 2024

View reviewed changes

pitdicker force-pushed the win_timezone branch 3 times, most recently from 1d930f1 to cd740d0 Compare January 27, 2024 21:05

pitdicker marked this pull request as ready for review January 27, 2024 21:11

pitdicker mentioned this pull request Jan 31, 2024

Adjusts windows local for millisecond and year #997

Closed

kubouch mentioned this pull request Feb 3, 2024

Problematic Dependencies nushell/nushell#11721

Open

pitdicker force-pushed the win_timezone branch 2 times, most recently from 5905846 to 9cc4572 Compare February 4, 2024 07:43

pitdicker mentioned this pull request Feb 7, 2024

Recognize our review capacity is limited for non-trivial work #1417

Open

djc reviewed Feb 7, 2024

View reviewed changes

pitdicker force-pushed the win_timezone branch 2 times, most recently from 846a6ff to f6fd7fa Compare February 7, 2024 14:40

djc reviewed Feb 10, 2024

View reviewed changes

djc approved these changes Feb 10, 2024

View reviewed changes

pitdicker added 4 commits February 10, 2024 19:20

Windows: rewrite using GetTimeZoneInformationForYear

7e9a472

And add test against previous implementation

Remove obsolete test

7fdd17f

Extend test to more distant dates

38edc49

Add test for issue 651

3d7daa5

pitdicker force-pushed the win_timezone branch from f6fd7fa to 3d7daa5 Compare February 10, 2024 18:20

pitdicker merged commit d1cf0e9 into chronotope:main Feb 10, 2024
29 of 35 checks passed

pitdicker deleted the win_timezone branch February 10, 2024 18:37

pitdicker mentioned this pull request Feb 10, 2024

Prepare 0.4.34 #1424

Merged

pitdicker mentioned this pull request Feb 19, 2024

0.5.x: Timezone conversion result (LocalResult) #1448

Open


		// We don't use `SystemTimeToTzSpecificLocalTime` because it doesn't support the same range of dates

Windows: base implementation on GetTimeZoneInformationForYear #1017

Windows: base implementation on GetTimeZoneInformationForYear #1017

Conversation

pitdicker commented Apr 17, 2023 • edited

djc commented Apr 17, 2023

esheppa commented Apr 17, 2023

djc commented Apr 17, 2023

pitdicker commented Apr 17, 2023 • edited

esheppa commented Apr 17, 2023

pitdicker commented Apr 17, 2023

djc left a comment

Choose a reason for hiding this comment

pitdicker commented Apr 19, 2023

pitdicker commented Apr 24, 2023

pitdicker commented Apr 24, 2023

Choose a reason for hiding this comment

jtmoon79 Apr 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtmoon79 Apr 30, 2023 • edited

Choose a reason for hiding this comment

pitdicker commented Apr 30, 2023

pitdicker commented May 4, 2023

pitdicker commented May 5, 2023 • edited

ChrisDenton commented Jan 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitdicker commented Jan 27, 2024

pitdicker commented Feb 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitdicker commented Feb 10, 2024

🎉

ChrisDenton commented Feb 12, 2024

Windows: base implementation on `GetTimeZoneInformationForYear` #1017

Windows: base implementation on `GetTimeZoneInformationForYear` #1017

pitdicker commented Apr 17, 2023 •

edited

pitdicker commented Apr 17, 2023 •

edited

jtmoon79 Apr 30, 2023 •

edited

jtmoon79 Apr 30, 2023 •

edited

pitdicker commented May 5, 2023 •

edited