Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(criterion_compat): fork criterion and add walltime support #85

Merged
merged 6 commits into from
Mar 10, 2025

Conversation

not-matthias
Copy link
Contributor

No description provided.

Copy link

codspeed-hq bot commented Mar 5, 2025

CodSpeed Instrumentation Performance Report

Merging #85 will degrade performances by 16.51%

Comparing cod-240-walltime-criterion-support (9102482) with main (17a5e27)

Summary

⚡ 1 improvements
❌ 6 regressions
✅ 156 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
from_elem[16384] 19 µs 19.6 µs -3.12%
from_elem[2048] 3.8 µs 4.1 µs -5.48%
from_elem[4096] 6.6 µs 7.2 µs -8.77%
from_elem[8192] 10.7 µs 11.4 µs -5.48%
from_elem_decimal[2048] 3.4 µs 4.1 µs -16.51%
mut_borrow 895.8 ns 866.7 ns +3.37%
recursive_memoized[BTreeMap<u64, u64>, 0] 371.7 ns 400.8 ns -7.28%

Copy link

codspeed-hq bot commented Mar 5, 2025

CodSpeed Walltime Performance Report

Merging #85 will degrade performances by 72.66%

Comparing cod-240-walltime-criterion-support (9102482) with main (17a5e27)

Summary

⚡ 1 improvements
❌ 16 regressions
✅ 92 untouched benchmarks
🆕 36 new benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
🆕 sum_fold N/A 304 ns N/A
🆕 build_vec N/A 204 ns N/A
🆕 Iterative N/A 21 ns N/A
🆕 Recursive N/A 52.3 µs N/A
🆕 Iterative[20] N/A 21 ns N/A
🆕 Iterative[21] N/A 22 ns N/A
🆕 Recursive[20] N/A 52.3 µs N/A
🆕 Recursive[21] N/A 84.7 µs N/A
🆕 fibonacci_custom_measurement N/A 421 ns N/A
🆕 large_drop N/A 169.9 µs N/A
🆕 small_drop N/A < 1 ns N/A
🆕 large_setup N/A 13 ns N/A
🆕 small_setup N/A 1 ns N/A
🆕 iter_with_setup N/A 39 ns N/A
🆕 iter N/A < 1 ns N/A
🆕 iter_batched_large_input N/A < 1 ns N/A
🆕 iter_batched_per_iteration N/A 40 ns N/A
🆕 iter_batched_ref_large_input N/A < 1 ns N/A
🆕 iter_batched_ref_per_iteration N/A 40 ns N/A
🆕 iter_batched_ref_small_input N/A < 1 ns N/A
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

@not-matthias not-matthias force-pushed the cod-240-walltime-criterion-support branch 2 times, most recently from 1a4962d to 1cfeccb Compare March 5, 2025 16:08
Copy link
Contributor

@GuillaumeLagrange GuillaumeLagrange left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Biggest change is how we handle the fork which feels a bit uncomfortable, I would like us to be able to have more control over clippy to make diff completely minimal with original criterion

@not-matthias not-matthias force-pushed the cod-240-walltime-criterion-support branch 5 times, most recently from a6e0a7c to f5ec4c8 Compare March 6, 2025 14:16
Copy link
Contributor

@GuillaumeLagrange GuillaumeLagrange left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very small comments, we can merge after this !

@not-matthias not-matthias force-pushed the cod-240-walltime-criterion-support branch 4 times, most recently from 31adf46 to 4abfae8 Compare March 6, 2025 17:09
Copy link
Contributor

@GuillaumeLagrange GuillaumeLagrange left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Let's wait for the display issue with 0 time benchmarks before merging, I just want to sanity check that the perf regression reported by codspeed is expected/known and not caused by changes here, but can't really do it without the report page

@not-matthias
Copy link
Contributor Author

Verified execution time of benchmarks with 0s, by comparing the measured walltime with local results:

  • small_drop 143.00 ps
  • iter 128.60 ps
  • iter_batched_large_input 89.355 ps
  • iter_batched_ref_large_input 87.253 ps
  • etc.

Looks good, so I don't think we have a bug in there.

@art049
Copy link
Member

art049 commented Mar 7, 2025

Just some small structure details. Otherwise looks good

@not-matthias not-matthias force-pushed the cod-240-walltime-criterion-support branch 2 times, most recently from e553cb8 to 2bf7839 Compare March 7, 2025 16:50
@not-matthias
Copy link
Contributor Author

Link for the uv-walltime-criterion integration: https://codspeed.io/AvalancheHQ/uv-criterion-walltime/runs

@not-matthias not-matthias requested a review from art049 March 7, 2025 18:17
Copy link
Member

@art049 art049 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets go!

Verified

This commit was signed with the committer’s verified signature.
not-matthias Matthias
Signed-off-by: not-matthias <matthias@codspeed.io>

Verified

This commit was signed with the committer’s verified signature.
not-matthias Matthias
Signed-off-by: not-matthias <matthias@codspeed.io>

Verified

This commit was signed with the committer’s verified signature.
not-matthias Matthias
Signed-off-by: not-matthias <matthias@codspeed.io>

Verified

This commit was signed with the committer’s verified signature.
not-matthias Matthias
Signed-off-by: not-matthias <matthias@codspeed.io>

Verified

This commit was signed with the committer’s verified signature.
not-matthias Matthias
Signed-off-by: not-matthias <matthias@codspeed.io>

Verified

This commit was signed with the committer’s verified signature.
not-matthias Matthias
This is needed so that the submodule can be cloned on CI without having to setup SSH keys.

Signed-off-by: not-matthias <matthias@codspeed.io>
@not-matthias not-matthias force-pushed the cod-240-walltime-criterion-support branch from 2bf7839 to 9102482 Compare March 10, 2025 14:56
@not-matthias not-matthias merged commit 9102482 into main Mar 10, 2025
5 of 6 checks passed
@not-matthias not-matthias deleted the cod-240-walltime-criterion-support branch March 10, 2025 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants