ci: add performance benchmarks

[agostbiro]

Adds automated performance regression checks for PRs and historic progress tracking in main. The benchmark replays RPC calls captured in third-party test suites.

Runner

We're using a self-hosted Github Action runner to run benchmarks in a reproducible environment on a bare metal instance with an Intel(R) Xeon(R) E-2276G CPU @ 3.80GHz cpu. The runner is executed with an unprivileged user and standard security hardening was applied to the instance. The runner is ran as a systemd service with automatic restarts and reboot if many restarts fail.

I spent some time trying find a managed CI provider with a deterministic execution environment, but couldn't, so it seems we're stuck with self-hosted runners at least mid term.

Reports

We are using github-action-benchmark to track historic progress and raise regression alerts on PRs.

Historic

Benchmark data is to be stored in a separate repo. The Github pages in this repo will show historic benchmark data for main like this. (The example only includes the seaport scenario as the entire benchmark run is slow).

Regressions in PRs

When a performance regression is detected in a PR based on a preconfigured threshold and compared to the latest commit in main, the following error is displayed in the PR:

Clicking into the workflow, the following table is displayed giving information about the magnitude of the regression (the Markdown isn't rendered properly, this is a bug in github-action-benchmark):

It is a failure if there is a regression on any of the metrics.

Snapshot testing

The benchmark workflow checks that the same RPC calls succeed and fail as when the scenario was collected. If the snapshot doesn't match, the workflow fails and the difference is reported.

TODOs for this PR

• Update expected failures snapshot in the repo to include all scenarios
• Calibrate regression alert threshold percentage
• Create benchmark data repo under NomicFoundation and configure permissions and self-hosted runner in NomicFoundation/hardhat

Limitations/things to improve in future

• Full run takes an hour and there is only one runner so tasks might queue up. We could improve things by making the benchmark run faster and by introducing more runners.
• Only runs for team members’ and collaborators’ PRs as it's unsafe to allow third-parties to run code on self-hosted runners according to Github.
• Currently we rerun the benchmark in main even if it was already ran in a PR that was merged. This could be probably avoided.
• Can't use Swatinem/rust-cache@v2 in the workflow as it needs sudo on the runner which is too dangerous. We should look into alternatives to prevent recompiling from scratch on every run.
• The regression error message doesn’t render Markdown properly. I’ll open an issue about this in github-action-benchmark.
• github-action-benchmark has a feature to comment on the PR which would allow seeing the regression results easier, but this is blocked by the action using the same Github token to push to the benchmark result repo and to comment. We’d need to open a PR in the action repo to support it.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 082edb9

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
hardhat	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 19, 2024 4:38pm

[agostbiro]

Only running for changes to EDR since it's very slow and there is only one runner + we'll do this anyway once EDR is a separate repo

[agostbiro]

Save this to disk instead of stdout since multiple commands operate on it

[Wodann]

LGTM! Thank you for the clear explanation in the PR description. That was very helpful!

I just have one question to clarify that my understanding of the GitHub workflow is correct.

[agostbiro]

Follow up: NomicFoundation/edr#337