Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(e2e): set up display documents e2e 2 #8908

Merged
merged 11 commits into from
Mar 14, 2025
Merged

Conversation

RitaDias
Copy link
Contributor

@RitaDias RitaDias commented Mar 12, 2025

Description

Note

The code is the same as the previous PR I had open, I just moved the tests to a single file

Added e2e tests to test document displayed from #8759
Same code as #8828 with minor changes

In order to reduce flake:

  • Tests for the releases are kept in a serial order to make sure BeforeAll and AfterAll behave in the manner expected
  • Skip firefox to avoid flakiness

What to review

Do the tests make sense?

Notes for release

N/A

Copy link

vercel bot commented Mar 12, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
page-building-studio ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 14, 2025 10:32am
performance-studio ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 14, 2025 10:32am
test-studio ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 14, 2025 10:32am
2 Skipped Deployments
Name Status Preview Comments Updated (UTC)
studio-workshop ⬜️ Ignored (Inspect) Visit Preview Mar 14, 2025 10:32am
test-next-studio ⬜️ Ignored (Inspect) Mar 14, 2025 10:32am

Copy link
Contributor

No changes to documentation

Copy link
Contributor

github-actions bot commented Mar 12, 2025

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 42.01% 54448 / 129588
🔵 Statements 42.01% 54448 / 129588
🔵 Functions 46.89% 2729 / 5820
🔵 Branches 79.4% 10199 / 12844
File CoverageNo changed files found.
Generated in workflow #32313 for commit efde74a by the Vitest Coverage Report Action

Copy link
Contributor

github-actions bot commented Mar 12, 2025

⚡️ Editor Performance Report

Updated Fri, 14 Mar 2025 10:48:31 GMT

Benchmark reference
latency of sanity@latest
experiment
latency of this branch
Δ (%)
latency difference
article (title) 23.3 efps (43ms) 25.6 efps (39ms) -4ms (-9.3%)
article (body) 67.1 efps (15ms) 67.6 efps (15ms) -0ms (-/-%)
article (string inside object) 25.6 efps (39ms) 27.0 efps (37ms) -2ms (-5.1%)
article (string inside array) 22.2 efps (45ms) 23.3 efps (43ms) -2ms (-4.4%)
recipe (name) 47.6 efps (21ms) 47.6 efps (21ms) +0ms (-/-%)
recipe (description) 50.0 efps (20ms) 55.6 efps (18ms) -2ms (-10.0%)
recipe (instructions) 99.9+ efps (5ms) 99.9+ efps (5ms) +0ms (-/-%)
synthetic (title) 19.8 efps (51ms) 19.4 efps (52ms) +1ms (+2.0%)
synthetic (string inside object) 20.0 efps (50ms) 19.2 efps (52ms) +2ms (+4.0%)

efps — editor "frames per second". The number of updates assumed to be possible within a second.

Derived from input latency. efps = 1000 / input_latency

Detailed information

🏠 Reference result

The performance result of sanity@latest

Benchmark latency p75 p90 p99 blocking time test duration
article (title) 43ms 47ms 62ms 242ms 540ms 11.6s
article (body) 15ms 19ms 35ms 204ms 310ms 5.8s
article (string inside object) 39ms 44ms 46ms 56ms 170ms 7.3s
article (string inside array) 45ms 47ms 53ms 71ms 219ms 7.3s
recipe (name) 21ms 22ms 24ms 42ms 0ms 9.3s
recipe (description) 20ms 21ms 22ms 28ms 0ms 4.8s
recipe (instructions) 5ms 7ms 7ms 13ms 0ms 3.2s
synthetic (title) 51ms 61ms 86ms 516ms 1418ms 15.8s
synthetic (string inside object) 50ms 53ms 56ms 229ms 685ms 8.0s

🧪 Experiment result

The performance result of this branch

Benchmark latency p75 p90 p99 blocking time test duration
article (title) 39ms 43ms 45ms 245ms 404ms 10.1s
article (body) 15ms 17ms 27ms 225ms 332ms 5.8s
article (string inside object) 37ms 39ms 41ms 140ms 156ms 6.6s
article (string inside array) 43ms 46ms 48ms 84ms 352ms 7.2s
recipe (name) 21ms 23ms 26ms 32ms 0ms 7.3s
recipe (description) 18ms 19ms 21ms 26ms 0ms 4.7s
recipe (instructions) 5ms 7ms 8ms 15ms 0ms 3.1s
synthetic (title) 52ms 59ms 71ms 408ms 1138ms 13.5s
synthetic (string inside object) 52ms 53ms 57ms 80ms 780ms 7.7s

📚 Glossary

column definitions

  • benchmark — the name of the test, e.g. "article", followed by the label of the field being measured, e.g. "(title)".
  • latency — the time between when a key was pressed and when it was rendered. derived from a set of samples. the median (p50) is shown to show the most common latency.
  • p75 — the 75th percentile of the input latency in the test run. 75% of the sampled inputs in this benchmark were processed faster than this value. this provides insight into the upper range of typical performance.
  • p90 — the 90th percentile of the input latency in the test run. 90% of the sampled inputs were faster than this. this metric helps identify slower interactions that occurred less frequently during the benchmark.
  • p99 — the 99th percentile of the input latency in the test run. only 1% of sampled inputs were slower than this. this represents the worst-case scenarios encountered during the benchmark, useful for identifying potential performance outliers.
  • blocking time — the total time during which the main thread was blocked, preventing user input and UI updates. this metric helps identify performance bottlenecks that may cause the interface to feel unresponsive.
  • test duration — how long the test run took to complete.

@RitaDias RitaDias force-pushed the sapp-2388-e2e-tests-2 branch from dd4db3e to 4ab317e Compare March 14, 2025 07:18
@RitaDias RitaDias force-pushed the sapp-2388-e2e-tests-2 branch from 4ab317e to 7c07c5f Compare March 14, 2025 07:29
Copy link
Member

@bjoerge bjoerge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great addition, thank you for @RitaDias! Left some comments, mostly minor stuff, please take a look. Would be great if we could avoid the duplication in the test.skip() condition.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
@RitaDias
Copy link
Contributor Author

@bjoerge moved the skip condition to its own method, call it wherever it's needed :)

Copy link
Member

@bjoerge bjoerge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 🎉 🎉

@RitaDias RitaDias merged commit 621fc11 into next Mar 14, 2025
60 checks passed
@RitaDias RitaDias deleted the sapp-2388-e2e-tests-2 branch March 14, 2025 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants