Error: cannot finalize before all snapshot resources are uploaded

[ozyx]

The problem

We encountered this percy client error on a recent build which caused it to fail:

[percy:client] Finalizing snapshot 1259268152... (269ms)
[percy:core] Uploading 2 snapshots...[percy:core] Uploading 2 snapshots...[percy:core] Uploading 2 snapshots...[percy:core] Uploading 2 snapshots...[percy:core] Uploading 2 snapshots...[percy:core] Uploading 2 snapshots...[percy:core] Uploading 2 snapshots...[percy:core] Uploading 2 snapshots...[percy:core] Uploading 0 snapshots...[percy:core] Uploading 0 snapshots...[percy:core:browser] Closing browser (139ms)
[percy:core:browser] Browser closed (22ms)
[percy:client] Finalizing build 22594096... (0ms)
[percy:cli] Error: Finalizing build 22594096 failed: cannot finalize before all snapshot resources are uploaded. This is likely a client error, please make sure that content for all SHAs in 'missing-resources' from the snapshot response are uploaded before calling finalize.
    at handleFinished (file:///root/project/node_modules/@percy/client/dist/utils.js:176:17)
    at IncomingMessage.<anonymous> (file:///root/project/node_modules/@percy/client/dist/utils.js:193:27)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) (185ms)

Exited with code exit status 1

This happened on a build which was introducing no functional changes, nor changes to our visual tests.

Environment

• Node version: 16.18.0
• @percy/cli version: 1.11.0
• Version of Percy SDK you’re using:
• If needed, a build or snapshot ID:
• OS version: Ubuntu 20.04.5 LTS
• Type of shell command-line [interface]:

Details

See above

Debug logs

More detailed logs can be found in the build logs here. (We run percy cli with debug mode on)

Code to reproduce issue

N/A, happened at random.

[thomasmattheussen]

We're seeing the same problem too since last week.

[itsjwala]

can you please update the SDK to v1.14, Please also share SDK logs with verbose mode on https://docs.percy.io/docs/debugging-sdks#debug-vs-verbose-logging

[thomasmattheussen]

I was already using 1.14

Logs here: https://gist.githubusercontent.com/thomasmattheussen/396d8ae7b824d07c81ded3bb3228011a/raw/fd801880d29f1378f67b02ed1f648052ac9f7669/percy%2520logs

[thomasmattheussen]

Hi, just wondering if you could give an update? This is severely hindering us.

[patrickberkeley]

We're seeing this problem as well.

As with @ozyx, our tests run on circle ci. We're not running in parallel. The command we run is yarn exec percy exec --verbose -- cypress run. The verbose error output is:

Exit code: 1
Command: percy
Arguments: exec cypress run
Directory: /home/circleci/app/lib/stone_messaging/maizzle
Output:

info Visit https://yarnpkg.com/en/docs/cli/exec for documentation about this command.
===KILLING PHX SERVER===

Exited with code exit status 1

We upgraded to @percy/cli v1.16.0, but still see intermittent failures.

[samarsault]

Can you try setting the asset discovery concurrency to 2 (ref)?

[thomasmattheussen]

yeah, that doesn't fix it.

[colinrotherham]

Hope this helps

We see the same issue quite frequently:

https://github.com/alphagov/govuk-frontend/actions/runs/5197923021/jobs/9374305996
https://github.com/alphagov/govuk-frontend/actions/runs/5199289909/jobs/9376530155
https://github.com/alphagov/govuk-frontend/actions/runs/5186189686/jobs/9347025798
https://github.com/alphagov/govuk-frontend/actions/runs/5199546425/jobs/9377108464

We run a Node.js script with concurrent calls to percySnapshot() via percy exec --

Output always shows 40+ snapshots taken:

[percy] Snapshot taken: js: back-link
[percy] Snapshot taken: js: accordion
[percy] Snapshot taken: js: breadcrumbs
[percy] Snapshot taken: js: button
[percy] Snapshot taken: no-js: button
[percy] Snapshot taken: no-js: accordion
[percy] Snapshot taken: js: cookie-banner
[percy] Snapshot taken: js: checkboxes
[percy] Snapshot taken: js: date-input
[percy] Snapshot taken: js: character-count
[percy] Snapshot taken: no-js: checkboxes
[percy] Snapshot taken: no-js: character-count
[percy] Snapshot taken: js: error-message
[percy] Snapshot taken: js: details
[percy] Snapshot taken: js: fieldset
[percy] Snapshot taken: js: error-summary
[percy] Snapshot taken: no-js: details
[percy] Snapshot taken: no-js: error-summary
[percy] Snapshot taken: js: footer
[percy] Snapshot taken: js: hint
[percy] Snapshot taken: js: file-upload
[percy] Snapshot taken: js: header
[percy] Snapshot taken: no-js: header
[percy] Snapshot taken: js: label
[percy] Snapshot taken: js: input
[percy] Snapshot taken: js: notification-banner
[percy] Snapshot taken: js: inset-text
[percy] Snapshot taken: no-js: notification-banner
[percy] Snapshot taken: js: panel
[percy] Snapshot taken: js: radios
[percy] Snapshot taken: js: phase-banner
[percy] Snapshot taken: js: pagination
[percy] Snapshot taken: no-js: radios
[percy] Snapshot taken: js: skip-link
[percy] Snapshot taken: js: summary-list
[percy] Snapshot taken: js: table
[percy] Snapshot taken: js: select
[percy] Snapshot taken: no-js: skip-link
[percy] Snapshot taken: js: warning-text
[percy] Snapshot taken: js: tag
[percy] Snapshot taken: js: tabs
[percy] Snapshot taken: js: textarea
[percy] Snapshot taken: no-js: tabs
[percy] Snapshot taken: js: text-alignment (example)
[percy] Snapshot taken: js: typography (example)

Then for most test runs Percy finalises without issue, although the snapshot count is always incorrect:

[percy] Uploading 2 snapshots...
[percy] Finalized build #4931: https://percy.io/55ba5e1a/govuk-frontend/builds/28024895

But we see frequent failures as shown in the links above:

[percy] Uploading 3 snapshots...
[percy] Error: Finalizing build 28024413 failed: cannot finalize before all snapshot resources are uploaded. This is likely a client error, please make sure that content for all SHAs in 'missing-resources' from the snapshot response are uploaded before calling finalize.

We worked around this problem in the past by adding --parallel

• Bug: Percy uploads should complete before “finalizing” alphagov/govuk-frontend#3036

But after the @percy/cli v1.24.2 update we started seeing finalize run twice:

• Remove Percy parallel builds due to double “finalize” issue alphagov/govuk-frontend#3670

[colinrotherham]

I've done some testing this afternoon and it appears to be a bug in npx percy exec

By enabling --verbose logging it's helped to add waitForPercyIdle() from @percy/sdk-utils to avoid any race conditions:

import { waitForPercyIdle } from '@percy/sdk-utils'

// Wait for Percy to finish
await waitForPercyIdle()

• Wait for Percy to idle before screenshot task finishes alphagov/govuk-frontend#3767 (comment)

[itsjwala]

hey Folks!

Appreciate your patience on this issue.

we’ve made some changes in our CLI release in 1.27.6-alpha.0 please try this out and let us know the stability. ( Note: with this you can also try removing out the concurrency config which was suggested earlier and have it default)

If it works we’ll release it in a stable channel.

[wannesdebacker]

This version fixed this issue for us (was broken in the previous (stable) version). Thank you!

[colinrotherham]

Thanks @itsjwala, looks like it's working:
https://github.com/alphagov/govuk-frontend/actions/runs/7003326836/job/19049083739#step:8:90

But the race condition didn't happen often enough to be fully confident

For example, removing the waitForPercyIdle() workaround logs an incorrect snapshot count from the CLI again:

[10:36:59] Finished 'screenshots' after 32 s
[percy] Uploading 2 snapshots...
[percy] Finalized build #7223: https://percy.io/55ba5e1a/govuk-frontend/builds/31376442

Our builds are public and actually have 66 snapshots total

[itsjwala]

Hey @colinrotherham 👋

Uploading x snapshots... is usually useful when you're seeing logs being tailed. and we can see the race condition happening here where 2 snapshots were still in the upload queue, though our fix in alpha delayed the build finalize.

You can ignore this logline.