improve turborepo caching #64493

sokra · 2024-04-15T09:32:15Z

What?

There is a race condition in the pull-build-cache script. When the remote cache entry was removed between the dry run and the real run, it will run the command and caches whatever is in the native folder.

Seems like there are some very old leftover files there which lead to an broken publish.

This changes the command to fail when there are no files and empties the folder before running the script. This should lead to pull-build-cache to failing instead.

Closes PACK-2957
Fixes #64468

ijjk · 2024-04-15T09:40:55Z

Failing test suites

Commit: 1c666c7

TURBOPACK=1 pnpm test test/integration/create-next-app/templates/app.test.ts (turbopack)

create-next-app --app (App Router) > should create JavaScript project with --js flag
create-next-app --app (App Router) > should create TypeScript project with --ts flag
create-next-app --app (App Router) > should create project inside "src" directory with --src-dir flag
create-next-app --app (App Router) > should create TailwindCSS project with --tailwind flag

Expand output

● create-next-app --app (App Router) › should create JavaScript project with --js flag

FetchError: request to http://localhost:37669/ failed, reason: connect ECONNREFUSED 127.0.0.1:37669

  at ClientRequest.<anonymous> (../node_modules/.pnpm/node-fetch@2.6.7/node_modules/node-fetch/lib/index.js:1491:11)

● create-next-app --app (App Router) › should create TypeScript project with --ts flag

FetchError: request to http://localhost:42841/ failed, reason: connect ECONNREFUSED 127.0.0.1:42841

  at ClientRequest.<anonymous> (../node_modules/.pnpm/node-fetch@2.6.7/node_modules/node-fetch/lib/index.js:1491:11)

● create-next-app --app (App Router) › should create project inside "src" directory with --src-dir flag

FetchError: request to http://localhost:44531/ failed, reason: connect ECONNREFUSED 127.0.0.1:44531

  at ClientRequest.<anonymous> (../node_modules/.pnpm/node-fetch@2.6.7/node_modules/node-fetch/lib/index.js:1491:11)

● create-next-app --app (App Router) › should create TailwindCSS project with --tailwind flag

FetchError: request to http://localhost:34357/ failed, reason: connect ECONNREFUSED 127.0.0.1:34357

  at ClientRequest.<anonymous> (../node_modules/.pnpm/node-fetch@2.6.7/node_modules/node-fetch/lib/index.js:1491:11)

Read more about building and testing Next.js in contributing.md.

TURBOPACK=1 pnpm test test/integration/create-next-app/templates/pages.test.ts (turbopack)

create-next-app --no-app (Pages Router) > should create JavaScript project with --js flag
create-next-app --no-app (Pages Router) > should create TypeScript project with --ts flag
create-next-app --no-app (Pages Router) > should create project inside "src" directory with --src-dir flag
create-next-app --no-app (Pages Router) > should create TailwindCSS project with --tailwind flag

Expand output

● create-next-app --no-app (Pages Router) › should create JavaScript project with --js flag

FetchError: request to http://localhost:35989/ failed, reason: connect ECONNREFUSED 127.0.0.1:35989

  at ClientRequest.<anonymous> (../node_modules/.pnpm/node-fetch@2.6.7/node_modules/node-fetch/lib/index.js:1491:11)

● create-next-app --no-app (Pages Router) › should create TypeScript project with --ts flag

FetchError: request to http://localhost:39235/ failed, reason: connect ECONNREFUSED 127.0.0.1:39235

  at ClientRequest.<anonymous> (../node_modules/.pnpm/node-fetch@2.6.7/node_modules/node-fetch/lib/index.js:1491:11)

● create-next-app --no-app (Pages Router) › should create project inside "src" directory with --src-dir flag

FetchError: request to http://localhost:36357/ failed, reason: connect ECONNREFUSED 127.0.0.1:36357

  at ClientRequest.<anonymous> (../node_modules/.pnpm/node-fetch@2.6.7/node_modules/node-fetch/lib/index.js:1491:11)

● create-next-app --no-app (Pages Router) › should create TailwindCSS project with --tailwind flag

FetchError: request to http://localhost:35629/ failed, reason: connect ECONNREFUSED 127.0.0.1:35629

  at ClientRequest.<anonymous> (../node_modules/.pnpm/node-fetch@2.6.7/node_modules/node-fetch/lib/index.js:1491:11)

Read more about building and testing Next.js in contributing.md.

pnpm test test/integration/edge-runtime-module-errors/test/module-imports.test.js

Edge runtime code with imports > Edge API statically importing node.js module > throws unsupported module error in dev at runtime and highlights the faulty line

Expand output

● Edge runtime code with imports › Edge API statically importing node.js module › throws unsupported module error in dev at runtime and highlights the faulty line

expect(received).toBe(expected) // Object.is equality

Expected: 500
Received: 404

  86 |       context.app = await launchApp(context.appDir, context.appPort, appOption)
  87 |       const res = await fetchViaHTTP(context.appPort, url)
> 88 |       expect(res.status).toBe(500)
     |                          ^
  89 |
  90 |       const text = await res.text()
  91 |       await check(async () => {

  at Object.toBe (integration/edge-runtime-module-errors/test/module-imports.test.js:88:26)

Read more about building and testing Next.js in contributing.md.

pnpm test-dev test/development/middleware-errors/index.test.ts

middleware - development errors > when there is a compilation error after boot > logs the error correctly

Expand output

● middleware - development errors › when there is a compilation error after boot › logs the error correctly

TIMED OUT: success

undefined

Error: expect(received).toEqual(expected) // deep equality

Expected: 2
Received: 3

  642 |
  643 |   if (hardError) {
> 644 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content + '\n\n' + lastErr)
      |           ^
  645 |   }
  646 |   return false
  647 | }

  at check (lib/next-test-utils.ts:644:11)
  at Object.<anonymous> (development/middleware-errors/index.test.ts:291:9)

Read more about building and testing Next.js in contributing.md.

ijjk · 2024-04-15T09:43:49Z

Stats from current PR

Default Build

General

	vercel/next.js canary	vercel/next.js sokra/fix-next-caching	Change
buildDuration	14.1s	14.1s	N/A
buildDurationCached	8.7s	6.4s	N/A
nodeModulesSize	199 MB	199 MB	✓
nextStartRea..uration (ms)	398ms	402ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js sokra/fix-next-caching	Change
2453-HASH.js gzip	31.4 kB	31.4 kB	N/A
3304.HASH.js gzip	181 B	181 B	✓
3f784ff6-HASH.js gzip	53.7 kB	53.7 kB	✓
8299-HASH.js gzip	5.1 kB	5.1 kB	N/A
framework-HASH.js gzip	45.2 kB	45.2 kB	✓
main-app-HASH.js gzip	242 B	242 B	✓
main-HASH.js gzip	32.2 kB	32.2 kB	N/A
webpack-HASH.js gzip	1.68 kB	1.68 kB	N/A
Overall change	99.3 kB	99.3 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js sokra/fix-next-caching	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js sokra/fix-next-caching	Change
_app-HASH.js gzip	196 B	197 B	N/A
_error-HASH.js gzip	184 B	184 B	✓
amp-HASH.js gzip	505 B	505 B	✓
css-HASH.js gzip	324 B	325 B	N/A
dynamic-HASH.js gzip	2.5 kB	2.5 kB	N/A
edge-ssr-HASH.js gzip	258 B	258 B	✓
head-HASH.js gzip	352 B	352 B	✓
hooks-HASH.js gzip	370 B	371 B	N/A
image-HASH.js gzip	4.27 kB	4.27 kB	✓
index-HASH.js gzip	259 B	259 B	✓
link-HASH.js gzip	2.67 kB	2.67 kB	N/A
routerDirect..HASH.js gzip	314 B	312 B	N/A
script-HASH.js gzip	386 B	386 B	✓
withRouter-HASH.js gzip	309 B	309 B	✓
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	6.63 kB	6.63 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js sokra/fix-next-caching	Change
_buildManifest.js gzip	483 B	485 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js sokra/fix-next-caching	Change
index.html gzip	527 B	529 B	N/A
link.html gzip	538 B	541 B	N/A
withRouter.html gzip	521 B	523 B	N/A
Overall change	0 B	0 B	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js sokra/fix-next-caching	Change
edge-ssr.js gzip	95.6 kB	95.6 kB	N/A
page.js gzip	3.06 kB	3.06 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	vercel/next.js sokra/fix-next-caching	Change
middleware-b..fest.js gzip	623 B	626 B	N/A
middleware-r..fest.js gzip	155 B	156 B	N/A
middleware.js gzip	25.5 kB	25.5 kB	N/A
edge-runtime..pack.js gzip	839 B	839 B	✓
Overall change	839 B	839 B	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js sokra/fix-next-caching	Change
app-page-exp...dev.js gzip	171 kB	171 kB	✓
app-page-exp..prod.js gzip	97.5 kB	97.5 kB	✓
app-page-tur..prod.js gzip	99.2 kB	99.2 kB	✓
app-page-tur..prod.js gzip	93.5 kB	93.5 kB	✓
app-page.run...dev.js gzip	145 kB	145 kB	✓
app-page.run..prod.js gzip	92 kB	92 kB	✓
app-route-ex...dev.js gzip	21.5 kB	21.5 kB	✓
app-route-ex..prod.js gzip	15.2 kB	15.2 kB	✓
app-route-tu..prod.js gzip	15.2 kB	15.2 kB	✓
app-route-tu..prod.js gzip	14.9 kB	14.9 kB	✓
app-route.ru...dev.js gzip	21.1 kB	21.1 kB	✓
app-route.ru..prod.js gzip	14.9 kB	14.9 kB	✓
pages-api-tu..prod.js gzip	9.55 kB	9.55 kB	✓
pages-api.ru...dev.js gzip	9.82 kB	9.82 kB	✓
pages-api.ru..prod.js gzip	9.55 kB	9.55 kB	✓
pages-turbo...prod.js gzip	22.5 kB	22.5 kB	✓
pages.runtim...dev.js gzip	23.1 kB	23.1 kB	✓
pages.runtim..prod.js gzip	22.5 kB	22.5 kB	✓
server.runti..prod.js gzip	51.3 kB	51.3 kB	✓
Overall change	948 kB	948 kB	✓

build cache

	vercel/next.js canary	vercel/next.js sokra/fix-next-caching	Change
0.pack gzip	1.59 MB	1.58 MB	N/A
index.pack gzip	107 kB	107 kB	N/A
Overall change	0 B	0 B	✓

Diff details

Diff for middleware.js

Diff too large to display

Commit: 1c666c7

timneutkens · 2024-04-15T10:42:35Z

Worked: https://github.com/vercel/next.js/actions/runs/8687534499

### What? There is a race condition in the `pull-build-cache` script. When the remote cache entry was removed between the dry run and the real run, it will run the command and caches whatever is in the native folder. Seems like there are some very old leftover files there which lead to an broken publish. This changes the command to fail when there are no files and empties the folder before running the script. This should lead to pull-build-cache to failing instead. Closes PACK-2957 Fixes #64468

improve turborepo caching

edc3cb4

sokra requested review from timneutkens, ijjk, shuding, huozhi and a team as code owners April 15, 2024 09:32

sokra requested review from molebox and delbaoliveira and removed request for a team April 15, 2024 09:32

ijjk added created-by: Turbopack team PRs by the turbopack team type: next labels Apr 15, 2024

timneutkens approved these changes Apr 15, 2024

View reviewed changes

sokra marked this pull request as draft April 15, 2024 10:21

fix command

1c666c7

timneutkens marked this pull request as ready for review April 15, 2024 10:41

timneutkens merged commit 2766ebc into canary Apr 15, 2024
86 of 98 checks passed

timneutkens deleted the sokra/fix-next-caching branch April 15, 2024 10:42

timneutkens mentioned this pull request Apr 15, 2024

next dev and next build crashing in next@>=14.2.1-canary.4 on Linux #64468

Closed

github-actions bot added the locked label Apr 30, 2024

github-actions bot locked as resolved and limited conversation to collaborators Apr 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve turborepo caching #64493

improve turborepo caching #64493

sokra commented Apr 15, 2024 •

edited by timneutkens

ijjk commented Apr 15, 2024 •

edited

ijjk commented Apr 15, 2024 •

edited

timneutkens commented Apr 15, 2024

improve turborepo caching #64493

improve turborepo caching #64493

Conversation

sokra commented Apr 15, 2024 • edited by timneutkens

What?

ijjk commented Apr 15, 2024 • edited

Failing test suites

ijjk commented Apr 15, 2024 • edited

Stats from current PR

timneutkens commented Apr 15, 2024

sokra commented Apr 15, 2024 •

edited by timneutkens

ijjk commented Apr 15, 2024 •

edited

ijjk commented Apr 15, 2024 •

edited