`_worker.js/` directory support in Pages #2966

GregBrimble · 2023-03-30T23:59:59Z

What this PR solves / how to test:

This PR adds support for an _worker.js/ directory in Pages. It adds support for:

wrangler pages dev
wrangler pages functions build
wrangler pages publish

The entrypoint (_worker.js/index.js) is bundled, and the rest of the files in the _worker.js/ directory are uploaded with the default rules + all .js files are treated as ESM.

This will be treated as an undocumented API for the time being.

Author has included the following, where applicable:

Tests
Changeset (Changeset guidelines)

Reviewer has performed the following, where applicable:

Checked for inclusion of relevant tests
Checked for inclusion of a relevant changeset
Checked for creation of associated docs updates
Manually pulled down the changes and spot-tested

changeset-bot · 2023-03-31T00:00:03Z

🦋 Changeset detected

Latest commit: 6853fb7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
wrangler	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2023-03-31T00:02:19Z

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/4862724294/npm-package-wrangler-2966

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/2966/npm-package-wrangler-2966

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/4862724294/npm-package-wrangler-2966 dev path/to/script.js

Additional artifacts:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/4862724294/npm-package-cloudflare-pages-shared-2966

Note that these links will no longer work once the GitHub Actions artifact expires.

codecov · 2023-04-05T10:33:51Z

Codecov Report

Merging #2966 (6853fb7) into main (0a77990) will increase coverage by 0.87%.
The diff coverage is 77.96%.

@@            Coverage Diff             @@
##             main    #2966      +/-   ##
==========================================
+ Coverage   73.54%   74.42%   +0.87%     
==========================================
  Files         167      168       +1     
  Lines       10499    10624     +125     
  Branches     2807     2855      +48     
==========================================
+ Hits         7722     7907     +185     
+ Misses       2777     2717      -60

Impacted Files	Coverage Δ
packages/wrangler/src/pages/dev.ts	`18.14% <9.09%> (-0.64%)`	⬇️
packages/wrangler/src/dev/start-server.ts	`67.07% <81.81%> (+1.46%)`	⬆️
packages/wrangler/src/api/pages/publish.tsx	`87.93% <92.30%> (-0.26%)`	⬇️
packages/wrangler/src/api/dev.ts	`59.34% <100.00%> (-0.88%)`	⬇️
packages/wrangler/src/bundle.ts	`93.21% <100.00%> (+0.06%)`	⬆️
packages/wrangler/src/dev.tsx	`85.12% <100.00%> (+0.12%)`	⬆️
packages/wrangler/src/entry.ts	`98.40% <100.00%> (ø)`
packages/wrangler/src/pages/build.ts	`68.08% <100.00%> (+1.05%)`	⬆️
...ckages/wrangler/src/pages/functions/buildWorker.ts	`68.53% <100.00%> (+3.98%)`	⬆️

... and 17 files with indirect coverage changes

dario-piotrowicz

Looks good to me 😄 (based on my almost nonexistent knowledge of the codebase 😅)

packages/wrangler/src/__tests__/pages/functions-build.test.ts

packages/wrangler/src/api/pages/publish.tsx

packages/wrangler/src/dev/start-server.ts

packages/wrangler/src/api/pages/publish.tsx

.changeset/cuddly-rules-rest.md

fixtures/pages-workerjs-directory/public/_worker.js/index.js

penalosa · 2023-04-24T13:21:41Z

packages/wrangler/src/__tests__/pages/functions-build.test.ts

+  async fetch(request, env) {
+		return new Response("Hello from _worker.js/index.js" + cat);
+	},
+};`


packages/wrangler/src/api/dev.ts

packages/wrangler/src/api/pages/publish.tsx

packages/wrangler/src/dev/use-esbuild.ts

packages/wrangler/src/pages/build.ts

penalosa · 2023-04-24T13:42:25Z

packages/wrangler/src/pages/build.ts

+			if (lstatSync(workerScriptPath).isDirectory()) {
+				const entrypoint = resolve(join(workerScriptPath, "index.js"));
+
+				const traverseModuleGraphResult = await traverseModuleGraph(
+					{
+						file: entrypoint,
+						directory: resolve(workerScriptPath),
+						format: "modules",
+						moduleRoot: resolve(workerScriptPath),
+					},
+					[
+						{
+							type: "ESModule",
+							globs: ["**/*.js"],
+						},
+					]
+				);
+
+				const bundleResult = await buildRawWorker({
+					workerScriptPath: entrypoint,
+					bundle: false,
+					outdir,
+					directory: buildOutputDirectory,
+					local: false,
+					sourcemap,
+					watch,
+					betaD1Shims: d1Databases,
+				});
+
+				bundle = {
+					modules: (traverseModuleGraphResult?.modules ??
+						bundleResult?.modules) as BundleResult["modules"],
+					dependencies: (bundleResult?.dependencies ??
+						traverseModuleGraphResult?.dependencies) as BundleResult["dependencies"],
+					resolvedEntryPointPath: (bundleResult?.resolvedEntryPointPath ??
+						traverseModuleGraphResult?.resolvedEntryPointPath) as BundleResult["resolvedEntryPointPath"],
+					bundleType: (bundleResult?.bundleType ??
+						traverseModuleGraphResult?.bundleType) as BundleResult["bundleType"],
+					stop: bundleResult?.stop,
+					sourceMapPath: bundleResult?.sourceMapPath,
+				};
+			} else {
+				/**
+				 * `buildRawWorker` builds `_worker.js`, but doesn't give us the bundle
+				 * we want to return, which includes the external dependencies (like wasm,
+				 * binary, text). Let's output that build result to memory and only write
+				 * to disk once we have the final bundle
+				 */
+				bundle = await buildRawWorker({
+					workerScriptPath,
+					outdir,
+					directory: buildOutputDirectory,
+					local: false,
+					sourcemap,
+					watch,
+					betaD1Shims: d1Databases,
+				});
+			}


It feels like a lot of this is the same as pages/dev. Could it be pulled out into a common helper function?

packages/wrangler/src/pages/dev.ts

packages/wrangler/src/__tests__/pages/functions-build.test.ts

jahands · 2023-05-02T14:52:34Z

packages/wrangler/src/pages/functions/buildWorker.ts

+	});
+
+	return {
+		modules: traverseModuleGraphResult?.modules ?? bundleResult?.modules ?? [],


Under what case would it use bundleResult?.modules? Mind adding a comment explaining what is happening here? I'm not as familiar with esbuild/etc and the way we're doing the traversal and then bundling (but not bundling) is confusing

Yeah, good shout. This was me copy-pasting from elsewhere, but you're right. It'll never fallback to bundleResults.

JacobMGEvans · 2023-05-03T15:00:57Z

.changeset/cuddly-rules-rest.md

+"wrangler": minor
+---
+
+feat: Add support for the undocumented `_worker.js/` directory in Pages


Should it be documented?

GregBrimble force-pushed the no-bundle-pages branch 2 times, most recently from eff07ee to 84a6fa4 Compare April 5, 2023 10:25

GregBrimble force-pushed the no-bundle-pages branch 6 times, most recently from 7830ccd to 1966c9d Compare April 20, 2023 14:38

GregBrimble force-pushed the no-bundle-pages branch 4 times, most recently from 0551501 to 1b16312 Compare April 20, 2023 23:16

GregBrimble marked this pull request as ready for review April 21, 2023 00:03

GregBrimble requested review from a team as code owners April 21, 2023 00:03

GregBrimble force-pushed the no-bundle-pages branch from 1b16312 to 6733524 Compare April 21, 2023 00:04

GregBrimble changed the title ~~WIP: proper no-bundle in Pages~~ _worker.js/ directory support in Pages Apr 21, 2023

GregBrimble force-pushed the no-bundle-pages branch 2 times, most recently from 4f632cb to 8a93d66 Compare April 21, 2023 00:34

dario-piotrowicz mentioned this pull request Apr 21, 2023

implement lazy loading via Pages' _worker.js directory cloudflare/next-on-pages#191

Merged

GregBrimble mentioned this pull request Apr 21, 2023

🚀 Feature Request: Add support for shims when no-bundling is applied #3074

Closed

GregBrimble force-pushed the no-bundle-pages branch from 8a93d66 to dd54629 Compare April 21, 2023 13:56

_worker.js/ directory support in Pages

da29860

GregBrimble force-pushed the no-bundle-pages branch from dd54629 to da29860 Compare April 21, 2023 17:03

dario-piotrowicz approved these changes Apr 24, 2023

View reviewed changes

packages/wrangler/src/__tests__/pages/functions-build.test.ts Outdated Show resolved Hide resolved

packages/wrangler/src/api/pages/publish.tsx Outdated Show resolved Hide resolved

packages/wrangler/src/dev/start-server.ts Outdated Show resolved Hide resolved

petebacondarwin reviewed Apr 24, 2023

View reviewed changes

packages/wrangler/src/api/pages/publish.tsx Outdated Show resolved Hide resolved

penalosa requested changes Apr 24, 2023

View reviewed changes

bruce-shi mentioned this pull request Apr 27, 2023

Failed to publish your Function. Got error: Error: Script startup exceeded CPU time limit. cloudflare/next-on-pages#54

Closed

GregBrimble force-pushed the no-bundle-pages branch 4 times, most recently from 74b10f2 to 780817d Compare May 1, 2023 22:02

GregBrimble requested a review from penalosa May 1, 2023 22:16

GregBrimble force-pushed the no-bundle-pages branch 5 times, most recently from ad6fd0f to 7616ffc Compare May 1, 2023 23:39

GregBrimble commented May 2, 2023

View reviewed changes

packages/wrangler/src/__tests__/pages/functions-build.test.ts Outdated Show resolved Hide resolved

GregBrimble force-pushed the no-bundle-pages branch 2 times, most recently from b66bda9 to 045c3df Compare May 2, 2023 14:28

jahands reviewed May 2, 2023

View reviewed changes

GregBrimble force-pushed the no-bundle-pages branch from 045c3df to a79049b Compare May 2, 2023 14:59

Refactor some of the shared no bundle logic and address misc PR comments

6853fb7

GregBrimble force-pushed the no-bundle-pages branch from a79049b to 6853fb7 Compare May 2, 2023 15:03

jahands approved these changes May 2, 2023

View reviewed changes

penalosa approved these changes May 3, 2023

View reviewed changes

JacobMGEvans reviewed May 3, 2023

View reviewed changes

GregBrimble merged commit e351afc into main May 5, 2023
11 checks passed

GregBrimble deleted the no-bundle-pages branch May 5, 2023 01:01

GregBrimble mentioned this pull request May 5, 2023

🐛 BUG: Dynamic variable chunk imports aren't found when using _worker.js/ directory #3148

Closed

github-actions bot mentioned this pull request May 5, 2023

Version Packages #3149

Merged

penalosa mentioned this pull request May 5, 2023

Fix lock file #3156

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`_worker.js/` directory support in Pages #2966

`_worker.js/` directory support in Pages #2966

GregBrimble commented Mar 30, 2023 •

edited

changeset-bot bot commented Mar 31, 2023 •

edited

github-actions bot commented Mar 31, 2023 •

edited

codecov bot commented Apr 5, 2023 •

edited

dario-piotrowicz left a comment

penalosa Apr 24, 2023

penalosa Apr 24, 2023

jahands May 2, 2023 •

edited

GregBrimble May 2, 2023

JacobMGEvans May 3, 2023

_worker.js/ directory support in Pages #2966

_worker.js/ directory support in Pages #2966

Conversation

GregBrimble commented Mar 30, 2023 • edited

changeset-bot bot commented Mar 31, 2023 • edited

🦋 Changeset detected

github-actions bot commented Mar 31, 2023 • edited

codecov bot commented Apr 5, 2023 • edited

Codecov Report

dario-piotrowicz left a comment

Choose a reason for hiding this comment

penalosa Apr 24, 2023

Choose a reason for hiding this comment

penalosa Apr 24, 2023

Choose a reason for hiding this comment

jahands May 2, 2023 • edited

Choose a reason for hiding this comment

GregBrimble May 2, 2023

Choose a reason for hiding this comment

JacobMGEvans May 3, 2023

Choose a reason for hiding this comment

`_worker.js/` directory support in Pages #2966

`_worker.js/` directory support in Pages #2966

GregBrimble commented Mar 30, 2023 •

edited

changeset-bot bot commented Mar 31, 2023 •

edited

github-actions bot commented Mar 31, 2023 •

edited

codecov bot commented Apr 5, 2023 •

edited

jahands May 2, 2023 •

edited