Update flags for Pipelines create and update #8753

cmackenzie1 · 2025-04-01T20:10:14Z

Transforms are not publicly available yet so we are hiding this from the help output. Private beta users will still be able to provide this flag.

This commit also introduces a new flag --shard-count for power users to hand tune their pipeline.

Fixes https://jira.cfdata.org/browse/PIPE-241, https://jira.cfdata.org/browse/PIPE-242

Tests
- TODO (before merge)
- Tests included
- Tests not necessary because:
Wrangler / Vite E2E Tests CI Job required? (Use "e2e" label or ask maintainer to run separately)
- I don't know
- Required
- Not required because: not needed
Public documentation
- TODO (before merge)
- Cloudflare docs PR(s): Initial pipelines docs cloudflare-docs#18595
- Documentation not necessary because:
Wrangler V3 Backport
- TODO (before merge)
- Wrangler PR:
- Not necessary because: Not public yet.

changeset-bot · 2025-04-01T20:10:19Z

🦋 Changeset detected

Latest commit: 08b6f8d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
wrangler	Patch
@cloudflare/vitest-pool-workers	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Transforms are not publicly available yet so we are hiding this from the help output. Private beta users will still be able to provide this flag. This commit also introduces a new flag `--shard-count` for power users to hand tune their pipeline. Fixes https://jira.cfdata.org/browse/PIPE-241, https://jira.cfdata.org/browse/PIPE-242

github-actions · 2025-04-01T20:59:28Z

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-wrangler-8753

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/8753/npm-package-wrangler-8753

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-wrangler-8753 dev path/to/script.js

Additional artifacts:

cloudflare-workers-bindings-extension:

wget https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-workers-bindings-extension-8753 -O ./cloudflare-workers-bindings-extension.0.0.0-v705539487.vsix && code --install-extension ./cloudflare-workers-bindings-extension.0.0.0-v705539487.vsix

create-cloudflare:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-create-cloudflare-8753 --no-auto-update

@cloudflare/kv-asset-handler:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-kv-asset-handler-8753

miniflare:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-miniflare-8753

@cloudflare/pages-shared:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-pages-shared-8753

@cloudflare/unenv-preset:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-unenv-preset-8753

@cloudflare/vite-plugin:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-vite-plugin-8753

@cloudflare/vitest-pool-workers:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-vitest-pool-workers-8753

@cloudflare/workers-editor-shared:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-workers-editor-shared-8753

@cloudflare/workers-shared:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-workers-shared-8753

@cloudflare/workflows-shared:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/14205075282/npm-package-cloudflare-workflows-shared-8753

Note that these links will no longer work once the GitHub Actions artifact expires.

wrangler@4.6.0 includes the following runtime dependencies:

Package	Constraint	Resolved
`miniflare`	workspace:*	4.20250321.1
`workerd`	1.20250321.0	1.20250321.0
`workerd --version`	1.20250321.0	2025-03-21

Please ensure constraints are pinned, and miniflare/workerd minor versions match.

petebacondarwin

One question about default values for shard count. Othewise LGTM.

petebacondarwin · 2025-04-02T15:06:24Z

packages/wrangler/src/pipelines/cli/create.ts

@@ -281,6 +291,10 @@ export async function createPipelineHandler(
 		pipelineConfig.destination.path.filename = args.fileTemplate;
 	}

+	if (args.shardCount) {


This will treat undefined and 0 the same. I assume this is what you want?

Yeah, this is fine. 0 is not an allowed value for number of shards.

petebacondarwin · 2025-04-02T15:06:50Z

packages/wrangler/src/pipelines/cli/create.ts

+			.option("shard-count", {
+				type: "number",
+				describe:
+					"Number of pipeline shards. More shards handle higher request volume; fewer shards produce larger output files",


What is the default number of shards? One?

I think 2, but that may change. We are trying to avoid baking defaults into wrangler and let the API set them instead.

petebacondarwin · 2025-04-02T15:07:48Z

packages/wrangler/src/pipelines/client.ts

@@ -11,16 +11,20 @@ import type { R2BucketInfo } from "../r2/helpers";

 // ensure this is in sync with:
 //   https://bitbucket.cfdata.org/projects/PIPE/repos/superpipe/browse/src/coordinator/types.ts#6


Is this still bitbucket or gitlab?

GitLab now. What is the policy on linking to internal codebases in a public project?

It's generally fine I believe since it is all locked behind Access.
But no need to update if you don't want to.

cmackenzie1 requested a review from a team as a code owner April 1, 2025 20:10

cmackenzie1 requested review from oliy and hhoughgg April 1, 2025 20:10

cmackenzie1 force-pushed the cole/pipelines-dev-week branch from 67f2fb5 to 08b6f8d Compare April 1, 2025 20:11

workers-devprod added this to workers-sdk Apr 1, 2025

github-project-automation bot moved this to Untriaged in workers-sdk Apr 1, 2025

oliy approved these changes Apr 1, 2025

View reviewed changes

petebacondarwin approved these changes Apr 2, 2025

View reviewed changes

github-project-automation bot moved this from Untriaged to Approved in workers-sdk Apr 2, 2025

petebacondarwin self-assigned this Apr 2, 2025

petebacondarwin merged commit 245cfbd into main Apr 2, 2025
19 checks passed

petebacondarwin deleted the cole/pipelines-dev-week branch April 2, 2025 19:41

github-project-automation bot moved this from Approved to Done in workers-sdk Apr 2, 2025

workers-devprod mentioned this pull request Apr 2, 2025

Version Packages #8762

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update flags for Pipelines create and update #8753

Update flags for Pipelines create and update #8753

cmackenzie1 commented Apr 1, 2025

changeset-bot bot commented Apr 1, 2025 •

edited

Loading

github-actions bot commented Apr 1, 2025

petebacondarwin left a comment

petebacondarwin Apr 2, 2025

cmackenzie1 Apr 2, 2025

petebacondarwin Apr 2, 2025

cmackenzie1 Apr 2, 2025

petebacondarwin Apr 2, 2025

cmackenzie1 Apr 2, 2025

petebacondarwin Apr 2, 2025

		@@ -11,16 +11,20 @@ import type { R2BucketInfo } from "../r2/helpers";

		// ensure this is in sync with:
		// https://bitbucket.cfdata.org/projects/PIPE/repos/superpipe/browse/src/coordinator/types.ts#6

Update flags for Pipelines create and update #8753

Update flags for Pipelines create and update #8753

Conversation

cmackenzie1 commented Apr 1, 2025

changeset-bot bot commented Apr 1, 2025 • edited Loading

🦋 Changeset detected

github-actions bot commented Apr 1, 2025

petebacondarwin left a comment

Choose a reason for hiding this comment

petebacondarwin Apr 2, 2025

Choose a reason for hiding this comment

cmackenzie1 Apr 2, 2025

Choose a reason for hiding this comment

petebacondarwin Apr 2, 2025

Choose a reason for hiding this comment

cmackenzie1 Apr 2, 2025

Choose a reason for hiding this comment

petebacondarwin Apr 2, 2025

Choose a reason for hiding this comment

cmackenzie1 Apr 2, 2025

Choose a reason for hiding this comment

petebacondarwin Apr 2, 2025

Choose a reason for hiding this comment

changeset-bot bot commented Apr 1, 2025 •

edited

Loading