Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: ensure unused KV and Cache blobs cleaned up #4466

Merged
merged 1 commit into from Dec 4, 2023
Merged

Conversation

mrbbot
Copy link
Contributor

@mrbbot mrbbot commented Nov 17, 2023

What this PR solves / how to test:

cloudflare/miniflare#656 introduced a bug in the SQL statement used for getting the old blob ID of entries to delete when overriding keys. This meant if a key was overridden, the old blob pointed to by that key was not deleted. This lead to an accumulation of garbage when kvPersist and cachePersist were enabled. This shouldn't have caused any problems, but could lead to persistence directories growing unnecessarily as they filled up with garbage blobs. This change ensures garbage blobs are deleted, and adds some regression tests to prevent this happening again.

Note existing garbage will not be cleaned up. If you'd like to do this, download this Node script (https://gist.github.com/mrbbot/68787e19dcde511bd99aa94997b39076). If you're using the default Wrangler persistence directory, run node gc.mjs kv .wrangler/state/v3/kv <namespace_id_1> <namespace_id_2> ... and node gc.mjs cache .wrangler/state/v3/cache default named:<cache_name_1> named:<cache_name_2> ... with each of your KV namespace IDs (not binding names) and named caches.

Author has addressed the following:

  • Tests
    • Included
    • Not necessary because:
  • Changeset (Changeset guidelines)
    • Included
    • Not necessary because:
  • Associated docs
    • Issue(s)/PR(s):
    • Not necessary because: fixing a bug in Miniflare's underlying storage implementation

Note for PR author:

We want to celebrate and highlight awesome PR review! If you think this PR received a particularly high-caliber review, please assign it the label highlight pr review so future reviewers can take inspiration and learn from it.

@mrbbot mrbbot requested a review from a team as a code owner November 17, 2023 12:31
Copy link

changeset-bot bot commented Nov 17, 2023

🦋 Changeset detected

Latest commit: 61c4b0b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
miniflare Patch
@cloudflare/pages-shared Patch
wrangler Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link
Contributor

github-actions bot commented Nov 17, 2023

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7085442368/npm-package-wrangler-4466

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/7085442368/npm-package-wrangler-4466

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7085442368/npm-package-wrangler-4466 dev path/to/script.js
Additional artifacts:
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7085442368/npm-package-miniflare-4466
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7085442368/npm-package-cloudflare-pages-shared-4466
npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7085442368/npm-package-create-cloudflare-4466

Note that these links will no longer work once the GitHub Actions artifact expires.


wrangler@3.18.0 includes the following runtime dependencies:

Package Constraint Resolved
miniflare workspace:* 3.20231030.2
workerd 1.20231030.0 1.20231030.0
workerd --version 1.20231030.0 2023-10-30

|

Please ensure constraints are pinned, and miniflare/workerd minor versions match.

Copy link

codecov bot commented Nov 17, 2023

Codecov Report

Merging #4466 (61c4b0b) into main (46a35e0) will increase coverage by 0.01%.
Report is 52 commits behind head on main.
The diff coverage is n/a.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4466      +/-   ##
==========================================
+ Coverage   75.46%   75.48%   +0.01%     
==========================================
  Files         225      240      +15     
  Lines       12478    12854     +376     
  Branches     3239     3312      +73     
==========================================
+ Hits         9417     9703     +286     
- Misses       3061     3151      +90     

see 36 files with indirect coverage changes

Copy link
Contributor

@petebacondarwin petebacondarwin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

cloudflare/miniflare#656 introduced a bug in the SQL statement used
for getting the old blob ID of entries to delete when overriding keys.
This meant if a key was overridden, the old blob pointed to by that
key was not deleted. This lead to an accumulation of garbage when
`kvPersist` and `cachePersist` were enabled.
@mrbbot mrbbot merged commit 71fb0b8 into main Dec 4, 2023
17 checks passed
@mrbbot mrbbot deleted the bcoll/fix-kv-cache-gc branch December 4, 2023 15:16
@workers-devprod workers-devprod mentioned this pull request Dec 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

2 participants