Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reclaim disk space by purging old builds #1255

Open
elruwen opened this issue Oct 31, 2023 · 0 comments
Open

Reclaim disk space by purging old builds #1255

elruwen opened this issue Oct 31, 2023 · 0 comments

Comments

@elruwen
Copy link

elruwen commented Oct 31, 2023

Is your feature request related to a problem? Please describe.
I got a set of agents which are running for a while. After some point, the disk space health check nukes them:

Disk space free: 4.7G
Not enough disk space free, cutoff is 5242880 🚨
Cleaning up docker resources older than 4h
Total reclaimed space: 0B
Checking disk space again
Disk space free: 4.7G
Not enough disk space free, cutoff is 5242880 🚨
Disk health checks failed

Those agents are being used on the default queue and they basically checkout EVERY single repo we got. I got 30 agents running on one EC2 instance. At some point all the checkouts take up all the disk space.

Describe the solution you'd like
The disk health check is already doing some purges like docker cache. Allow it to purge everything under
/var/lib/buildkite-agent/builds/<agent name> as a last resort. In order to not break backwards compatibility, you might want to provide it opt-in.

Describe alternatives you've considered
Building it myself via environment hooks.

Additional context
Add any other context or screenshots about the feature request here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant