Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: gc policy for windows to use 20% of disk space #4865

Merged
merged 1 commit into from Apr 23, 2024

Conversation

profnandaa
Copy link
Collaborator

@profnandaa profnandaa commented Apr 22, 2024

Initially we had the GC Policy for Windows use only 2 GB (2e9 bytes) of disk space and this was limiting for some build scenarios that need more than that, especially ServerCore images.

This commit makes the policy to use percentages as it is on Linux. Also going for 20%, double for that on Linux, since Windows images are generally larger. For instance, after the change, on a dev machine, the configured space was 74694367232 / (1 << 30) = 69.5 GiB.

fixes #4858 docker/buildx#2411


Before:

> main.getGCPolicy() C:/dev/container-core/buildkit/cmd/buildkitd/main.go:875 (hits goroutine(1):1 total:1) (PC: 0x1e1347e)
   870:                         All:          rule.All,
   871:                         KeepBytes:    rule.KeepBytes.AsBytes(root),
   872:                         KeepDuration: rule.KeepDuration.Duration,
   873:                 })
   874:         }
=> 875:         return out
   876: }
   877:
   878: func getBuildkitVersion() client.BuildkitVersion {
   879:         return client.BuildkitVersion{
   880:                 Package:  version.Package,
(dlv) p out
[]github.com/moby/buildkit/client.PruneInfo len: 4, cap: 4, [
        {
                Filter: []string len: 1, cap: 1, [
                        "type==source.local,type==exec.cachemount,type==source.git.checko...+2 more",
                ],
                All: false,
                KeepDuration: 172800000000000,
                KeepBytes: 512000000,},
        {
                Filter: []string len: 0, cap: 0, nil,
                All: false,
                KeepDuration: 5184000000000000,
                KeepBytes: 2000000000,},
        {
                Filter: []string len: 0, cap: 0, nil,
                All: false,
                KeepDuration: 0,
                KeepBytes: 2000000000,},
        {
                Filter: []string len: 0, cap: 0, nil,
                All: true,
                KeepDuration: 0,
                KeepBytes: 2000000000,},
]

After:

> main.getGCPolicy() C:/dev/container-core/buildkit/cmd/buildkitd/main.go:875 (hits goroutine(1):1 total:1) (PC: 0x2c63a5e)
   870:                         All:          rule.All,
   871:                         KeepBytes:    rule.KeepBytes.AsBytes(root),
   872:                         KeepDuration: rule.KeepDuration.Duration,
   873:                 })
   874:         }
=> 875:         return out
   876: }
   877:
   878: func getBuildkitVersion() client.BuildkitVersion {
   879:         return client.BuildkitVersion{
   880:                 Package:  version.Package,
(dlv) p out
[]github.com/moby/buildkit/client.PruneInfo len: 4, cap: 4, [
        {
                Filter: []string len: 1, cap: 1, [
                        "type==source.local,type==exec.cachemount,type==source.git.checko...+2 more",
                ],
                All: false,
                KeepDuration: 172800000000000,
                KeepBytes: 512000000,},
        {
                Filter: []string len: 0, cap: 0, nil,
                All: false,
                KeepDuration: 5184000000000000,
                KeepBytes: 140000000000,},
        {
                Filter: []string len: 0, cap: 0, nil,
                All: false,
                KeepDuration: 0,
                KeepBytes: 140000000000,},
        {
                Filter: []string len: 0, cap: 0, nil,
                All: true,
                KeepDuration: 0,
                KeepBytes: 140000000000,},
]

cmd/buildkitd/config/gcpolicy_windows.go Outdated Show resolved Hide resolved
cmd/buildkitd/config/gcpolicy_windows.go Outdated Show resolved Hide resolved
@profnandaa profnandaa force-pushed the fix-buildx-2411-wcow-gcpolicy branch from 61399b6 to 2b9917d Compare April 22, 2024 18:09
@profnandaa profnandaa changed the title fix: gc policy for windows to use 10% of disk space fix: gc policy for windows to use 20% of disk space Apr 22, 2024
cmd/buildkitd/config/gcpolicy_unix.go Outdated Show resolved Hide resolved
cmd/buildkitd/config/gcpolicy.go Outdated Show resolved Hide resolved
@profnandaa profnandaa force-pushed the fix-buildx-2411-wcow-gcpolicy branch 2 times, most recently from d58718d to 56d3745 Compare April 23, 2024 04:42
@tonistiigi tonistiigi added this to the v0.13.2 milestone Apr 23, 2024
cmd/buildkitd/config/gcpolicy.go Outdated Show resolved Hide resolved
@profnandaa profnandaa force-pushed the fix-buildx-2411-wcow-gcpolicy branch from 56d3745 to 761cb10 Compare April 23, 2024 20:45
Initially we had the GC Policy for Windows use only
2 GB (2e9 bytes) of disk space and this was limiting
for some build scenarios that need more than that,
especially ServerCore images.

This commit makes the policy to use percentages
as it is on Linux. Also going for 20%, double that
of Linux, since Windows images tend to be larger.

fixes moby#4858 docker/buildx#2411

Also, refactors the diskSize logic to simplify
it by bringing the `d.AsByte` function back from
platform specific files to `gcpolicy.go`.

Signed-off-by: Anthony Nandaa <profnandaa@gmail.com>
@profnandaa profnandaa force-pushed the fix-buildx-2411-wcow-gcpolicy branch from 761cb10 to a5f9e42 Compare April 23, 2024 20:55
@AkihiroSuda AkihiroSuda merged commit d1783df into moby:master Apr 23, 2024
73 checks passed
@profnandaa profnandaa deleted the fix-buildx-2411-wcow-gcpolicy branch April 23, 2024 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[WCOW] GC storage limit is initialized with constant value
5 participants