Fetch only necessary commits for build #95

Felixoid · 2024-05-07T08:25:12Z

In docker-library/official-images#15846 @LaurentGoderre asked me to move the Dockerfiles into an orphan branch because bashbrew spends a huge time to fetch the git references.

Here's the simplest change improves the fetch by an order of magnitude:

> time (rm -rf ~/.cache/bashbrew && ~/Space/go/bin/bashbrew --debug build library/clickhouse)
Using bashbrew/cache:e3168289a003ed8aa644f472b1d6a4521683d5c1a6a12765f9a0329d375dd4d1 (clickhouse:latest)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "clickhouse:latest" "--tag" "clickhouse:focal" "--tag" "clickhouse:24" "--tag" "clickhouse:24-focal" "--tag" "clickhouse:24.4" "--tag" "clickhouse:24.4-focal" "--tag" "clickhouse:24.4.1" "--tag" "clickhouse:24.4.1-focal" "--tag" "clickhouse:24.4.1.2088" "--tag" "clickhouse:24.4.1.2088-focal" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:e3168289a003ed8aa644f472b1d6a4521683d5c1a6a12765f9a0329d375dd4d1
 ---> e6a434d83fd8
Successfully built e6a434d83fd8
Successfully tagged clickhouse:latest
Successfully tagged clickhouse:focal
Successfully tagged clickhouse:24
Successfully tagged clickhouse:24-focal
Successfully tagged clickhouse:24.4
Successfully tagged clickhouse:24.4-focal
Successfully tagged clickhouse:24.4.1
Successfully tagged clickhouse:24.4.1-focal
Successfully tagged clickhouse:24.4.1.2088
Successfully tagged clickhouse:24.4.1.2088-focal
( rm -rf ~/.cache/bashbrew && ~/Space/go/bin/bashbrew --debug build ; ) 
 4,64s user 0,78s system 34% cpu 15,659 total

> time (rm -rf ~/.cache/bashbrew && bashbrew --debug build library/clickhouse)
Using bashbrew/cache:e3168289a003ed8aa644f472b1d6a4521683d5c1a6a12765f9a0329d375dd4d1 (clickhouse:latest)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "clickhouse:latest" "--tag" "clickhouse:focal" "--tag" "clickhouse:24" "--tag" "clickhouse:24-focal" "--tag" "clickhouse:24.4" "--tag" "clickhouse:24.4-focal" "--tag" "clickhouse:24.4.1" "--tag" "clickhouse:24.4.1-focal" "--tag" "clickhouse:24.4.1.2088" "--tag" "clickhouse:24.4.1.2088-focal" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:e3168289a003ed8aa644f472b1d6a4521683d5c1a6a12765f9a0329d375dd4d1
 ---> e6a434d83fd8
Successfully built e6a434d83fd8
Successfully tagged clickhouse:latest
Successfully tagged clickhouse:focal
Successfully tagged clickhouse:24
Successfully tagged clickhouse:24-focal
Successfully tagged clickhouse:24.4
Successfully tagged clickhouse:24.4-focal
Successfully tagged clickhouse:24.4.1
Successfully tagged clickhouse:24.4.1-focal
Successfully tagged clickhouse:24.4.1.2088
Successfully tagged clickhouse:24.4.1.2088-focal
( rm -rf ~/.cache/bashbrew && bashbrew --debug build library/clickhouse; ) 
 255,70s user 15,34s system 113% cpu 3:59,07 total

$ bashbrew --version
bashbrew version v0.1.13
$ ~/Space/go/bin/bashbrew --version
bashbrew version v0.1.13

Felixoid · 2024-05-07T08:36:31Z

Here's another test to see if it works with different refs

Building alpine with different refs in the manifest file

> time (rm -rf ~/.cache/bashbrew && ~/Space/go/bin/bashbrew --debug build library/alpine)
Using bashbrew/cache:a549aabbc73b701e3dff468635173f5a31fb8df23d45674d810037f188e4a74b (alpine:20230901)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "alpine:20230901" "--tag" "alpine:edge" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:a549aabbc73b701e3dff468635173f5a31fb8df23d45674d810037f188e4a74b
 ---> 8b96eaa5d551
Successfully built 8b96eaa5d551
Successfully tagged alpine:20230901
Successfully tagged alpine:edge
Using bashbrew/cache:54d04be340227935d149a64d37b5ada2234fac0cba400486f71e1af87fdf0bbd (alpine:3.18.5)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "alpine:3.18.5" "--tag" "alpine:3.18" "--tag" "alpine:3" "--tag" "alpine:latest" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:54d04be340227935d149a64d37b5ada2234fac0cba400486f71e1af87fdf0bbd
 ---> 0a39b18eeb74
Successfully built 0a39b18eeb74
Successfully tagged alpine:3.18.5
Successfully tagged alpine:3.18
Successfully tagged alpine:3
Successfully tagged alpine:latest
Using bashbrew/cache:bd8ebcaf28443dbdc6a23325432bc0fc7cda6cb0f11bc4c9e06ad35ab16c1d9e (alpine:3.17.6)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "alpine:3.17.6" "--tag" "alpine:3.17" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:bd8ebcaf28443dbdc6a23325432bc0fc7cda6cb0f11bc4c9e06ad35ab16c1d9e
 ---> 710cfc90b7ca
Successfully built 710cfc90b7ca
Successfully tagged alpine:3.17.6
Successfully tagged alpine:3.17
Using bashbrew/cache:799ae2e8a34101169cc25094ff566e0841a31c9ed7149348f6c8710324d8a72a (alpine:3.16.8)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "alpine:3.16.8" "--tag" "alpine:3.16" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:799ae2e8a34101169cc25094ff566e0841a31c9ed7149348f6c8710324d8a72a
 ---> 0f11e7a385b4
Successfully built 0f11e7a385b4
Successfully tagged alpine:3.16.8
Successfully tagged alpine:3.16
Using bashbrew/cache:f96d9751c747839ab074ce20fa5e24f69aba0e03402db471c4e74b34bbea41aa (alpine:3.15.11)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "alpine:3.15.11" "--tag" "alpine:3.15" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:f96d9751c747839ab074ce20fa5e24f69aba0e03402db471c4e74b34bbea41aa
 ---> 7620a8176e11
Successfully built 7620a8176e11
Successfully tagged alpine:3.15.11
Successfully tagged alpine:3.15
( rm -rf ~/.cache/bashbrew && ~/Space/go/bin/bashbrew --debug build ; ) 
 1,88s user 0,74s system 24% cpu 10,485 total

> time (rm -rf ~/.cache/bashbrew && bashbrew --debug build library/alpine)
Using bashbrew/cache:a549aabbc73b701e3dff468635173f5a31fb8df23d45674d810037f188e4a74b (alpine:20230901)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "alpine:20230901" "--tag" "alpine:edge" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:a549aabbc73b701e3dff468635173f5a31fb8df23d45674d810037f188e4a74b
 ---> 8b96eaa5d551
Successfully built 8b96eaa5d551
Successfully tagged alpine:20230901
Successfully tagged alpine:edge
Using bashbrew/cache:54d04be340227935d149a64d37b5ada2234fac0cba400486f71e1af87fdf0bbd (alpine:3.18.5)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "alpine:3.18.5" "--tag" "alpine:3.18" "--tag" "alpine:3" "--tag" "alpine:latest" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:54d04be340227935d149a64d37b5ada2234fac0cba400486f71e1af87fdf0bbd
 ---> 0a39b18eeb74
Successfully built 0a39b18eeb74
Successfully tagged alpine:3.18.5
Successfully tagged alpine:3.18
Successfully tagged alpine:3
Successfully tagged alpine:latest
Using bashbrew/cache:bd8ebcaf28443dbdc6a23325432bc0fc7cda6cb0f11bc4c9e06ad35ab16c1d9e (alpine:3.17.6)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "alpine:3.17.6" "--tag" "alpine:3.17" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:bd8ebcaf28443dbdc6a23325432bc0fc7cda6cb0f11bc4c9e06ad35ab16c1d9e
 ---> 710cfc90b7ca
Successfully built 710cfc90b7ca
Successfully tagged alpine:3.17.6
Successfully tagged alpine:3.17
Using bashbrew/cache:799ae2e8a34101169cc25094ff566e0841a31c9ed7149348f6c8710324d8a72a (alpine:3.16.8)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "alpine:3.16.8" "--tag" "alpine:3.16" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:799ae2e8a34101169cc25094ff566e0841a31c9ed7149348f6c8710324d8a72a
 ---> 0f11e7a385b4
Successfully built 0f11e7a385b4
Successfully tagged alpine:3.16.8
Successfully tagged alpine:3.16
Using bashbrew/cache:f96d9751c747839ab074ce20fa5e24f69aba0e03402db471c4e74b34bbea41aa (alpine:3.15.11)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "alpine:3.15.11" "--tag" "alpine:3.15" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:f96d9751c747839ab074ce20fa5e24f69aba0e03402db471c4e74b34bbea41aa
 ---> 7620a8176e11
Successfully built 7620a8176e11
Successfully tagged alpine:3.15.11
Successfully tagged alpine:3.15
( rm -rf ~/.cache/bashbrew && bashbrew --debug build library/alpine; ) 
 1,93s user 0,76s system 24% cpu 11,047 total

Felixoid · 2024-05-07T11:14:46Z

Here's an example of building an alpine image from different commits:

Building library/alpine from different commits to test cache works

$ rm ~/.cache/bashbrew/ -rf
$ git checkout  master >/dev/null
Already on 'master'
$ time ~/Space/go/bin/bashbrew build library/alpine
Building bashbrew/cache:a549aabbc73b701e3dff468635173f5a31fb8df23d45674d810037f188e4a74b (alpine:20230901)
Building bashbrew/cache:54d04be340227935d149a64d37b5ada2234fac0cba400486f71e1af87fdf0bbd (alpine:3.18.5)
Building bashbrew/cache:bd8ebcaf28443dbdc6a23325432bc0fc7cda6cb0f11bc4c9e06ad35ab16c1d9e (alpine:3.17.6)
Building bashbrew/cache:799ae2e8a34101169cc25094ff566e0841a31c9ed7149348f6c8710324d8a72a (alpine:3.16.8)
Building bashbrew/cache:f96d9751c747839ab074ce20fa5e24f69aba0e03402db471c4e74b34bbea41aa (alpine:3.15.11)

real    0m16,301s
user    0m3,681s
sys     0m1,260s
$ git checkout $(git log -1 --format=%H library/alpine)~
Note: switching to '7e13453ce3b47c537e4ffe79127d4900825e9d03~'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at ca1ff8a21 Merge pull request #15797 from docker-library-bot/external-pins-tianon
$ time ~/Space/go/bin/bashbrew build library/alpine
Using bashbrew/cache:a549aabbc73b701e3dff468635173f5a31fb8df23d45674d810037f188e4a74b (alpine:20230901)
Building bashbrew/cache:9eb60d921d482965f048fe6217d50bb1a54c90efde85f6944716525287864f64 (alpine:3.18.4)
Building bashbrew/cache:43f876d1c162b59663bbefd823db152fe59d3960b863880c73a8ffd716af57c7 (alpine:3.17.5)
Building bashbrew/cache:9c9ead487f115deb249c82247e1373f3bab1aa5f31efe01ae44499d35c4d4bbe (alpine:3.16.7)
Building bashbrew/cache:a57aaa93382aa13b957f42b41cc78fb61f0496df2f52ac64759c1c20463f570e (alpine:3.15.10)

real    0m12,929s
user    0m2,813s
sys     0m1,027s
$ git checkout $(git log -1 --format=%H library/alpine)~
Previous HEAD position was ca1ff8a21 Merge pull request #15797 from docker-library-bot/external-pins-tianon
HEAD is now at ff00e9909 Merge pull request #15439 from docker-library-bot/wordpress
$ time ~/Space/go/bin/bashbrew build library/alpine
Using bashbrew/cache:a549aabbc73b701e3dff468635173f5a31fb8df23d45674d810037f188e4a74b (alpine:20230901)
Building bashbrew/cache:85083aa00523429a3d2708776410b89bb2ef26d79ed43b8a617b69d0a6a5592e (alpine:3.18.3)
Using bashbrew/cache:43f876d1c162b59663bbefd823db152fe59d3960b863880c73a8ffd716af57c7 (alpine:3.17.5)
Using bashbrew/cache:9c9ead487f115deb249c82247e1373f3bab1aa5f31efe01ae44499d35c4d4bbe (alpine:3.16.7)
Using bashbrew/cache:a57aaa93382aa13b957f42b41cc78fb61f0496df2f52ac64759c1c20463f570e (alpine:3.15.10)

real    0m4,049s
user    0m0,980s
sys     0m0,430s
$ git checkout $(git log -1 --format=%H library/alpine)~
Previous HEAD position was ff00e9909 Merge pull request #15439 from docker-library-bot/wordpress
HEAD is now at 6507e98e7 Merge pull request #15294 from docker-library-bot/openjdk
$ time ~/Space/go/bin/bashbrew build library/alpine
Building bashbrew/cache:2860aed394846a7f5b47777edd6f10f0c8433a8db6969fecc1038cada367f14e (alpine:20230329)
Using bashbrew/cache:85083aa00523429a3d2708776410b89bb2ef26d79ed43b8a617b69d0a6a5592e (alpine:3.18.3)
Using bashbrew/cache:43f876d1c162b59663bbefd823db152fe59d3960b863880c73a8ffd716af57c7 (alpine:3.17.5)
Using bashbrew/cache:9c9ead487f115deb249c82247e1373f3bab1aa5f31efe01ae44499d35c4d4bbe (alpine:3.16.7)
Using bashbrew/cache:a57aaa93382aa13b957f42b41cc78fb61f0496df2f52ac64759c1c20463f570e (alpine:3.15.10)

real    0m4,503s
user    0m1,027s
sys     0m0,545s
$ git checkout $(git log -1 --format=%H library/alpine)~
Previous HEAD position was 6507e98e7 Merge pull request #15294 from docker-library-bot/openjdk
HEAD is now at 5433844f5 Merge pull request #15150 from docker-library-bot/ghost
$ time ~/Space/go/bin/bashbrew build library/alpine
Using bashbrew/cache:2860aed394846a7f5b47777edd6f10f0c8433a8db6969fecc1038cada367f14e (alpine:20230329)
Building bashbrew/cache:c888cad40c964c0f6de9d7493ebdd83bff0ea773ad37e297a7e427893149ffd6 (alpine:3.18.2)
Building bashbrew/cache:f439ee9f147e9e793107ac3db503360ea5115ed542347aa011cf81fee04359c3 (alpine:3.17.4)
Building bashbrew/cache:785b7c78d4aa1a8595ebda7e50bd5105e3930032cf20fab7a4e450b02d3038d3 (alpine:3.16.6)
Building bashbrew/cache:6e6a5e0713a61f93b9f306af55fb688f795a1d168c1d6b1f620c9d64c02bfad9 (alpine:3.15.9)

real    0m12,934s
user    0m2,802s
sys     0m1,096s
$ git checkout $(git log -1 --format=%H library/alpine)~
Previous HEAD position was 5433844f5 Merge pull request #15150 from docker-library-bot/ghost
HEAD is now at 19f4f9c2d Merge pull request #14848 from amsteinb-orcl/update-oraclelinux
$ time ~/Space/go/bin/bashbrew build library/alpine
Using bashbrew/cache:2860aed394846a7f5b47777edd6f10f0c8433a8db6969fecc1038cada367f14e (alpine:20230329)
Building bashbrew/cache:73cf7ede3ed18aca96e7120a03dd7bb6c512d792dd928a1f9c72fbcab4786870 (alpine:3.18.0)
Building bashbrew/cache:addd6e6541e0f2bb34ae37a3ea1374e1f6db88808f7419a4dd7e6eaff906dcae (alpine:3.17.3)
Building bashbrew/cache:f6cbf1d672caaec6d46e037f93d7a79079ded11fb0fb0e7ab3897ef3adb95af2 (alpine:3.16.5)
Building bashbrew/cache:05fd9c1efca3611374c2e78bc4044648a0c9aad75be5a3892b88fe60de18e7f9 (alpine:3.15.8)

real    0m14,306s
user    0m2,878s
sys     0m1,100s
$ git checkout master
Previous HEAD position was 19f4f9c2d Merge pull request #14848 from amsteinb-orcl/update-oraclelinux
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
$ time ~/Space/go/bin/bashbrew build library/alpine
Using bashbrew/cache:a549aabbc73b701e3dff468635173f5a31fb8df23d45674d810037f188e4a74b (alpine:20230901)
Using bashbrew/cache:54d04be340227935d149a64d37b5ada2234fac0cba400486f71e1af87fdf0bbd (alpine:3.18.5)
Using bashbrew/cache:bd8ebcaf28443dbdc6a23325432bc0fc7cda6cb0f11bc4c9e06ad35ab16c1d9e (alpine:3.17.6)
Using bashbrew/cache:799ae2e8a34101169cc25094ff566e0841a31c9ed7149348f6c8710324d8a72a (alpine:3.16.8)
Using bashbrew/cache:f96d9751c747839ab074ce20fa5e24f69aba0e03402db471c4e74b34bbea41aa (alpine:3.15.11)

real    0m0,462s
user    0m0,181s
sys     0m0,182s
$ du -hscx ~/.cache/bashbrew/
373M    /home/felixoid/.cache/bashbrew/
373M    total

Felixoid · 2024-05-07T12:13:46Z

Compare the workflow running against master vs running against PR

The build time decreased from 11m to 1m

tianon · 2024-05-07T20:56:24Z

Unfortunately, this would need to be more complicated -- effectively, if we're pulling from a Git platform which does not support direct fetch-by-commit, we rely on GitFetch to point us to a tree which contains GitCommit, and only fetching the top commit in those cases would not necessarily successfully fetch the commit.

Also, in our newer build system, we pass these Git URLs directly to docker buildx build (not using bashbrew build at all) and BuildKit becomes responsible for the git clone (so it can record accurate provenance data directly in the provenance objects it generates), and it doesn't currently even do direct fetch-by-commit (it fetches all refs hoping one of them contains the requested commit) 😭

Felixoid · 2024-05-07T22:22:28Z

Oh my.. how would this build kit approach work with repositories as big as ClickHouse/ClickHouse or even bigger?

Is it just a blocker to use such big repos?

Aside from that, can you give an example of what should still work, but would be probably broken? Like, some library manifest to test?

update: it looks like library/almalinux is broken. I'll try to improve the changes.

Felixoid · 2024-05-08T12:28:41Z

if we're pulling from a Git platform which does not support direct fetch-by-commit, we rely on GitFetch to point us to a tree which contains GitCommit, and only fetching the top commit in those cases would not necessarily successfully fetch the commit.

Ok, I am pretty sure it's a bug in the go-git addressed somewhere between 5.5 and 5.12. Just updating this module fixes everything.

Update: go-git/go-git#305 it's fixed in 5.8.0 by go-git/go-git#778

Felixoid · 2024-05-08T14:12:41Z

Here are the test results for the binary built from the latest state. The go-git is updated:

$ time ( : > /tmp/bashbrew.log; rm -rf ~/.cache/bashbrew; for lib in $(git grep -l GitFetch library/); do  ~/Space/go/bin/bashbrew --debug build --dry-run  "$lib" || echo "Failed '$lib'" >> /tmp/bashbrew.log ; [ -f /tmp/test-stop ] && break || :; done ; ) &> /tmp/bashbrew-first-run.log
( : > /tmp/bashbrew.log; rm -rf ~/.cache/bashbrew; for lib in ; do;  --debug   71,36s user 36,75s system 23% cpu 7:36,95 total

$ time ( : > /tmp/bashbrew.log; for lib in $(git grep -l GitFetch library/); do  ~/Space/go/bin/bashbrew --debug build --dry-run  "$lib" || echo "Failed '$lib'" >> /tmp/bashbrew.log ; [ -f /tmp/test-stop ] && break || :; done ; ) &> /tmp/bashbrew-second-run.log 2>&1
( : > /tmp/bashbrew.log; for lib in $(git grep -l GitFetch library/); do;      11,96s user 13,90s system 103% cpu 24,953 total

$ du -hsx ~/.cache/bashbrew/git
3,7G    /home/felixoid/.cache/bashbrew/git

bashbrew-first-run.log
bashbrew-second-run.log

There's no issue like commit is unreached.

Here's what it looks on vanilla:

$ git checkout upstream/master # without clickhouse
$ time ( : > /tmp/bashbrew.log; rm -rf ~/.cache/bashbrew/git; for lib in $(git grep -l GitFetch library/); do  bashbrew --debug build --dry-run  "$lib" || echo "Failed '$lib'" >> /tmp/bashbrew.log ; [ -f /tmp/test-stop ] && break || :; done ; ) &> /tmp/bashbrew-vanila-run.log
( : > /tmp/bashbrew.log; rm -rf ~/.cache/bashbrew/git; for lib in ; do;        107,38s user 42,18s system 28% cpu 8:42,09 total
$ du -hsx ~/.cache/bashbrew/git
4,2G    /home/felixoid/.cache/bashbrew/git
$ git checkout master # adding clickhouse
$ bashbrew --debug build --dry-run library/clickhouse
Building bashbrew/cache:f51c2d7033cc3b6a7c6a5aa277def27d223939b7823a609b5e590055b6c7135e (clickhouse:latest)
$ du -hsx ~/.cache/bashbrew/git
5,2G    /home/felixoid/.cache/bashbrew/git

And this is how it looks for library/almalinux without updated go-git built from a31e673:

$ mv ~/.cache/bashbrew/git ~/.cache/bashbrew/git.bak
$ ~/Space/go/bin/bashbrew --debug build library/almalinux
failed sorting entries list for "library/almalinux"
failed fetching Git repo for arch "amd64" from entry "Maintainers: The AlmaLinux OS Foundation <cloud-infra@almalinux.org> (@AlmaLinux)\nTags: latest, 9, 9.4, 9.4-20240506\nArchitectures: amd64, arm64v8, ppc64le, s390x\nGitRepo: https://github.com/AlmaLinux/container-images.git\nGitFetch: refs/heads/9\nGitCommit: 35df378109dcb34fe4aa7c6105ec748bafcda307\nDirectory: .\nFile: Dockerfile\namd64-Directory: default/amd64\narm64v8-Directory: default/arm64\nppc64le-Directory: default/ppc64le\ns390x-Directory: default/s390x"
failed fetching "35df378109dcb34fe4aa7c6105ec748bafcda307:refs/remotes/temp3980198148/temp": object not found
failed fetching "refs/heads/9:refs/remotes/temp3980198148/temp": object not found

And this is how it works with updated go-git built from 91fdec5:

$ rm ~/.cache/bashbrew/git -rf
$ ~/Space/go/bin/bashbrew --debug build library/almalinux
Using bashbrew/cache:e03d403fd20245a12d30c2a7470ce2abad2ee862b4e4ebc725e8fbbe04ca6b89 (almalinux:8)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "almalinux:8" "--tag" "almalinux:8.9" "--tag" "almalinux:8.9-20240410" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:e03d403fd20245a12d30c2a7470ce2abad2ee862b4e4ebc725e8fbbe04ca6b89
 ---> 7cd74b4cdf6c
Successfully built 7cd74b4cdf6c
Successfully tagged almalinux:8
Successfully tagged almalinux:8.9
Successfully tagged almalinux:8.9-20240410
Using bashbrew/cache:d155f03f6d23cbe78009c9e609250c7ad47e8557610f089d2f30461fb9af088a (almalinux:8-minimal)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "almalinux:8-minimal" "--tag" "almalinux:8.9-minimal" "--tag" "almalinux:8.9-minimal-20240410" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:d155f03f6d23cbe78009c9e609250c7ad47e8557610f089d2f30461fb9af088a
 ---> de7d13df44bc
Successfully built de7d13df44bc
Successfully tagged almalinux:8-minimal
Successfully tagged almalinux:8.9-minimal
Successfully tagged almalinux:8.9-minimal-20240410
Using bashbrew/cache:c4428d08811b7acdc6d5f7d9649429def8f34a1d357e533787add1e7c5b4f813 (almalinux:latest)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "almalinux:latest" "--tag" "almalinux:9" "--tag" "almalinux:9.4" "--tag" "almalinux:9.4-20240506" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:c4428d08811b7acdc6d5f7d9649429def8f34a1d357e533787add1e7c5b4f813
 ---> 4d8f626ad1a3
Successfully built 4d8f626ad1a3
Successfully tagged almalinux:latest
Successfully tagged almalinux:9
Successfully tagged almalinux:9.4
Successfully tagged almalinux:9.4-20240506
Using bashbrew/cache:4ddb99bfa953cffb29060911a389a00add0dcfef65142581cd3b04adf8e838cc (almalinux:minimal)
$ export DOCKER_BUILDKIT=0
$ docker ["build" "--tag" "almalinux:minimal" "--tag" "almalinux:9-minimal" "--tag" "almalinux:9.4-minimal" "--tag" "almalinux:9.4-minimal-20240506" "--rm" "--force-rm" "-"]
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/1 : FROM bashbrew/cache:4ddb99bfa953cffb29060911a389a00add0dcfef65142581cd3b04adf8e838cc
 ---> d14bf457534c
Successfully built d14bf457534c
Successfully tagged almalinux:minimal
Successfully tagged almalinux:9-minimal
Successfully tagged almalinux:9.4-minimal
Successfully tagged almalinux:9.4-minimal-20240506

Felixoid · 2024-05-08T14:31:15Z

and BuildKit doesn't currently even do direct fetch-by-commit (it fetches all refs hoping one of them contains the requested commit) 😭

I think it should really fetch only the necessary commit. Otherwise, the amount of unnecessary network traffic and compute time is enormous.

Felixoid · 2024-05-09T19:55:09Z

To get rid of possible remaining questions, here's the run for all the official library manifests:

> time ( : > /tmp/bashbrew.log; for lib in $(ls library/*); do  ~/Space/go/bin/bashbrew --debug build --dry-run  "$lib" || echo "Failed '$lib'" >> /tmp/bashbrew.log ; [ -f /tmp/test-stop ] && break || :; done ; ) &> /tmp/bashbrew-full-run.log 2>&1
( : > /tmp/bashbrew.log; for lib in $(ls library/*); do;  --debug build   ||   33,57s user 40,60s system 106% cpu 1:09,48 total

> cat /tmp/bashbrew.log

> du -hs ~/.cache/bashbrew
3,8G    /home/felixoid/.cache/bashbrew

With the full output for bashbrew-full-run.log

Felixoid · 2024-05-10T11:17:18Z

Kindly hope for the next review round

Fetch only necessary commits for build

a31e673

Felixoid requested review from tianon and yosifkit as code owners May 7, 2024 08:25

Felixoid mentioned this pull request May 7, 2024

Initial commitment of clickhouse official image docker-library/official-images#15846

Open

10 tasks

Felixoid added a commit to ClickHouse/docker-library-official-images that referenced this pull request May 7, 2024

Test docker-library/bashbrew#95

96f9446

Felixoid added a commit to ClickHouse/docker-library-official-images that referenced this pull request May 7, 2024

Test docker-library/bashbrew#95

ea69aaf

Felixoid added a commit to ClickHouse/docker-library-official-images that referenced this pull request May 7, 2024

Test docker-library/bashbrew#95

41a8805

Updated go-git as go get github.com/go-git/go-git/v5@v5.12.0

91fdec5

Remove warning about deprevated ioutil

06b49a9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fetch only necessary commits for build #95

Fetch only necessary commits for build #95

Felixoid commented May 7, 2024 •

edited

Felixoid commented May 7, 2024 •

edited

Felixoid commented May 7, 2024 •

edited

Felixoid commented May 7, 2024 •

edited

tianon commented May 7, 2024

Felixoid commented May 7, 2024 •

edited

Felixoid commented May 8, 2024 •

edited

Felixoid commented May 8, 2024 •

edited

Felixoid commented May 8, 2024

Felixoid commented May 9, 2024

Felixoid commented May 10, 2024

Fetch only necessary commits for build #95

Are you sure you want to change the base?

Fetch only necessary commits for build #95

Conversation

Felixoid commented May 7, 2024 • edited

Felixoid commented May 7, 2024 • edited

Felixoid commented May 7, 2024 • edited

Felixoid commented May 7, 2024 • edited

tianon commented May 7, 2024

Felixoid commented May 7, 2024 • edited

Felixoid commented May 8, 2024 • edited

Felixoid commented May 8, 2024 • edited

Felixoid commented May 8, 2024

Felixoid commented May 9, 2024

Felixoid commented May 10, 2024

Felixoid commented May 7, 2024 •

edited

Felixoid commented May 7, 2024 •

edited

Felixoid commented May 7, 2024 •

edited

Felixoid commented May 7, 2024 •

edited

Felixoid commented May 7, 2024 •

edited

Felixoid commented May 8, 2024 •

edited

Felixoid commented May 8, 2024 •

edited