-
-
Notifications
You must be signed in to change notification settings - Fork 4.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: improve Composer's output reproducibility #11663
feat: improve Composer's output reproducibility #11663
Conversation
Sorry but I am really confused as to what this tries to achieve? :) It seems to Also not sure how it relates to #7049 at all.. But then again there are so many posts talking about different things in that thread, maybe I missed your point. |
Hi @Seldaek, I apologize for any confusion. This PR is part of an initiative to achieve reproducible builds with By zeroing out the timestamps of all This is totally related to #7049 from @lstrojny since the goal of that PR was precisely what I'm doing here. Ensuring reproducible For more context on why reproducible builds are important, I'd refer you to this page: https://reproducible-builds.org/docs/buy-in/ I've added a screenshot demonstrating the default non-deterministic behavior of Composer. Running I believe this feature would be a significant step in making PHP applications more secure and predictable through Composer. Additionally, this would greatly benefit Linux distributions that aim to package PHP applications in a secure and predictable manner. |
Yeah ok, I have already fixed a ton of issues with reproducibility so you don't need to convince me, but this one here seems like the nuclear option to me. We could perhaps by default copy the filemtime of the composer.lock to the files Composer creates in vendor? That would be more sensible IMO than 0, and the other files in vendor should not need modifying as these have filemtime controlled by their own package source, and zeroing these feels really wrong to me. On the other hand, we also touch the vendor dir itself after dependencies get modified (https://github.com/composer/composer/blob/main/src/Composer/Installer.php#L801), but this probably could be changed to match the lock file's mtime? I think it would still achieve the original goal of #2764 |
Yes, this is also an option, as long as this is predictable, it's fine. I'll modify the PR accordingly right now and report back. |
Ok so could we do this without --deterministic flag? I'd really rather have these things just work out of the box, and avoid special behavior where possible. Also could the Regarding the autoload suffix, I think we could always use the lock file hash if there is one, that's probably unique enough. If there is no lock file available then just pass null and have it default to generating one or reusing existing one. You anyway won't have anything deterministic without lock file so it shouldn't matter in that case. |
This is assuming you don't need to carpet bomb the whole vendor dir with |
LGTM aside from the two minor comments, thanks for the quick turnaround here. Given the scope change I now think it's safe to move this to 2.6 too. |
Thanks |
@Seldaek Do you think it would be possible to edit the Github changelog and add a line about this, so I can spread the word about it ? |
Sure, added to https://github.com/composer/composer/releases/tag/2.6.4 sorry :) |
Published in the September report of Reproducible Builds initiative! https://reproducible-builds.org/reports/2023-09/ |
Since composer/composer#11663, the composer output is stable and reproducible, therefore, that prefix is now obsolete.
Since composer/composer#11663, the composer output is stable and reproducible, therefore, that prefix is now obsolete.
Since composer 2.6.4 [0], the autoloader suffix reuses the content-hash from the lock file in order to make the builds more reproducible [1]. This lead to a conflict when analyzing scip-php itself when using the Docker container. [0] https://github.com/composer/composer/releases/tag/2.6.4 [1] composer/composer#11663
Since composer 2.6.4 [0], the autoloader suffix reuses the content-hash from the lock file in order to make the builds more reproducible [1]. This lead to a conflict when analyzing scip-php itself when using the Docker container. [0] https://github.com/composer/composer/releases/tag/2.6.4 [1] composer/composer#11663
The issue was likely caused by Composer 2.6.4 making the autoloader generation (more) reproducible (composer/composer#11663) We can either try to change the generated autoloader with an autoloader suffix, or just change the Psalm root directory for the smoke test. The latter approach seems easier. :P
Composer 2.6.4+ wants to have reproducible outputs so it preserves `mtime` when copying files: composer/composer#11663 This becomes an issue with `vendor/composer/ClassLoader.php`, which is copied from Nix store so it has `mtime` of 0 (1970-01-01), because that is not supported by ZIP: ValueError: ZIP does not support timestamps before 1980 Let’s disable strict timestamps to clamp the out-of-range timestamps to ZIP epoch. That argument was actually introduced because of reproducible builds: https://bugs.python.org/issue34097
Composer 2.6.4+ wants to have reproducible outputs so it preserves `mtime` when copying files: composer/composer#11663 This becomes an issue with `vendor/composer/ClassLoader.php`, which is copied from Nix store so it has `mtime` of 0 (1970-01-01), because that is not supported by ZIP: ValueError: ZIP does not support timestamps before 1980 Let’s disable strict timestamps to clamp the out-of-range timestamps to ZIP epoch. That argument was actually introduced because of reproducible builds: https://bugs.python.org/issue34097
This PR:
vendor
directory). Currently, only the file content is guaranteed to be reproducible, the file time metadata are not (yet?) reproducible.Reproduce this at home, inside the
composer
source directory: