feat: improve Composer's output reproducibility

[drupol]

This PR:

• Improve reproducibility of Composer's output (the vendor directory). Currently, only the file content is guaranteed to be reproducible, the file time metadata are not (yet?) reproducible.
• Should fix Reproducable build: deterministic APCU prefix #7049

Reproduce this at home, inside the composer source directory:

[Seldaek]

Sorry but I am really confused as to what this tries to achieve? :) It seems to touch() all vendor files with a 0 timestamp. What's the point? And why would you need this flag vs doing it manually with find vendor -type f -exec touch {} 0 or smth if it's a special need it sounds ok to me?

Also not sure how it relates to #7049 at all.. But then again there are so many posts talking about different things in that thread, maybe I missed your point.

[drupol]

Hi @Seldaek,

I apologize for any confusion. This PR is part of an initiative to achieve reproducible builds with composer. The aim is to ensure that composer can create a vendor/ directory that is bit-for-bit identical across multiple runs of the same command.

By zeroing out the timestamps of all vendor/ files, we eliminate one variable that could make builds non-deterministic. Integrating this feature into composer itself would streamline the process and make it more accessible for users focused on reproducibility (debian, NixOS, Arch, etc etc).

This is totally related to #7049 from @lstrojny since the goal of that PR was precisely what I'm doing here. Ensuring reproducible vendor/ directories aligns with this goal.

For more context on why reproducible builds are important, I'd refer you to this page: https://reproducible-builds.org/docs/buy-in/

I've added a screenshot demonstrating the default non-deterministic behavior of Composer. Running composer install twice resulted in two different directory hashes. However, when using the new --deterministic flag, the directory hashes were identical in multiple runs.

I believe this feature would be a significant step in making PHP applications more secure and predictable through Composer. Additionally, this would greatly benefit Linux distributions that aim to package PHP applications in a secure and predictable manner.

[Seldaek]

Yeah ok, I have already fixed a ton of issues with reproducibility so you don't need to convince me, but this one here seems like the nuclear option to me.

We could perhaps by default copy the filemtime of the composer.lock to the files Composer creates in vendor? That would be more sensible IMO than 0, and the other files in vendor should not need modifying as these have filemtime controlled by their own package source, and zeroing these feels really wrong to me.

On the other hand, we also touch the vendor dir itself after dependencies get modified (https://github.com/composer/composer/blob/main/src/Composer/Installer.php#L801), but this probably could be changed to match the lock file's mtime? I think it would still achieve the original goal of #2764

[drupol]

Yes, this is also an option, as long as this is predictable, it's fine.

I'll modify the PR accordingly right now and report back.

[drupol]

Just made the modifications, this is all good !

[Seldaek]

Ok so could we do this without --deterministic flag? I'd really rather have these things just work out of the box, and avoid special behavior where possible.

Also could the touch be done directly in the AutoloadGenerator instead of twice in Installer & DumpAutoloadCommand?

Regarding the autoload suffix, I think we could always use the lock file hash if there is one, that's probably unique enough. If there is no lock file available then just pass null and have it default to generating one or reusing existing one. You anyway won't have anything deterministic without lock file so it shouldn't matter in that case.

[Seldaek]

I'd really rather have these things just work out of the box, and avoid special behavior where possible.

This is assuming you don't need to carpet bomb the whole vendor dir with touch, because this is IMO unacceptable behavior to modify vendor files.

[drupol]

I definitely agree that this should be the default behavior.

I reworked the PR from the beginning and this is now the default behavior:

I still need to fix the tests and add new tests.

[Seldaek]

LGTM aside from the two minor comments, thanks for the quick turnaround here. Given the scope change I now think it's safe to move this to 2.6 too.

[Seldaek]

Thanks

[drupol]

@Seldaek Do you think it would be possible to edit the Github changelog and add a line about this, so I can spread the word about it ?

[Seldaek]

Sure, added to https://github.com/composer/composer/releases/tag/2.6.4 sorry :)

[drupol]

Published in the September report of Reproducible Builds initiative! https://reproducible-builds.org/reports/2023-09/