To enable reproducible builds AbstractArchiver#addFileSet should add the files in order

[plamentotev]

In order to enable reproducible builds, Archiver instances should add their entries in some predictable and reproducible way. To this end AbstractArchiver#addFileSet should add the files in some order. Sorted alphabetically for example.

As the current implementation does not guarantee any particular order we could just modify it to add the entries in alphabetical order, but we may add addition argument that indicates if the entries should be sorted. What do you think?

To implement this a change in PlexusIoFileResourceCollection is required as well. I've opened a ticket for it too - codehaus-plexus/plexus-io#18

[hboutemy]

no need to sort entries, which would cost some CPU cycles
Commons Compress has been updated to keep order of entries vs order of addition: see https://issues.apache.org/jira/browse/COMPRESS-485

[plamentotev]

Correct me if I'm wrong but the change in Common Compress keeps the entries in the same order they are added. But what if we add a directory? Is there any guarantee that the files will be iterated in any particular order (added to the archive in any particular order)?

…

On 16.8.2019 г. 22:46, Hervé Boutemy wrote: no need to sort entries, which would cost some CPU cycles Commons Compress has been updated to keep order of entries vs order of addition: see https://issues.apache.org/jira/browse/COMPRESS-485 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#114?email_source=notifications&email_token=ADBOQ3INEGIHT6KDZB2RBDTQE372NA5CNFSM4HSOPKYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4PRDAA#issuecomment-522129792>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADBOQ3ML4VSCLX7TDKYSPF3QE372NANCNFSM4HSOPKYA>.

[hboutemy]

Yes, the change in Common Compress keeps the entries in the same order they are added.
I don't know really if the addition is really reproducible: nobody wrote any test, I'm going step by step, fixing non-reproducible parts as they are clearly identified.
I just incorporated the current fixes (order as added + timestamp) in maven-sources-plugin to test: https://github.com/apache/maven-source-plugin/tree/MSOURCES-120
The result I got is:

• on my Linux machine, with JDK 7, I get reproducible content from one git checkout
• but as soon as I change JDK version, I get a different result in a reproducible manner (tested with JDK 8 and JDK 9, each giving a different sha1)
• and even with JDK 7, if I build from another checkout, I get another sha1
• I didn't yet test on other OSes

then clearly, order in which files are iterated is not reproducible, as you guessed (but did not prove ;) )
need to see more precisely where this can be done...

[michael-o]

@hboutemy, I can easily test on FreeBSD with Java 7, 8, 11, 12 if you tell me what to do.

[hboutemy]

API added in plexus-archiver (using codehaus-plexus/plexus-io#18 and codehaus-plexus/plexus-utils#70 updates)
Please test maven-source-plugin MSOURCES-120 branch to check that it works in any situation and report in associated Jira issue

[plamentotev]

@hboutemy great work!