Add options for including build dependencies in compiled output #1681
Conversation
|
@apljungquist could you also document the new feature? |
tests/test_cli_compile.py
Outdated
| "--no-annotate", | ||
| "--no-emit-options", | ||
| "--no-header", | ||
| "--build-system-requires", |
There was a problem hiding this comment.
We also need a test case showing how to only extract the build system requirements without the runtime requirements.
Oh, and how about being able to separately request sdist build requirements, wheel build requirements and editable install requirements?
There was a problem hiding this comment.
We also need a test case showing how to only extract the build system requirements without the runtime requirements.
Makes sense given that the CLI flag presents this as an option. Will fix.
Oh, and how about being able to separately request sdist build requirements, wheel build requirements and editable install requirements
Good catch, I had missed that build systems can state additional requirements for sdist and wheel. But I cannot find any information on editable installs, do you remember where you learned about it?
I am however unsure about the use case for locking sdist and wheel requirements separately. What do you have in mind?
If there is no clear use case, is it worth increasing the size of the UI and the test matrix?
There was a problem hiding this comment.
But I cannot find any information on editable installs, do you remember where you learned about it?
https://peps.python.org/pep-0660/#get-requires-for-build-editable
I am however unsure about the use case for locking sdist and wheel requirements separately. What do you have in mind?
The fact that they are dynamically calculated suggests that they may be different or even conflicting.
Also, building a wheel from a Git checkout may be different than building a wheel from an sdist due to possible inconsistencies in what files are bundles in a wheel.
I imagine the same may happen with the editable checkout. Also, I could see a case when the editable install deps include some development helpers that further restrict the constraints.
If there is no clear use case, is it worth increasing the size of the UI and the test matrix?
It's probably worth thinking about edge cases. One case that I just realized is that the build deps can have env markers (both direct and transitive). This means that there could be different constraints produced depending on which OS/arch/Python version is used to generate them. This is an interesting case to explore. I currently experiment with having a matrix of constraint files per env combo, for runtime. Having that for build deps will probably be useful for non pure Python projects...
There was a problem hiding this comment.
We also need a test case showing how to only extract the build system requirements without the runtime requirements.
Makes sense given that the CLI flag presents this as an option. Will fix.
Now that I read the code again I am not sure what I was referring to. Instead it seems to me that we would have to add another flag to accomplish this e.g. --no-install-requires. This opens up questions like how such an option should interact with the --extra argument and I am starting to think that this should be treated as a separate issue.
I am also thinking about what would be a good interface for telling pip-compile what build requirements to include. I'm leaning towards replacing the previously proposed --build-system-requires flag with an --all-build flag and a --build parameter that works similarly to the --all-extra flag and --extra parameter respectively but takes editable, sdist or wheel as arguments.
Whatever the shape of the interface I think I need to spend some time exploring how the hooks work and how they can be tested. I will try to make some time for this over the coming weeks.
There was a problem hiding this comment.
Extras are tightly coupled with the normal runtime deps. So if they aren't used, they shouldn't be pulled in...
There was a problem hiding this comment.
I have implemented support for this but I am not sure I like it because it adds complexity to both the public interface and the implementation and it is not clear to me that anyone will actually use the feature.
|
@apljungquist would you mind addressing the comments? |
|
Thanks for the reminder. I will take care of it this weekend.
…On Tue, Nov 22, 2022, 20:22 Sviatoslav Sydorenko ***@***.***> wrote:
@apljungquist <https://github.com/apljungquist> would you mind addressing
the comments?
—
Reply to this email directly, view it on GitHub
<#1681 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABQ7T76VKSB7JW5HEYTPIDLWJUMRBANCNFSM6AAAAAAQPNJLRQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@apljungquist @webknjaz Just to let you know that I'm really looking forward to this - thanks a lot! Please let me know if there is anything I can help with. |
Thanks for the encouragement. Can I ask how you plan to use it? |
@apljungquist In nix-community/dream2nix#401 we're aiming for fully reproducible (build) environments. |
tests/test_data/packages/small_fake_with_build_deps/backend/backend.py
Outdated
Show resolved
Hide resolved
|
It was confused that GH was talking about some mysterious required statuses, which is why I was pushing all the buttons to closer/reopen the PR and such. At some point, it occurred to me that it doesn't track my prior approval for some reason (eventual consistency bugs?) and I re-approved, and then I saw GH doing what it should've done earlier — the PR got queued up. It should merge this, once the testing completes. |
|
Oh, and additionally, Jannis dropped the requirement for the branches to be up-to-date, which doesn't make sense with merge queues. |
|
🎉 @apljungquist thanks again for such an involved and well-thought contribution that took over a year to get polished! |
|
Thanks. And thank you for patiently helping me! |
Fixes #1396. Add options for including build dependencies in the output from pip-compile. This allows users to improve the reproducibility of their CI since they can ensure that the same versions of build dependencies are used to build their package every time.
Note that build dependencies of build dependencies are not included in the output. This is one reason why builds will still not be completely reproducible.
Contributor checklist
Maintainer checklist
backwards incompatible,feature,enhancement,deprecation,bug,dependency,docsorskip-changelogas they determine changelog listing.