Add --only-deps (and --only-build-deps) option(s)

[flying-sheep]

#11440 (comment) is the currently agreed upon user-facing design for this feature.

---

What's the problem this feature will solve?

In #8049, we identified an use case for installing just the dependencies from pyproject.toml.

As described in the solution section below --only-deps=<spec> would determine all dependencies of <spec> excluding that package itself and install those without installing the package. It could be used to

1. allow specifying environment variables that are active only while building a package of interest (without having it be active while potentially building its dependencies).
2. separate dependency installation from building and installing a package, allowing to rebuild a package in a docker build while the dependency installation step is loaded from cache.

This example shows both use cases:

# copy project metadata and install (only) dependencies
COPY pyproject.toml /myproj/
WORKDIR /myproj/
RUN pip install --extra-index-url="$PIP_INDEX" --only-deps=.[floob]

# copy project source files, build in a controlled environment and install our package
COPY src/mypkg/ /myproj/src/mypkg/
RUN env SETUPTOOLS_SCM_PRETEND_VERSION=2.0.2 python3 -m build --no-isolation --wheel
RUN pip install --no-cache-dir --no-dependencies dist/*.whl

Instead of the solution from #8049, @pradyunsg prefers a solution similar to the one below: #8049 (comment)

Describe the solution you'd like

One of those two, or similar:

1. (used in the example above)

   --only-deps would work like -r in that it’s not a flag globally modifying pip’s behavior but a CLI option with one argument that can be specified multiple times. Unlike -r it accepts a dependency spec and not a path to a file containing dependency specs.

   Where pip install <spec> first installs all dependencies and then (build and) install the package referred to by the spec itself, pip install --only-deps=<spec> would only install the dependencies.

2. --only-deps would work like --[no|only]-binary, in that it requires an argument specifying what package not to install. A placeholder like :requested: could be used, e.g.:

   pip install --only-deps=:requested: .[floob]

Alternative Solutions

• Re-using -r instead of adding --only-deps.

  I don’t think this is a good idea, since people would be tempted to do -r pyproject.toml which would be wrong (Dependency specs including file paths look like like ./path/to/pkg[extra1,extra2])

• Making --only-deps a global switch modifying pip’s behavior like e.g. --pre.

  I have found that global switches like that are dangerous and not very intuitive. To install a dev version of your package, doing pip install --pre mypkg seems innocuous but will actually install dev versions of mypkg and all its dependencies that have any dev versions. It’s safer to do something like pip install mypkg>=0.1.post0.dev0 to limit dev version installations to one package. Similarly it’s unclear what a --only-deps switch would apply to. Would pip install -r reqs.txt --only-deps install the dependencies of every package specified in the file but none of those packages?

• Using e.g. beni to convert PEP 621 dependencies to a requirements.txt.

  This works even today but feels like is shouldn’t be necessary as it involves quite a few steps, including writing a file to disk.

Additional context

NA

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

Thanks for filing this @flying-sheep!

I wonder if it would be better for --only-deps to mirror how --no-deps behaves.

[flying-sheep]

If I interpret the spartan docs for --no-deps correctly, it doesn’t take an argument. I added motivation to “Alternative Solutions” on why I think --only-deps should accept an argument. Do you disagree? If yes: why? If no, (and you didn’t change your mind) it seems like I didn’t understand what you mean: in what way should --only-deps work like --no-deps?

[uranusjr]

There’s actually yet another possibility: make --only-deps a global switch that takes one single value, like --no-binary and --only-binary.

I think I personally like --only-deps=<names> best, followed by --only-deps=<spec> (proposed in this PR), and a value-less --only-deps last (and we really should find a way to make --no-deps work the same as --no-binary).

[flying-sheep]

I see, those options are modifiers that pick out individual packages from the flattened dependency list and modify pip’s behavior towards that ones. So one would do:

cd mypkg  # project dir
pip install --only-deps=mypkg .[floob]

I think it makes sense regarding consistency with --[no|only]-binary, but isn’t 100% practical, as the only use case that came up so far is the one above, so users will always have to specify both dist name and relative path to the project.

[uranusjr]

It may make sense to create wild card names e.g. :requested: to simplify the UX somewhat. --no-binary etc. have :all:, which of course does not make sense for --only-deps, but can be a good inspiration.

[flisboac]

This would be extremely useful to prepare lambda layers, or any kind of "pre-provided" environment, whilst keeping the exact requirements (including locks) properly versioned in a git repository. The target environment could then be replicated with ease, e.g. when developing locally, or when testing.

Follows an example pyproject.toml:

[build-system]
requires = [
    "setuptools >= 45",
    "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "my-lambda"
requires-python = ">= 3.7"
version = "0.1.0"

# Again, this is just an example!
[project.optional-dependencies]
provided = [
    "typing-extensions >= 4",
    "requests ~= 2.23.0",
    "requests_aws4auth ~= 0.9",
    "boto3 ~= 1.13.14",
    "certifi >= 2020.4.5.1",
    "elasticsearch ~= 7.7.0",
    "elasticsearch_dsl ~= 7.2.0",
    "aws_requests_auth ~= 0.4.2",
]
pre-commit = [
    'nox >= 2022.1',
    'pytest >= 7.1.2',
    'black[d] >= 22',
    'mypy >= 0.950',
    'pre-commit >= 2.17.0',
    'flake8 >= 4; python_version >= "3.8"',
    'flake8 < 4; python_version < "3.8"',
    'pydocstyle[toml] >= 6.1.1',
    'isort >= 5.10.1',
]

Then, when creating a new "provided" environment (e.g. a lambda layer):

# Must be run in a similar environment as the target one.
# Advantage of this over `pip download` is the ability
# of mixing source and binary distributions, whenever
# necessary (e.g. downloading both numpy and pyspark).
# Could also add locks/pinning, via `--constraint`.

mkdir -p dist/python
pip3 install \
  .[provided] \
  --target dist/python \
  --only-deps=:requested:
( cd dist && zip ../dist.provided.zip ./python )

And in a development or CI-like environment:

# May be cached.
python3 -m venv venv
source ./venv/bin/activate

# Gets all development tools, and anything used to run
# automated tasks.
# Could also add locks/pinning, via `--constraint`.
./venv/in/pip3 install -e .[provided,pre-commit]

[flying-sheep]

@uranusjr Sure, I’m not married to the semantics I suggested. I’m fine with your design. Now we just need someone to implement it lol.

[sbidoul]

Would it be reasonable to have --only-deps work only when exactly one top level requirement is provided, and fail otherwise?

[pradyunsg]

Would it be reasonable to have --only-deps work only when exactly one top level requirement is provided, and fail otherwise?

Yes.

[rgommers]

I'm not sure I understand the :requested: and trying to pick out individual packages. My $2c here is that there are multiple keys in pyproject.toml, and those keys are the only things to be selecting on. The keys are: dependencies, requires (under [build-system]), and then optional-dependencies which may have multiple entries. It'd be great to spell out how all those get installed. Something like:

pip install . --only-deps  dependencies
pip install . --only-deps  requires
pip install . --only-deps  doc  # for a "doc" key under optional-dependencies

And the two comments above say that if you need two of those, you need multiple invocations of pip rather than providing two key names at once?

[pradyunsg]

pip install . --only-deps doc

IMO, pip install --only-deps .[doc] has clearer syntax, especially when not using a PEP 621 style pyproject.toml-based dependency declaration.

None the less, I agree that we don't need the additional complexity here.

[rgommers]

Okay, so:

pip install --only-deps  .  # dependencies
pip install --only-deps  .[doc]  # optional-dependencies `doc`

And just to confirm, pip install . --only-deps .[requires] for the build requirements (meaning it's the optional-dependencies syntax, but requires is reserved and people shouldn't use a requires key under optional-dependencies)?

[sbidoul]

I'd suggest a small variation, to avoid overloading the [extras] syntax.

pip install --only-deps  .  # project.dependencies
pip install --only-deps  .[doc]  # project.dependencies and project.optional-dependencies.doc
pip install --only-build-deps .  # build-system.requires

[netsandbox]

Maybe there are situations where you only want to install the optional dependencies without the project dependencies.
So I would extend @sbidoul suggestion with:

pip install --only-optional-deps  .[doc]  # only project.optional-dependencies.doc

[sbidoul]

Maybe there are situations where you only want to install the optional dependencies without the project dependencies.

I'm not comfortable with that. Indeed extras are additive to the base dependencies by definition, so such a mechanism sounds a bit awkward to me.

[brettcannon]

The desire for this feature just came up in a discussion at work around whether dependencies should be recorded in requirements.txt (since that's where pip has historically put them) or in pyproject.toml (as that's backed by a spec)? And the problem with the latter is it requires you set up a build system so you can do pip install -e ., even if you didn't need one and just want to install stuff that you wrote down somewhere.

The motivating scenario is beginners who have started coding, have some script or package going (i.e., not doing a src/ layout), and now they want to install something (assume we will create a virtual environment for them). We would like to teach people to record what they install and follow standards where we can, but to use pyproject.toml we need to also get a build system chosen and set up which is another level of complicated. Otherwise we would need to forgo standards and go with requirements.txt and decide what the appropriate worfklow is in that scenario (e.g., do you record in requirements.txt, requirements-dev.txt, dev-requirements.txt, requirements/dev.txt, etc.?).

[dstufft]

Without a build system there's no way for pip to determine what the dependencies are, pip doesn't (afaik, and it shouldn't) read dependencies from pyproject.toml (other than build dependencies), it just looks for a build system and then asks the build system what the dependencies are. It is the responsibility of the build system to read pyproject.toml and determine what (if any) dependencies there are.

[uranusjr]

@brettcannon In the use case, what is the motivation behind not including . in the installation?

[brettcannon]

Without a build system there's no way for pip to determine what the dependencies are

If dependencies was listed in dynamic I would agree, but if it's not then how it is different than a requirements file?

It is the responsibility of the build system to read pyproject.toml and determine what (if any) dependencies there are.

Are you saying we need to come up with a separate standard to list arbitrary dependencies like requirements files allow for (this is not the same a lock file to me; think of it as the input file of your top-level dependencies for a lock file generation tool)?

what is the motivation behind not including . in the installation?

Beginners don't typically need it (i.e., they aren't doing a src/ layout), and I personally don't want to try and educate a beginner trying to install something on why they suddenly need to select a pure Python build backend to record what they installed and how to make an appropriate decision.

[pfmoore]

If dependencies was listed in dynamic I would agree, but if it's not then how it is different than a requirements file?

Yes, it's technically true that if the pyproject.toml states that the dependencies are non-dynamic, then reading that file for the data is valid. But during the discussions on PEP 621, the idea of relying on metadata from pyproject.toml like this was pretty controversial. I don't have a specific reference, but I do recall being told that the intended use was very much for backends to read the data, not for it to be the canonical source (I proposed PEP 643 mainly because reading pyproject.toml from the sdist was "not the intended use").

I understand the use case, and in a practical sense, getting the data from pyproject.toml would work (until someone wants it to work with dynamic dependencies, and we have a debate over why that is out of scope). But I don't think it's the right way for pip to go.

Is there a reason this couldn't be an external tool?

# Warning: code has only been lightly tested!
with open(FILE, "rb") as f:
    data = tomllib.load(f)

if "project" not in data:
    raise ValueError("No PEP 621 metadata in pyproject.toml")
if "dependencies" in data["project"].get("dynamic", []):
    raise ValueError("Dependencies cannot be dynamic")

deps = data["project"].get("dependencies")

if deps:
    cmd = [sys.executable, "-m", "pip", "install", *deps]
    subprocess.run(cmd)

[sbidoul]

Thanks for the detailed analysis @pfmoore. This perfectly matches my understanding of the matter.

[pfmoore]

Cool. So where does that leave us - are we simply waiting at this point for someone to implement the feature based on this analysis? I think this approach addresses any reservations I had about the feature.

Note that I don't think this addresses --only-build-deps, and I definitely think we should consider that idea separately. It's fundamentally not possible to get the correct set of build dependencies without (1) querying the backend, and (2) knowing what we want to build (as get_requires_for_build_wheel, get_requires_for_build_sdist, and get_requires_for_build_editable are all separate backend calls). So I think we should be very cautious about offering an option that claims to install the "build dependencies" when we know we cannot actually do that... (actually, I'm personally a strong -1 on --only-build-deps, but I'm willing to hear arguments as to why I'm wrong...)

[MohammadRaziei]

I really need this feature. Is it planned for the future?

[stefanv]

In scikit-image, we've resorted to writing out various requirement.txt files each time pyproject.toml is updated.

[webknjaz]

PSA: It will be possible to extract the build deps (with their transitive requirements) of a project in the next pip-tools release — 3 weeks ago we merged a PR that took like a year to polish: jazzband/pip-tools#1681.

@stefanv I also used to maintain a manually extracted copy of the build deps for use with PIP_CONSTRAINT in the past, but the new feature is going to cover that need. If you'd like to try it out, test the version from Git. See https://pip-tools.rtfd.io/en/stable/#maximizing-reproducibility for the hints.

@pfmoore this new pip-tools feature extracts both static and dynamic deps using said callbacks FYI.

P.S. As for the --only-deps feature — I think it's a can of worms, this would encourage projects to expose things through metadata that are meant to be private. Like, why would an end user (not a contributor) need access to the docs or test deps? Those deps usually need env-dependent pinned constraints rather than follow the normal pkg runtime dep considerations… Ah, it looks like PEP 735 is attempting to address this.

[stefanv]

PSA: It will be possible to extract the build deps (with their transitive requirements) of a project in the next pip-tools release — 3 weeks ago we merged a PR that took like a year to polish: jazzband/pip-tools#1681.

From reading that PR, it looks like you will have to compile the package in order to extract the build dependencies?

[webknjaz]

@stefanv no, you can feed it a pyproject.toml. The PEP 517 hooks are called to augment the statically listed build deps. But then, there's also dependency resolution to get the transitive deps, so that might end up building some of the packages in the tree, if that's what you're asking.

[evbo]

Python developers for years if not decades have followed a very simple deploy process for standalone apps:

1. create a venv (virtual environment) and source ./venv/bin/activate
2. pip install - r <requirements.txt>
3. run the code as-is, package it with pyinstaller, or build a wheel as needed...

For reverse compatibility and to unify the Python development experience, as part of this PR could you please add to pyproject.toml a [build-system] setting that disables every other step besides install dependencies? Then we get an even simpler command (pip install .) and identical behavior exactly like pip install -r provided. It's important this be codified inside pyproject.toml so that it's self-documenting how to build the code (without redundantly coding it for CI automation with --only-deps arguments, or simply without having to remember when installing manually, etc.).

tl;dr: adding a only_deps flag to [build-system] is valuable for 3 reasons:

1. complete backwards compatibility with pip install -r into a venv deployment flow described above
2. encourages migration of legacy projects to completely co-locate everything in pyproject.toml
   • Like Rust/Scala/NPM and so many other build systems for other languages
3. self-documenting pyproject.toml
   • No extra coding in CI automation using lesser known --only-deps flags or side-effecting pip-compile

Similar sentiments expressed:
https://stackoverflow.com/questions/74354009/install-dependencies-from-pyproject-toml-but-not-the-package
https://stackoverflow.com/questions/77150911/equivalent-to-pip-install-r-requirements-dev-txt-for-pyproject-toml
astral-sh/uv#1516
pypa/flit#631
#12425
#2171
#11584

[webknjaz]

PSA: It will be possible to extract the build deps (with their transitive requirements) of a project in the next pip-tools release — 3 weeks ago we merged a PR that took like a year to polish: jazzband/pip-tools#1681.

FTR that feature has finally been released in pip-tools v7.4.0. So the build deps constraints/lockfile are extractable using something like pip-compile --all-build-deps --output-file=build-constraints.txt --strip-extras pyproject.toml now. Hopefully, this will provide a workaround for a portion of this feature request.

From reading that PR, it looks like you will have to compile the package in order to extract the build dependencies?

@stefanv I only now realized why you were asking about compiling 😆 The command pip-compile is a pip-tools CLI for making constraint files out of input deps, that's what "compiling" means in that context.

[matterhorn103]

Edit: this was just a plea to remember the science use-case, but it added nothing to the discussion in the light of @webknjaz's reply. Sorry!

[webknjaz]

@matterhorn103 pretty sure that it's pending the acceptance/completion of PEP 735 so that there could be an installer-agnostic way of declaring those. It specifically mentions the scientific projects among the use cases, by the way. Please, check it out, it's an interesting read.

[matterhorn103]

@webknjaz Ah, I should have known better than to comment on such an old thread... Thanks for the reply and the pointers. I also since then found threads like this one discussing the future options and particularly the scientific context.