Selecting a Python interpreter #2386

zanieb · 2024-03-12T16:50:58Z

In #2338 some questions came up around Python interpreter selection. Here we'll summarize our expected behavior and have some discussion around it.

Interpreters can be selected with the following formats:

Direct path e.g. /bin/python3
Version e.g. 3.10
Implementation e.g. pypy
Implementation and version e.g. pypy3.10 or cpython3.10

Using python as the implementation should allow any implementation. Its behavior should be equivalent to X.Y.

Python interpreters may be discovered via:

Path directly provided
The spawning interpreter (i.e. python -m uv)
An activated virtual environment
A virtual environment in the current directory
Executable found in PATH

When a user does not specify an interpreter, we should do our best to infer one in the order above.

When a user specifies an interpreter via a direct path:

We should only use the exact provided interpreter
We do not check the version or implementation, though they must be supported by uv

When a user specifies an interpreter via a version:

We do not check the interpreter implementation
We should follow the discovery order, but ignore interpreters that do not satisfy the version
If we are compiling dependencies, we can fallback to a mismatched version if no matching version is found

When a user specifies an interpreter with an implementation (and version):

If the implementation is unknown, we should error and recommend a direct path.
We should follow the discovery order, but ignore interpreters that do not satisfy the implementation (and version)
If we are compiling dependencies, we can fallback to a matching implementation with a mismatched version if no matching version is found

Interpreters can either belong to a virtual environment or the system.

When an interpreter belongs to the system, we will not modify packages (i.e. install, uninstall) in the interpreter's environment without the --system flag or specification of the interpreter with a direct path. We use system interpreters for compiling dependencies without opt-in.

Examples

Compile using system Python interpreter discovered by being the spawning intepreter:

python -m uv pip compile ...  # Ok: System Python used

Install using system Python interpreter discovered by being the spawning intepreter:

python -m uv pip install ...   # Error: System Python cannot be used without opt-in `--system`

Install using system Python interpreter discovered by direct path:

uv pip install --python /bin/python ...   # Ok: System can be used with direct path

Compile using system Python interpreter discovered by being the spawning intepreter with matching version:

python3.10 -m uv pip compile --python 3.10  # Ok: System Python used since version matches

Compile using Python interpreter discovered on path since spawning intepreter with mismatched version:

python3.10 -m uv pip compile --python 3.12  # Ok: Python 3.12 is discovered on path since version does not match spawning

Install using Python interpreter in the working directory:

uv venv
uv pip install ...  # Ok: Python is discovered in the working directory

Install using Python interpreter in the active virtual environment:

uv venv
source .venv/bin/activate
mkdir foo && cd foo
uv pip install ...  # Ok: Python is discovered from `VIRTUAL_ENV`

Install using Python interpreter in the active virtual environment with mismatched version:

uv venv --python 3.10
source .venv/bin/activate
uv pip install --python 3.12  # Error: Requested Python version does not match version from virtual environment

Compile using Python interpreter in the active virtual environment with mismatched version:

uv venv --python 3.10
source .venv/bin/activate
uv pip compile --python 3.12  # Ok: Either finds 3.12 on path or uses 3.10 for compilation with a warning

Install using Python interpreter via direct path with active virtual environment:

uv venv --python 3.10
source .venv/bin/activate
uv pip install --python ~/python/python3.12  # Ok: Uses direct path and ignores virtual environment entirely

Install using Python with an unknown implementation:

uv pip install --python foobar3.12  # Error: Unknown implementations are only allowed via direct paths

Install using Python with an unknown implementation via direct path:

uv pip install --python /bin/foobar3.12  # Ok: As long as we support the discovered implementation

The text was updated successfully, but these errors were encountered:

konstin · 2024-03-12T16:56:38Z

Do we support looking up -p my_not_cpython3.10, where my_not_cpython3.10 is in PATH but my_not_cpython is not a known implementation (currently cpython, pypy and pyston)? If so, do we give the user indication whether a prefix was recognized or not? If not, should we remove the current PATH lookup support?

zanieb · 2024-03-12T16:58:24Z

@konstin I think we should explicitly not support that and require a direct path if you want to use an unknown implementation.

(edited the post to reflect this)

charliermarsh · 2024-03-12T17:01:12Z

(For clarity: we do support that today, I think. The behavior Zanie proposed above would be a change.)

See #2386 for the motivation. We keep only two behaviours for `-p`, we have an absolute or relative path (containing a system path separator) or `-p <implementation><version>`. Open question: Should `python3.10` or `3.10` force cpython 3.10 or is any python implementation acceptable? My assumption right now is that i assume that a user specifying nothing wouldn't like it if we picked pypy, rather we should force cpython and make pypy explicit. I will add tests tomorrow, please already give the semantics a look.

konstin · 2024-03-12T17:51:45Z

Another question from #2388: With -p 3.10, would we pick pypy, or should we only do cpython by default and alternative implementations be an opt-in? Given how incompatible other implementations are, i'm in favor of cpython-by-default and leaving pypy/pyston for explicit opt-in.

zanieb · 2024-03-12T17:57:08Z

We've got a couple discussions going about this elsewhere:

I think those make it clear that we should support pypy without opt-in and I think the top-level post here reflects that already.

konstin · 2024-03-12T18:00:02Z

#2316 (comment):

Yes, if no other python is present.

The way I interpret it, python or the lack of the implementation basically translates to any implementation. However for historical reasons if multiple implementations are available cpython wins.

That would again we very different from what we currently, this requires us to collect all python implementations and rank!

zanieb · 2024-03-12T18:07:24Z

I think it's fine to take the first one on the path and not support the historical discovery noted there. We can consider something more complicated if we hear it causes problems?

charliermarsh · 2024-03-12T18:45:58Z

When a user provides, e.g., cpython3.10, how we will find the relevant executables in PATH?

zanieb · 2024-03-12T18:51:20Z

I think when an implementation is provided we could

First, scan for an exact match in the PATH
If not found, scan for pythonX.Y and python in PATH and check if the implementation matches

We could also consider behavior specific to the implementation and the binaries we know they ship (this probably makes a lot of sense but I don't know what's common). Since we require you to match a known implementation we could definitely do this.

mkleinbort-ic · 2024-03-14T13:59:41Z

Just to add to this...

I was having issues today where python -m uv pip install --system --prerelease=allow -r pyproject.toml was not updating my dependencies as per the pyproject.toml

It seems the issue is that which python
returns /c/Users/mkleinbort/AppData/Local/Programs/Python/Python311/python

but which python3 returns /c/Users/mkleinbort/AppData/Local/Microsoft/WindowsApps/python3

Not hard to fix, but I would have expected the original command to install the dependencies in the python as referenced by python

haarisr · 2024-04-03T17:10:01Z

Might not be exactly related

But how about writing to a file such as pyproject.toml under tool.uv or a uv specific file when the venv is created in the first place.

If not uv writing to the file, let the user specify it under the toml so that uv can look it up and activate that venv/python on running a python/uv command.

This also might blend well with creating a default venv when there's no venv present in the current directory in #2665 and the path of the venv can be added/appended to the toml file.

charliermarsh · 2024-04-16T15:18:21Z

A brief proposed addendum on top of the above logic, quoting #3060. When looking at a --python value...

If the --python value is a path to an existing file, use that as the interpreter.
If it's a directory, assume it's a venv and use the venv interpreter.
Otherwise, if it contains a path separator, fail with an error.
Finally (a filename with no path part) try a search on PATH.

Moves all of `uv-toolchain` into `uv-interpreter`. We may split these out in the future, but the refactoring I want to do for interpreter discovery is easier if I don't have to deal with entanglement. Includes some restructuring of `uv-interpreter`. Part of #2386

Updates our Python interpreter discovery to conform to the rules described in #2386, please see that issue for a full description of the behavior. Briefly, we now will search for interpreters that satisfy a requested version without stopping at the first Python executable. Additionally, if retrieving information about an interpreter fails we will continue to search for a working interpreter. We also add the plumbing necessary to request Python implementations other than CPython, though we do not add support for other implementations at this time. A major internal goal of this work is to prepare for user-facing managed toolchains i.e. fetching a requested version during `uv run`. These APIs are not introduced, but there is some managed toolchain handling as required for our test suite. Some noteworthy implementation changes: - The `uv_interpreter::find_python` module has been removed in favor of a `uv_interpreter::discovery` module. - There are new types to help structure interpreter requests and track sources - Executable discovery is implemented as a big lazy iterator and is a central authority for source precedence - `uv_interpreter::Error` variants were split into scoped types in each module - There's much more unit test coverage, but not for Windows yet Remaining work: - [x] Write new test cases - [x] Determine correct behavior around executables in the current directory - _Future_: Combine `PythonVersion` and `VersionRequest` - _Future_: Consider splitting `ManagedToolchain` into local and remote variants - _Future_: Add Windows unit test coverage - _Future_: Explore behavior around implementation precedence (i.e. CPython over PyPy) Refactors split into: - #3329 - #3330 - #3331 - #3332 Closes #2386

konstin self-assigned this Mar 12, 2024

konstin mentioned this issue Mar 12, 2024

Stop looking for -p in PATH #2388

Closed

konstin mentioned this issue Mar 13, 2024

Use python from python -m uv as default #2338

Closed

konstin removed their assignment Mar 13, 2024

zanieb mentioned this issue Mar 16, 2024

[0.1.19 regression] uv no longer uses the Python interpreter set by pyenv #2488

Closed

zanieb mentioned this issue Mar 25, 2024

uv use wrong interpreter when uv venv, and activated wrong environment when activate venv #2649

Closed

bschoenmaeckers mentioned this issue Mar 27, 2024

feat: self-update for default pip installs #2675

Closed

zanieb mentioned this issue Apr 3, 2024

Feature Request: Expose python and venv discovery as a machine interpretable uv subcommand #2813

Open

charliermarsh added the enhancement New feature or request label Apr 16, 2024

This was referenced Apr 16, 2024

Allow the --python flag in uv pip install to take the location of a venv #3060

Closed

uv pip --python doesn't recognise / as a path separator on Windows #3062

Closed

This was referenced Apr 25, 2024

Merge uv-toolchain and uv-interpreter #3265

Merged

Rewrite Python interpreter discovery #3266

Merged

zanieb closed this as completed in #3266 May 21, 2024

strangemonad mentioned this issue Jun 3, 2024

Make the Python experience wonderful zed-industries/zed#7808

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Selecting a Python interpreter #2386

Selecting a Python interpreter #2386

zanieb commented Mar 12, 2024 •

edited

konstin commented Mar 12, 2024

zanieb commented Mar 12, 2024 •

edited

charliermarsh commented Mar 12, 2024

konstin commented Mar 12, 2024

zanieb commented Mar 12, 2024

konstin commented Mar 12, 2024

zanieb commented Mar 12, 2024

charliermarsh commented Mar 12, 2024

zanieb commented Mar 12, 2024 •

edited

mkleinbort-ic commented Mar 14, 2024

haarisr commented Apr 3, 2024 •

edited

charliermarsh commented Apr 16, 2024

Selecting a Python interpreter #2386

Selecting a Python interpreter #2386

Comments

zanieb commented Mar 12, 2024 • edited

Examples

konstin commented Mar 12, 2024

zanieb commented Mar 12, 2024 • edited

charliermarsh commented Mar 12, 2024

konstin commented Mar 12, 2024

zanieb commented Mar 12, 2024

konstin commented Mar 12, 2024

zanieb commented Mar 12, 2024

charliermarsh commented Mar 12, 2024

zanieb commented Mar 12, 2024 • edited

mkleinbort-ic commented Mar 14, 2024

haarisr commented Apr 3, 2024 • edited

charliermarsh commented Apr 16, 2024

zanieb commented Mar 12, 2024 •

edited

zanieb commented Mar 12, 2024 •

edited

zanieb commented Mar 12, 2024 •

edited

haarisr commented Apr 3, 2024 •

edited