Using `.*` with operators other than `==` and `!=` #645

jack-melchert · 2022-12-19T22:48:05Z

In packaging 21.3 having a wildcard in the install_requires section of a setup.py worked fine.
Example:

setup(
...
    install_requires=["hwtypes>=1.0.*"],
...
)

In version 22.0 it throws this error:

  File "/aha/lib/python3.8/site-packages/packaging/requirements.py", line 35, in __init__

    parsed = parse_requirement(requirement_string)

  File "/aha/lib/python3.8/site-packages/packaging/_parser.py", line 64, in parse_requirement

    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))

  File "/aha/lib/python3.8/site-packages/packaging/_parser.py", line 82, in _parse_requirement

    url, specifier, marker = _parse_requirement_details(tokenizer)

  File "/aha/lib/python3.8/site-packages/packaging/_parser.py", line 122, in _parse_requirement_details

    marker = _parse_requirement_marker(

  File "/aha/lib/python3.8/site-packages/packaging/_parser.py", line 143, in _parse_requirement_marker

    tokenizer.raise_syntax_error(

  File "/aha/lib/python3.8/site-packages/packaging/_tokenizer.py", line 161, in raise_syntax_error

    raise ParserSyntaxError(

packaging._tokenizer.ParserSyntaxError: Expected end or semicolon (after version specifier)

    hwtypes>=1.0.*

           ~~~~~^

The text was updated successfully, but these errors were encountered:

uranusjr · 2022-12-20T03:37:17Z

I believe this is intended. Wildcard cannot be used with >= (according to PEP 440 it can only be used with == and !=), but prior to 22.0 it’s parsed into a LegacyVersion and appears to work (and “fails” more subtly when a user tries to install the package).

This has caused much confusion I wonder if we should extend the spec a bit to allow * in other operators. For >=, >, <, and <=, the wildcard suffix can simply be ignored (i.e. >= 1.0.* simply means >= 1.0). ~= is more complicated but I think we can continue to disallow the usage for now since I don’t recall anyone has complained about it yet.

pradyunsg · 2022-12-20T07:45:10Z

This isn't a bug per se, but an intentional change.

I do wonder if a slightly nicer error message is a better strategy here, long term than relaxing the syntax to silently ignore part of the input.

pombredanne · 2022-12-22T07:50:10Z

I think failing to process stars is a good thing. Stars are evil. :)

woodruffw · 2022-12-22T17:01:40Z

Looks like this snared a pip-audit user as well: pypa/pip-audit#445

(I think this is a good change! But it looks like there are some packages on PyPI that use >= a.b.* in their data-requires-python constraint, so it's possible that installation of those packages is now broken.)

pradyunsg · 2022-12-22T17:50:01Z

FWIW, the functionality that allowed this is also the functionality that considered the following to be a valid requirement.

❯ echo "foobar >= kgjldsfggiuwbeigvr203492-8n29o3.DEV0.alpha1.*" | python -c "import packaging; print(packaging.__version__); from packaging.requirements import Requirement; print(repr(Requirement(input())))"
21.3
<Requirement('foobar>=kgjldsfggiuwbeigvr203492-8n29o3.DEV0.alpha1.*')>

❯ echo "foobar >= kgjldsfggiuwbeigvr203492-8n29o3.DEV0.alpha1.*" | python -c "import packaging; print(packaging.__version__); from packaging.requirements import Requirement; print(repr(Requirement(input())))"
22.0
Traceback (most recent call last):
  File "/Users/pradyunsg/Developer/github/pradyunsg.github.io/.venv/lib/python3.11/site-packages/packaging/requirements.py", line 35, in __init__
    parsed = parse_requirement(requirement_string)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pradyunsg.github.io/.venv/lib/python3.11/site-packages/packaging/_parser.py", line 64, in parse_requirement
    return _parse_requirement(Tokenizer(source, rules=DEFAULT_RULES))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pradyunsg.github.io/.venv/lib/python3.11/site-packages/packaging/_parser.py", line 82, in _parse_requirement
    url, specifier, marker = _parse_requirement_details(tokenizer)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pradyunsg.github.io/.venv/lib/python3.11/site-packages/packaging/_parser.py", line 116, in _parse_requirement_details
    specifier = _parse_specifier(tokenizer)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pradyunsg.github.io/.venv/lib/python3.11/site-packages/packaging/_parser.py", line 205, in _parse_specifier
    parsed_specifiers = _parse_version_many(tokenizer)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pradyunsg.github.io/.venv/lib/python3.11/site-packages/packaging/_parser.py", line 223, in _parse_version_many
    version_token = tokenizer.expect("VERSION", expected="version after operator")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pradyunsg.github.io/.venv/lib/python3.11/site-packages/packaging/_tokenizer.py", line 136, in expect
    raise self.raise_syntax_error(f"Expected {expected}")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pradyunsg.github.io/.venv/lib/python3.11/site-packages/packaging/_tokenizer.py", line 161, in raise_syntax_error
    raise ParserSyntaxError(
packaging._tokenizer.ParserSyntaxError: Expected version after operator
    foobar >= kgjldsfggiuwbeigvr203492-8n29o3.DEV0.alpha1.*
             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/pradyunsg/Developer/github/pradyunsg.github.io/.venv/lib/python3.11/site-packages/packaging/requirements.py", line 37, in __init__
    raise InvalidRequirement(str(e)) from e
packaging.requirements.InvalidRequirement: Expected version after operator
    foobar >= kgjldsfggiuwbeigvr203492-8n29o3.DEV0.alpha1.*
             ^

We follow `pip`'s lead here and ignore these entirely, treating them as if they don't exist (while also warning the user). Fixes #445. See: pypa/packaging#645 Signed-off-by: William Woodruff <william@trailofbits.com>

* pip_audit, test: handle invalid requires-python specifiers We follow `pip`'s lead here and ignore these entirely, treating them as if they don't exist (while also warning the user). Fixes #445. See: pypa/packaging#645 Signed-off-by: William Woodruff <william@trailofbits.com> * CHANGELOG: record changes Signed-off-by: William Woodruff <william@trailofbits.com> Signed-off-by: William Woodruff <william@trailofbits.com>

pradyunsg · 2023-01-14T13:05:24Z

#662 adds the better error messaging I mentioned in #645 (comment).

pradyunsg · 2023-01-14T13:56:04Z

And, opened https://discuss.python.org/t/should-we-permit-with-more-comparision-operators-in-version-specifiers/22782

pelson · 2023-01-30T13:16:20Z

This has caused much confusion I wonder if we should extend the spec a bit to allow * in other operators. For >=, >, <, and <=, the wildcard suffix can simply be ignored (i.e. >= 1.0.* simply means >= 1.0). ~= is more complicated but I think we can continue to disallow the usage for now since I don’t recall anyone has complained about it yet.

Strictly speaking, I don't think this is true when pre-releases are considered:

>>> from packaging.requirements import Requirement
>>> req1 = Requirement('pkg==1.*')
>>> req2 = Requirement('pkg>=1.0')

>>> s1 = req1.specifier
>>> s2 = req2.specifier

>>> list(s1.filter(['1.0rc0', '1.0'], prereleases=True))
['1.0rc0', '1.0']
>>> list(s2.filter(['1.0rc0', '1.0'], prereleases=True))
['1.0']

I personally have used this wildcard form to say "any version less than the next major one, including that version's release candidates", namely pkg<=3.*. This can't be written in an equivalent way when supporting multiple major versions, as far as I can see, as pkg==1.*,==2.* represents an AND, without any way to specify an OR.

I didn't see this before now, and I am only aware that the specifier was wrong due to the recent setuptools release (issue).

woodruffw · 2023-01-30T16:06:55Z

This can't be written in an equivalent way when supporting multiple major versions, as far as I can see, as pkg==1.,==2. represents an AND, without any way to specify an OR.

The example you gave (pkg<=3.*) should be exactly equivalent to pkg<=3, unless I'm misunderstanding what you meant. That should select any valid version whose major is less than or equal to 3.

pradyunsg · 2023-01-30T17:16:26Z

The example you gave (pkg<=3.*) should be exactly equivalent to pkg<=3, unless I'm misunderstanding what you meant. That should select any valid version whose major is less than or equal to 3.

>>> from packaging.version import Version
>>> from packaging.specifiers import SpecifierSet
>>> Version("3.1") in SpecifierSet("<= 3")
False

No, it's not -- < operators with .* are expected to have a different semantic (since it's not "equal" to 3 but starts with 3).

pradyunsg · 2023-01-30T17:18:27Z

I will note that "less than" with .* has never "worked" correctly:

>>> import packaging
>>> packaging.__version__
'21.3'
>>> from packaging.version import Version
>>> from packaging.specifiers import SpecifierSet
>>> Version("3.1") in SpecifierSet("<= 3.*")
False

pradyunsg · 2023-01-30T17:20:28Z

FWIW, Paul Moore also made this point in https://discuss.python.org/t/should-we-permit-with-more-comparision-operators-in-version-specifiers/22782/10?u=pradyunsg

At this point, instead of having this discussion in two places, I'll request that folks direct any future comments to that thread.

To that end, I'm gonna close this out, to reflect that the discussion on what to do for these cases should happen over there and not here.

The old versioning scheme made it difficult to write useful version requirements, since e.g. `yowasp-yosys>=0.20` excludes pre-releases. Before pypa/packaging#645, `yowasp-yosys>=0.20.*` happened to work by accident, but this is no longer the case and specifying such a requirement in `pyproject.toml` is not possible anyway. All versions using the new scheme are strictly greater (even for old commits), so this should not cause any confusion.

wildcard specifiers such as `cloudpickle>=1.6.*` are invalid since packaging 22.0, see also discussion at pypa/packaging#645

brettcannon added bug packaging.requirements labels Dec 20, 2022

pradyunsg removed the bug label Dec 20, 2022

jack-melchert mentioned this issue Dec 20, 2022

According to PEP 440 this wildcard syntax is no longer allowed phanrahan/mantle#198

Closed

woodruffw mentioned this issue Dec 22, 2022

Fails to parse data-requires-python with a * char pypa/pip-audit#445

Closed

woodruffw mentioned this issue Dec 22, 2022

pip_audit, test: handle invalid requires-python specifiers pypa/pip-audit#447

Merged

pradyunsg changed the title ~~Wildcard in setup.py install_requires list causing parsing error~~ Using .* with operators other than == and != Jan 14, 2023

pelson mentioned this issue Jan 30, 2023

[BUG] "'extras_require' must be a dictionary..." regression pypa/setuptools#3801

Closed

pradyunsg closed this as completed Jan 30, 2023

whitequark mentioned this issue Feb 2, 2023

[BUG] Cannot use .* with >= in version specifiers pypa/setuptools#3807

Closed

manujosephv mentioned this issue Feb 5, 2023

Issues getting environment properly installed on M1 Mac PacktPublishing/Modern-Time-Series-Forecasting-with-Python#4

Open

czyszCTDS mentioned this issue Feb 8, 2023

Python 3.6.x is EOL and no longer supported. warpnet/salt-lint#304

Closed

JeanChristopheMorinPerso mentioned this issue Feb 11, 2023

Rez pip support for: pip_package>=x.y.* (syntax) AcademySoftwareFoundation/rez#1441

Closed

ppoffice mentioned this issue Feb 16, 2023

fix incorrect dependency version requirement uma-pi1/kge#271

Merged

uweschmitt added a commit to uweschmitt/nbmanips that referenced this issue Nov 4, 2023

fix broken dependency specifications

559af99

wildcard specifiers such as `cloudpickle>=1.6.*` are invalid since packaging 22.0, see also discussion at pypa/packaging#645

uweschmitt mentioned this issue Nov 4, 2023

fix broken dependency specifications hmiladhia/nbmanips#16

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using `.*` with operators other than `==` and `!=` #645

Using `.*` with operators other than `==` and `!=` #645

jack-melchert commented Dec 19, 2022

uranusjr commented Dec 20, 2022

pradyunsg commented Dec 20, 2022

pombredanne commented Dec 22, 2022

woodruffw commented Dec 22, 2022

pradyunsg commented Dec 22, 2022

pradyunsg commented Jan 14, 2023

pradyunsg commented Jan 14, 2023

pelson commented Jan 30, 2023

woodruffw commented Jan 30, 2023

pradyunsg commented Jan 30, 2023 •

edited

pradyunsg commented Jan 30, 2023

pradyunsg commented Jan 30, 2023 •

edited

Using .* with operators other than == and != #645

Using .* with operators other than == and != #645

Comments

jack-melchert commented Dec 19, 2022

uranusjr commented Dec 20, 2022

pradyunsg commented Dec 20, 2022

pombredanne commented Dec 22, 2022

woodruffw commented Dec 22, 2022

pradyunsg commented Dec 22, 2022

pradyunsg commented Jan 14, 2023

pradyunsg commented Jan 14, 2023

pelson commented Jan 30, 2023

woodruffw commented Jan 30, 2023

pradyunsg commented Jan 30, 2023 • edited

pradyunsg commented Jan 30, 2023

pradyunsg commented Jan 30, 2023 • edited

Using `.*` with operators other than `==` and `!=` #645

Using `.*` with operators other than `==` and `!=` #645

pradyunsg commented Jan 30, 2023 •

edited

pradyunsg commented Jan 30, 2023 •

edited