BUG: Python version constraints in releases causes issues for less strict downstream versions

[cpcloud]

Describe your issue.

It looks like starting the 1.10.x series, the Python version upper bound is now set (<3.12 in 1.10.x, <3.13 in 1.11.x).

This causes problems when depending (including transitively) on scipy in a Poetry project whose python constraint is less strict.

I understand the desire to do this.

A negative consequence of this choice along with some other transitive dependencies' constraints (statsmodels not allowing scipy 1.9.2) results in poetry choosing scipy 1.6.1 for python >=3.10, and since there are no wheels a full build of scipy is attempted.

I can't install a newer version, because my project's (https://github.com/ibis-project/ibis) version constraints are wider than allowed by scipy.

Reproducing Code Example

Create a minimal poetry project with python = "^3.9" and then run

$ poetry add scipy@1.10

Error message

❯ poetry add scipy@1.10

Updating dependencies
Resolving dependencies... (0.0s)

The current project's Python requirement (>=3.9,<4.0) is not compatible with some of the required packages Python requirement:
  - scipy requires Python <3.12,>=3.8, so it will not be satisfied for Python >=3.12,<4.0

Because ibis-framework depends on scipy (1.10.0) which requires Python <3.12,>=3.8, version solving failed.

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties

    For scipy, a possible solution would be to set the `python` property to ">=3.9,<3.12"

    https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
    https://python-poetry.org/docs/dependency-specification/#using-environment-markers

SciPy/NumPy/Python version and system information

❯ python -c 'import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()'
1.9.3 1.25.2 sys.version_info(major=3, minor=10, micro=13, releaselevel='final', serial=0)
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['/usr/local/lib']
Supported SIMD extensions in this NumPy install:
    baseline = SSE,SSE2,SSE3
    found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2
    not found = AVX512F,AVX512CD,AVX512_KNL,AVX512_KNM,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL

[tylerjereddy]

We had another issue with poetry reported recently as well, though it was a different one: #19437

[cpcloud]

Yep! I saw that one while hunting to see if anyone reported something similar :)

I think these two issues are not related except for that they are both encountered with Poetry.

[andyfaff]

Exhaustively discussed in numpy/numpy#24810 and #17957 (I think)

[andyfaff]

The solution in numpy/numpy#24962 was to remove the upper pin on the release branch.

[tylerjereddy]

Alright, I'll probably unpin in 1.11.4 then (cross-listed PR). Not a huge fan of it, but if NumPy is doing it to avoid the complexity of the situation in 1.26.2 I suppose we may as well do that too.

[tylerjereddy]

This should be handled when I get 1.11.4 out, the change is already in the maintenance branch.

[cpcloud]

Thank you!