Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLD: Boost.Math standalone submodule #17432

Merged
merged 53 commits into from Feb 27, 2023
Merged

Conversation

mckib2
Copy link
Contributor

@mckib2 mckib2 commented Nov 16, 2022

@rgommers
Copy link
Member

Thanks @mckib2. Looks like a good idea, I'll have a closer look once CI is happier.

@WarrenWeckesser
Copy link
Member

The failure

___________________ test_moments[nct-arg76-True-True-False] ___________________
[gw0] win32 -- Python 3.10.8 C:\hostedtoolcache\windows\Python\3.10.8\x64\python.exe
C:\hostedtoolcache\windows\Python\3.10.8\x64\lib\site-packages\scipy\stats\tests\test_continuous_basic.py:322: in test_moments
    check_kurt_expect(distfn, arg, m, v, k, distname)
        arg        = (14, 0.24045031331198066)
        distfn     = <scipy.stats._continuous_distns.nct_gen object at 0x000001D4ED6965C0>
        distname   = 'nct'
        higher_ok  = True
        is_xfailing = False
        k          = -2.393967385402602
        m          = 0.2543673273832778
        normalization_ok = True
        s          = 0.06412769986308967
        sup        = <numpy.testing._private.utils.suppress_warnings object at 0x000001D4B7883760>
        v          = 1.1694163414603562
C:\hostedtoolcache\windows\Python\3.10.8\x64\lib\site-packages\scipy\stats\tests\common_tests.py:85: in check_kurt_expect
    npt.assert_allclose(m4e, (k + 3.) * np.power(v, 2), atol=1e-5, rtol=1e-5,
E   AssertionError: 
E   Not equal to tolerance rtol=1e-05, atol=1e-05
E   nct - kurtosis
E   Mismatched elements: 1 / 1 (100%)
E   Max absolute difference: 4.10260374
E   Max relative difference: 4.95022863
E    x: array(4.931374)
E    y: array(0.828771)
        arg        = (14, 0.24045031331198066)
        distfn     = <scipy.stats._continuous_distns.nct_gen object at 0x000001D4ED6965C0>
        k          = -2.393967385402602
        m          = 0.2543673273832778
        m4e        = 4.93137429589607
        msg        = 'nct'
        v          = 1.1694163414603562

appears to be an off-by-3 error in the calculation of the kurtosis of the noncentral t distribution, which is likely related to boostorg/math#800. I haven't dug deep enough to know why this fails in only a few of the jobs.

@mdhaber
Copy link
Contributor

mdhaber commented Dec 4, 2022

@rgommers CI is happier now. Could you take a look?
It would be great to get the Boost updates in 1.10 because they are supposed to fix gh-17368 and at least some of gh-14901. gh-17388 would also be resolved by the latest Boost commits.
I'm going to see whether CI is still happy when I revert 233555e (gh-17272).

Update: it's not. I don't think this fixes any of gh-14901 as-is.

@rgommers rgommers added enhancement A new feature or improvement Build issues Issues with building from source, including different choices of architecture, compilers and OS C/C++ Items related to the internal C/C++ code base and removed 3rd party binaries labels Dec 5, 2022
@rgommers
Copy link
Member

rgommers commented Dec 5, 2022

I'd prefer to merge this right after rather than <24 hrs before creation of a release branch. It looks pretty clean overall; the removal of warning filters does lead to 14 failures on macOS arm64 though, so that has to be looked at before this can go in. Limiting the relevant filters to macOS arm64 only should be fine I'd think.

@mdhaber
Copy link
Contributor

mdhaber commented Dec 5, 2022

Since this is not fixing gh-14901 and gh-17388 in its current state, waiting sounds good.

@mborland
Copy link
Contributor

Boost.Math 1.81.0 Standalone has been published on our page, and should contain your fixes. If there are things missing we can cherry pick them into a 1.81.1.

@tupui tupui added this to the 1.11.0 milestone Dec 15, 2022
@mckib2 mckib2 requested a review from andyfaff as a code owner January 6, 2023 19:05
@mborland
Copy link
Contributor

@mckib2 I think that's a good summary. Again the error should be safe to ignore because it is a floating point environment flag being set, not a true exception. I don't know enough about how cython/pybind11 work, but you could also see if you could clear the flag right after calling the function using std::feclearexcept.

@rgommers
Copy link
Member

These changes that just landed in Xcode 14.3 beta may be relevant:

  • -ffp-exception-behavior=strict is now supported on arm64
  • FENV_ACCESS ON is now supported on arm64

FP strict mode is particularly useful in vectorized code where operations on partially filled vectors can raise FP exceptions. Related, FENV_ACCESS ON is the standard way for developers to notify the compiler that they'll be inspecting FP flags. If that's not enabled, the compiler is free to optimize operations and might produce FP exceptions that aren't present in source code.

@rgommers
Copy link
Member

This code LGTM, ready to go. The one thing I see left to do is figure out how to tell everyone to deal with the moved git submodule. This can be a major pain, when git submodules move and history has zero overlap, weird things happen depending on git version. On Linux with git 2.39.0 I could convince it to work with a few git submodule sync && git submodule update --init tries. On macOS with git 2.37.1 it's still a mess in my fork. It looks like it needs some form of git submodule deinit followed by manual cleaning and then re-adding the submodule (as in the accepted answer of https://stackoverflow.com/questions/60003502/git-how-to-change-url-path-of-a-submodule).

@mdhaber
Copy link
Contributor

mdhaber commented Feb 22, 2023

The one thing I see left to do is figure out how to tell everyone to deal with the moved git submodule. This can be a major pain, when git submodules move and history has zero overlap, weird things happen depending on git version.

Huh, I have not had trouble with this using git version 2.29.2.windows.2. When I initially checked out this branch and tried git submodule update --init, it failed to find the commit. Nicholas and I talked, and we decided that I should try to fetch from boostorg/math manually, and I haven't had a problem since. (I navigated to scipy/_lib/boost, added boostorg/math as a remote, fetched, and then navigated back to /.) Now, whenever I git submodule update --init, it seems to fetch from the remote automatically and check out the commit it's supposed to.

I tested this tonight - the last time I had worked on this branch locally was in early February, and the commit we're at now (9a864d3c520c5fa89043ee7c3edd5ba65fea8cbc) wasn't even available then.

(scipydev) C:\Users\matth\scipy\scipy\_lib\boost>git status
HEAD detached at 3af99e6d5
nothing to commit, working tree clean

(scipydev) C:\Users\matth\scipy\scipy\_lib\boost>git remote -v
origin  https://github.com/mckib2/boost-headers-only (fetch)
origin  https://github.com/mckib2/boost-headers-only (push)
upstream        git@github.com:boostorg/math.git (fetch)
upstream        git@github.com:boostorg/math.git (push)

(scipydev) C:\Users\matth\scipy\scipy\_lib\boost>cd ..

(scipydev) C:\Users\matth\scipy\scipy\_lib>cd ..

(scipydev) C:\Users\matth\scipy\scipy>cd ..

(scipydev) C:\Users\matth\scipy>git status
On branch use-boost-math-standalone
Your branch is behind 'mckib/use-boost-math-standalone' by 159 commits, and can be fast-forwarded.
  (use "git pull" to update your local branch)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   scipy/_lib/boost (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

(scipydev) C:\Users\matth\scipy>git log
commit 25db8d2a7c7691ea16edee7e12998134ef960ec4 (HEAD -> use-boost-math-standalone)
Merge: c435f4d7d5 ac0f78f9a3
Author: Nicholas McKibben <nicholas.bgp@gmail.com>
Date:   Fri Feb 3 19:49:12 2023 -0700

    Merge branch 'use-boost-math-standalone' of github.com:mckib2/scipy into use-boost-math-standalone

commit c435f4d7d5432b5c41975aa250a560067bd35e66
Author: Nicholas McKibben <nicholas.bgp@gmail.com>
Date:   Fri Feb 3 19:49:01 2023 -0700

    MAINT: update to latest boost math develop

commit ac0f78f9a32d14a3c1e7a1224977446dd4eee64a
Author: Matt Haberland <mdhaber@mit.edu>
Date:   Wed Feb 1 09:33:04 2023 -0800

    TST: stats.binom: add test for gh-17146

commit c3562176406f5dc7f9ee47383bb64f61a7f104f5
Author: Matt Haberland <mdhaber@mit.edu>
Date:   Mon Jan 30 10:13:52 2023 -0800

    STY: stats._discrete_distns: remove unused import

commit 6abf5613ac3b5e258628f4647683b863a7396b36
Merge: 80960f8d72 fefb726441
Author: Matt Haberland <mdhaber@mit.edu>
Date:   Mon Jan 30 10:10:56 2023 -0800

(scipydev) C:\Users\matth\scipy>git pull
Enter passphrase for key '/c/Users/matth/.ssh/id_ed25519':
Updating 25db8d2a7c..0db92f1423
Fast-forward
 .git-blame-ignore-revs                             |   10 +-
 .github/ISSUE_TEMPLATE/Bug_report.yml              |    2 +-
 .github/workflows/macos.yml                        |   41 +-
 .github/workflows/macos_meson.yml                  |    2 +-
 .github/workflows/wheels.yml                       |   20 +-
 .gitignore                                         |   13 +-
 .mailmap                                           |    1 +
 HACKING.rst.txt                                    |    2 +-
 README.rst                                         |    6 +-
 azure-pipelines.yml                                |   33 +-
 ci/cirrus_general_ci.yml                           |    4 +-
 ci/cirrus_wheels.yml                               |    4 +-
 dev.py                                             |   13 +-
 doc/API.rst.txt                                    |   45 +-
 doc/source/dev/contributor/pep8.rst                |    2 +-
 doc/source/reference/ccallback.rst                 |   30 -
 doc/source/reference/cluster.rst                   |    6 -
 doc/source/reference/linalg.rst                    |    9 -
 doc/source/reference/main_namespace.rst            |   48 +
 doc/source/reference/optimize.rst                  |    6 -
 doc/source/reference/spatial.rst                   |    5 -
 doc/source/release.0.10.0.rst                      |    1 -
 doc/source/release.0.10.1.rst                      |    1 -
 doc/source/release.0.11.0.rst                      |    1 -
 doc/source/release.0.12.0.rst                      |    1 -
 doc/source/release.0.12.1.rst                      |    1 -
 doc/source/release.0.13.0.rst                      |    1 -
 doc/source/release.0.13.1.rst                      |    1 -
 doc/source/release.0.13.2.rst                      |    1 -
 doc/source/release.0.14.0.rst                      |    1 -
 doc/source/release.0.14.1.rst                      |    1 -
 doc/source/release.0.15.0.rst                      |    1 -
 doc/source/release.0.15.1.rst                      |    1 -
 doc/source/release.0.16.0.rst                      |    1 -
 doc/source/release.0.16.1.rst                      |    1 -
 doc/source/release.0.17.0.rst                      |    1 -
 doc/source/release.0.17.1.rst                      |    1 -
 doc/source/release.0.18.0.rst                      |    1 -
 doc/source/release.0.18.1.rst                      |    1 -
 doc/source/release.0.19.0.rst                      |    1 -
 doc/source/release.0.19.1.rst                      |    1 -
 doc/source/release.0.7.0.rst                       |    1 -
 doc/source/release.0.7.1.rst                       |    1 -
 doc/source/release.0.7.2.rst                       |    1 -
 doc/source/release.0.8.0.rst                       |    1 -
 doc/source/release.0.9.0.rst                       |    1 -
 doc/source/release.1.0.0.rst                       |    1 -
 doc/source/release.1.0.1.rst                       |    1 -
 doc/source/release.1.1.0.rst                       |    1 -
 doc/source/release.1.10.0.rst                      |    1 -
 doc/source/release.1.11.0.rst                      |    1 -
 doc/source/release.1.2.0.rst                       |    1 -
 doc/source/release.1.2.1.rst                       |    1 -
 doc/source/release.1.2.2.rst                       |    1 -
 doc/source/release.1.2.3.rst                       |    1 -
 doc/source/release.1.3.0.rst                       |    1 -
 doc/source/release.1.3.1.rst                       |    1 -
 doc/source/release.1.3.2.rst                       |    1 -
 doc/source/release.1.3.3.rst                       |    1 -
 doc/source/release.1.4.0.rst                       |    1 -
 doc/source/release.1.4.1.rst                       |    1 -
 doc/source/release.1.5.0.rst                       |    1 -
 doc/source/release.1.5.1.rst                       |    1 -
 doc/source/release.1.5.2.rst                       |    1 -
 doc/source/release.1.5.3.rst                       |    1 -
 doc/source/release.1.5.4.rst                       |    1 -
 doc/source/release.1.6.0.rst                       |    1 -
 doc/source/release.1.6.1.rst                       |    1 -
 doc/source/release.1.6.2.rst                       |    1 -
 doc/source/release.1.6.3.rst                       |    1 -
 doc/source/release.1.7.0.rst                       |    1 -
 doc/source/release.1.7.1.rst                       |    1 -
 doc/source/release.1.7.2.rst                       |    1 -
 doc/source/release.1.7.3.rst                       |    1 -
 doc/source/release.1.8.0.rst                       |    1 -
 doc/source/release.1.8.1.rst                       |    1 -
 doc/source/release.1.9.0.rst                       |    1 -
 doc/source/release.1.9.1.rst                       |    1 -
 doc/source/release.1.9.2.rst                       |    1 -
 doc/source/release.1.9.3.rst                       |    1 -
 doc/source/release.rst                             |  120 +-
 doc/{ => source}/release/0.10.0-notes.rst          |    0
 doc/{ => source}/release/0.10.1-notes.rst          |    0
 doc/{ => source}/release/0.11.0-notes.rst          |    0
 doc/{ => source}/release/0.12.0-notes.rst          |    0
 doc/{ => source}/release/0.12.1-notes.rst          |    0
 doc/{ => source}/release/0.13.0-notes.rst          |    0
 doc/{ => source}/release/0.13.1-notes.rst          |    0
 doc/{ => source}/release/0.13.2-notes.rst          |    0
 doc/{ => source}/release/0.14.0-notes.rst          |    0
 doc/{ => source}/release/0.14.1-notes.rst          |    0
 doc/{ => source}/release/0.15.0-notes.rst          |    0
 doc/{ => source}/release/0.15.1-notes.rst          |    0
 doc/{ => source}/release/0.16.0-notes.rst          |    0
 doc/{ => source}/release/0.16.1-notes.rst          |    0
 doc/{ => source}/release/0.17.0-notes.rst          |    0
 doc/{ => source}/release/0.17.1-notes.rst          |    0
 doc/{ => source}/release/0.18.0-notes.rst          |    0
 doc/{ => source}/release/0.18.1-notes.rst          |    0
 doc/{ => source}/release/0.19.0-notes.rst          |    0
 doc/{ => source}/release/0.19.1-notes.rst          |    0
 doc/{ => source}/release/0.7.0-notes.rst           |    0
 doc/{ => source}/release/0.7.1-notes.rst           |    0
 doc/{ => source}/release/0.7.2-notes.rst           |    0
 doc/{ => source}/release/0.8.0-notes.rst           |    0
 doc/{ => source}/release/0.9.0-notes.rst           |    0
 doc/{ => source}/release/1.0.0-notes.rst           |    0
 doc/{ => source}/release/1.0.1-notes.rst           |    0
 doc/{ => source}/release/1.1.0-notes.rst           |    0
 doc/{ => source}/release/1.10.0-notes.rst          |    0
 doc/{ => source}/release/1.11.0-notes.rst          |    0
 doc/{ => source}/release/1.2.0-notes.rst           |    0
 doc/{ => source}/release/1.2.1-notes.rst           |    0
 doc/{ => source}/release/1.2.2-notes.rst           |    0
 doc/{ => source}/release/1.2.3-notes.rst           |    0
 doc/{ => source}/release/1.3.0-notes.rst           |    0
 doc/{ => source}/release/1.3.1-notes.rst           |    0
 doc/{ => source}/release/1.3.2-notes.rst           |    0
 doc/{ => source}/release/1.3.3-notes.rst           |    0
 doc/{ => source}/release/1.4.0-notes.rst           |    0
 doc/{ => source}/release/1.4.1-notes.rst           |    0
 doc/{ => source}/release/1.5.0-notes.rst           |    0
 doc/{ => source}/release/1.5.1-notes.rst           |    0
 doc/{ => source}/release/1.5.2-notes.rst           |    0
 doc/{ => source}/release/1.5.3-notes.rst           |    0
 doc/{ => source}/release/1.5.4-notes.rst           |    0
 doc/{ => source}/release/1.6.0-notes.rst           |    0
 doc/{ => source}/release/1.6.1-notes.rst           |    0
 doc/{ => source}/release/1.6.2-notes.rst           |    0
 doc/{ => source}/release/1.6.3-notes.rst           |    0
 doc/{ => source}/release/1.7.0-notes.rst           |    0
 doc/{ => source}/release/1.7.1-notes.rst           |    0
 doc/{ => source}/release/1.7.2-notes.rst           |    0
 doc/{ => source}/release/1.7.3-notes.rst           |    0
 doc/{ => source}/release/1.8.0-notes.rst           |    0
 doc/{ => source}/release/1.8.1-notes.rst           |    0
 doc/{ => source}/release/1.9.0-notes.rst           |    0
 doc/{ => source}/release/1.9.1-notes.rst           |    0
 doc/{ => source}/release/1.9.2-notes.rst           |    0
 doc/{ => source}/release/1.9.3-notes.rst           |    0
 doc_requirements.txt                               |    1 +
 environment.yml                                    |    8 +-
 meson_options.txt                                  |    2 +-
 pyproject.toml                                     |    6 +-
 scipy/__config__.py.in                             |  147 +++
 scipy/__init__.py                                  |   27 +-
 scipy/_lib/_ccallback.py                           |   20 +
 scipy/_lib/_ccallback_c.pyx                        |    1 -
 scipy/_lib/_testutils.py                           |   29 +-
 scipy/_lib/_util.py                                |    6 +-
 scipy/_lib/boost                                   |    2 +-
 scipy/cluster/__init__.py                          |    8 +-
 scipy/cluster/_hierarchy.pyx                       |    2 +-
 scipy/fft/_pocketfft/pypocketfft.cxx               |    4 +-
 scipy/fftpack/convolve.pyx                         |    2 +-
 scipy/integrate/_quad_vec.py                       |  105 +-
 scipy/integrate/tests/test__quad_vec.py            |   78 +-
 scipy/interpolate/_bspl.pyx                        |    4 +-
 scipy/interpolate/_ppoly.pyx                       |    2 +-
 scipy/interpolate/interpnd.pyx                     |    6 +-
 scipy/io/matlab/_mio5_utils.pyx                    |   93 +-
 scipy/io/matlab/_streams.pyx                       |    2 +-
 scipy/linalg/__init__.py                           |    9 +
 scipy/linalg/meson.build                           |   13 +-
 scipy/linalg/tests/test_solvers.py                 |   13 +-
 scipy/meson.build                                  |   84 +-
 scipy/misc/tests/meson.build                       |    3 +-
 scipy/misc/tests/test_config.py                    |   44 +
 scipy/ndimage/src/_cytest.pyx                      |    2 +-
 scipy/optimize/__init__.py                         |    5 +
 scipy/optimize/_dual_annealing.py                  |    9 +-
 scipy/optimize/_linprog.py                         |   28 +-
 scipy/optimize/_minpack_py.py                      |   24 +-
 scipy/optimize/_optimize.py                        |   54 +-
 scipy/optimize/_shgo.py                            |  747 ++++++------
 scipy/optimize/_shgo_lib/_complex.py               | 1226 ++++++++++++++++++++
 scipy/optimize/_shgo_lib/_vertex.py                |  458 ++++++++
 scipy/optimize/_shgo_lib/meson.build               |    3 +-
 scipy/optimize/_shgo_lib/triangulation.py          |  661 -----------
 scipy/optimize/_zeros_py.py                        |   10 +-
 scipy/optimize/meson.build                         |    2 +-
 scipy/optimize/tests/test__dual_annealing.py       |   17 +
 scipy/optimize/tests/test__shgo.py                 |  447 +++++--
 scipy/optimize/tests/test_minpack.py               |    6 +
 scipy/optimize/tests/test_optimize.py              |   39 +
 scipy/signal/_filter_design.py                     |   11 +-
 scipy/signal/_spectral_py.py                       |    5 +-
 scipy/sparse/__init__.py                           |    6 +
 scipy/sparse/csgraph/_flow.pyx                     |    1 -
 scipy/sparse/csgraph/_shortest_path.pyx            |   12 +-
 scipy/sparse/csgraph/tests/test_shortest_path.py   |   64 +-
 .../linalg/_eigen/lobpcg/tests/test_lobpcg.py      |    9 +-
 scipy/sparse/linalg/_isolve/minres.py              |   20 -
 scipy/sparse/linalg/_isolve/tests/test_lsmr.py     |   45 +-
 scipy/sparse/linalg/_isolve/tests/test_lsqr.py     |   33 -
 scipy/spatial/__init__.py                          |    5 +
 scipy/spatial/_ckdtree.pyx                         |    6 +-
 scipy/spatial/_qhull.pyx                           |   18 +-
 scipy/spatial/tests/test_distance.py               |    2 +-
 scipy/special/_ellip_harm_2.pyx                    |    1 -
 scipy/special/_test_internal.pyx                   |    2 +-
 scipy/special/meson.build                          |    8 +
 scipy/stats/__init__.py                            |    1 +
 scipy/stats/_axis_nan_policy.py                    |   21 +-
 scipy/stats/_constants.py                          |    7 +-
 scipy/stats/_continuous_distns.py                  |  125 +-
 scipy/stats/_covariance.py                         |    1 +
 scipy/stats/_crosstab.py                           |    1 +
 scipy/stats/_discrete_distns.py                    |   20 +-
 scipy/stats/_distn_infrastructure.py               |    3 +-
 scipy/stats/_kde.py                                |    8 +-
 scipy/stats/_ksstats.py                            |    3 +
 scipy/stats/_morestats.py                          |   60 +
 scipy/stats/_mstats_basic.py                       |   15 +-
 scipy/stats/_multivariate.py                       |  412 ++++++-
 scipy/stats/_qmc.py                                |   14 +-
 scipy/stats/_qmvnt.py                              |  533 +++++++++
 scipy/stats/_resampling.py                         |    1 +
 scipy/stats/_stats.pyx                             |    5 +-
 scipy/stats/_stats_mstats_common.py                |    2 +-
 scipy/stats/_stats_py.py                           |   23 +-
 scipy/stats/_stats_pythran.py                      |    2 +
 scipy/stats/_tukeylambda_stats.py                  |    2 +-
 scipy/stats/_unuran/unuran_wrapper.pyx             |   29 -
 scipy/stats/meson.build                            |   13 +-
 scipy/stats/tests/common_tests.py                  |   34 +-
 scipy/stats/tests/data/_mvt.py                     |  172 +++
 scipy/stats/tests/data/meson.build                 |    1 +
 scipy/stats/tests/test_axis_nan_policy.py          |   24 +
 scipy/stats/tests/test_continuous_basic.py         |   36 +-
 scipy/stats/tests/test_discrete_basic.py           |    2 +-
 scipy/stats/tests/test_distributions.py            |  184 ++-
 scipy/stats/tests/test_kdeoth.py                   |    4 +-
 scipy/stats/tests/test_morestats.py                |   23 +-
 scipy/stats/tests/test_mstats_basic.py             |   46 +-
 scipy/stats/tests/test_mstats_extras.py            |   44 +-
 scipy/stats/tests/test_multivariate.py             |  413 ++++++-
 scipy/stats/tests/test_qmc.py                      |   10 +
 scipy/stats/tests/test_rank.py                     |    3 +
 scipy/stats/tests/test_stats.py                    |   35 +-
 scipy/stats/tests/test_tukeylambda_stats.py        |    1 -
 setup.py                                           |    1 +
 tools/config_utils.py                              |   13 -
 tools/lint.py                                      |   14 +-
 tools/openblas_support.py                          |   59 +-
 tools/pre-commit-hook.py                           |   88 ++
 tools/pre-commit-hook.sh                           |   36 -
 tools/wheels/cibw_before_build_macos.sh            |   31 +-
 tools/wheels/cibw_before_build_win.sh              |   16 +-
 249 files changed, 5656 insertions(+), 2123 deletions(-)
 delete mode 100644 doc/source/reference/ccallback.rst
 create mode 100644 doc/source/reference/main_namespace.rst
 delete mode 100644 doc/source/release.0.10.0.rst
 delete mode 100644 doc/source/release.0.10.1.rst
 delete mode 100644 doc/source/release.0.11.0.rst
 delete mode 100644 doc/source/release.0.12.0.rst
 delete mode 100644 doc/source/release.0.12.1.rst
 delete mode 100644 doc/source/release.0.13.0.rst
 delete mode 100644 doc/source/release.0.13.1.rst
 delete mode 100644 doc/source/release.0.13.2.rst
 delete mode 100644 doc/source/release.0.14.0.rst
 delete mode 100644 doc/source/release.0.14.1.rst
 delete mode 100644 doc/source/release.0.15.0.rst
 delete mode 100644 doc/source/release.0.15.1.rst
 delete mode 100644 doc/source/release.0.16.0.rst
 delete mode 100644 doc/source/release.0.16.1.rst
 delete mode 100644 doc/source/release.0.17.0.rst
 delete mode 100644 doc/source/release.0.17.1.rst
 delete mode 100644 doc/source/release.0.18.0.rst
 delete mode 100644 doc/source/release.0.18.1.rst
 delete mode 100644 doc/source/release.0.19.0.rst
 delete mode 100644 doc/source/release.0.19.1.rst
 delete mode 100644 doc/source/release.0.7.0.rst
 delete mode 100644 doc/source/release.0.7.1.rst
 delete mode 100644 doc/source/release.0.7.2.rst
 delete mode 100644 doc/source/release.0.8.0.rst
 delete mode 100644 doc/source/release.0.9.0.rst
 delete mode 100644 doc/source/release.1.0.0.rst
 delete mode 100644 doc/source/release.1.0.1.rst
 delete mode 100644 doc/source/release.1.1.0.rst
 delete mode 100644 doc/source/release.1.10.0.rst
 delete mode 100644 doc/source/release.1.11.0.rst
 delete mode 100644 doc/source/release.1.2.0.rst
 delete mode 100644 doc/source/release.1.2.1.rst
 delete mode 100644 doc/source/release.1.2.2.rst
 delete mode 100644 doc/source/release.1.2.3.rst
 delete mode 100644 doc/source/release.1.3.0.rst
 delete mode 100644 doc/source/release.1.3.1.rst
 delete mode 100644 doc/source/release.1.3.2.rst
 delete mode 100644 doc/source/release.1.3.3.rst
 delete mode 100644 doc/source/release.1.4.0.rst
 delete mode 100644 doc/source/release.1.4.1.rst
 delete mode 100644 doc/source/release.1.5.0.rst
 delete mode 100644 doc/source/release.1.5.1.rst
 delete mode 100644 doc/source/release.1.5.2.rst
 delete mode 100644 doc/source/release.1.5.3.rst
 delete mode 100644 doc/source/release.1.5.4.rst
 delete mode 100644 doc/source/release.1.6.0.rst
 delete mode 100644 doc/source/release.1.6.1.rst
 delete mode 100644 doc/source/release.1.6.2.rst
 delete mode 100644 doc/source/release.1.6.3.rst
 delete mode 100644 doc/source/release.1.7.0.rst
 delete mode 100644 doc/source/release.1.7.1.rst
 delete mode 100644 doc/source/release.1.7.2.rst
 delete mode 100644 doc/source/release.1.7.3.rst
 delete mode 100644 doc/source/release.1.8.0.rst
 delete mode 100644 doc/source/release.1.8.1.rst
 delete mode 100644 doc/source/release.1.9.0.rst
 delete mode 100644 doc/source/release.1.9.1.rst
 delete mode 100644 doc/source/release.1.9.2.rst
 delete mode 100644 doc/source/release.1.9.3.rst
 rename doc/{ => source}/release/0.10.0-notes.rst (100%)
 rename doc/{ => source}/release/0.10.1-notes.rst (100%)
 rename doc/{ => source}/release/0.11.0-notes.rst (100%)
 rename doc/{ => source}/release/0.12.0-notes.rst (100%)
 rename doc/{ => source}/release/0.12.1-notes.rst (100%)
 rename doc/{ => source}/release/0.13.0-notes.rst (100%)
 rename doc/{ => source}/release/0.13.1-notes.rst (100%)
 rename doc/{ => source}/release/0.13.2-notes.rst (100%)
 rename doc/{ => source}/release/0.14.0-notes.rst (100%)
 rename doc/{ => source}/release/0.14.1-notes.rst (100%)
 rename doc/{ => source}/release/0.15.0-notes.rst (100%)
 rename doc/{ => source}/release/0.15.1-notes.rst (100%)
 rename doc/{ => source}/release/0.16.0-notes.rst (100%)
 rename doc/{ => source}/release/0.16.1-notes.rst (100%)
 rename doc/{ => source}/release/0.17.0-notes.rst (100%)
 rename doc/{ => source}/release/0.17.1-notes.rst (100%)
 rename doc/{ => source}/release/0.18.0-notes.rst (100%)
 rename doc/{ => source}/release/0.18.1-notes.rst (100%)
 rename doc/{ => source}/release/0.19.0-notes.rst (100%)
 rename doc/{ => source}/release/0.19.1-notes.rst (100%)
 rename doc/{ => source}/release/0.7.0-notes.rst (100%)
 rename doc/{ => source}/release/0.7.1-notes.rst (100%)
 rename doc/{ => source}/release/0.7.2-notes.rst (100%)
 rename doc/{ => source}/release/0.8.0-notes.rst (100%)
 rename doc/{ => source}/release/0.9.0-notes.rst (100%)
 rename doc/{ => source}/release/1.0.0-notes.rst (100%)
 rename doc/{ => source}/release/1.0.1-notes.rst (100%)
 rename doc/{ => source}/release/1.1.0-notes.rst (100%)
 rename doc/{ => source}/release/1.10.0-notes.rst (100%)
 rename doc/{ => source}/release/1.11.0-notes.rst (100%)
 rename doc/{ => source}/release/1.2.0-notes.rst (100%)
 rename doc/{ => source}/release/1.2.1-notes.rst (100%)
 rename doc/{ => source}/release/1.2.2-notes.rst (100%)
 rename doc/{ => source}/release/1.2.3-notes.rst (100%)
 rename doc/{ => source}/release/1.3.0-notes.rst (100%)
 rename doc/{ => source}/release/1.3.1-notes.rst (100%)
 rename doc/{ => source}/release/1.3.2-notes.rst (100%)
 rename doc/{ => source}/release/1.3.3-notes.rst (100%)
 rename doc/{ => source}/release/1.4.0-notes.rst (100%)
 rename doc/{ => source}/release/1.4.1-notes.rst (100%)
 rename doc/{ => source}/release/1.5.0-notes.rst (100%)
 rename doc/{ => source}/release/1.5.1-notes.rst (100%)
 rename doc/{ => source}/release/1.5.2-notes.rst (100%)
 rename doc/{ => source}/release/1.5.3-notes.rst (100%)
 rename doc/{ => source}/release/1.5.4-notes.rst (100%)
 rename doc/{ => source}/release/1.6.0-notes.rst (100%)
 rename doc/{ => source}/release/1.6.1-notes.rst (100%)
 rename doc/{ => source}/release/1.6.2-notes.rst (100%)
 rename doc/{ => source}/release/1.6.3-notes.rst (100%)
 rename doc/{ => source}/release/1.7.0-notes.rst (100%)
 rename doc/{ => source}/release/1.7.1-notes.rst (100%)
 rename doc/{ => source}/release/1.7.2-notes.rst (100%)
 rename doc/{ => source}/release/1.7.3-notes.rst (100%)
 rename doc/{ => source}/release/1.8.0-notes.rst (100%)
 rename doc/{ => source}/release/1.8.1-notes.rst (100%)
 rename doc/{ => source}/release/1.9.0-notes.rst (100%)
 rename doc/{ => source}/release/1.9.1-notes.rst (100%)
 rename doc/{ => source}/release/1.9.2-notes.rst (100%)
 rename doc/{ => source}/release/1.9.3-notes.rst (100%)
 create mode 100644 scipy/__config__.py.in
 create mode 100644 scipy/misc/tests/test_config.py
 create mode 100644 scipy/optimize/_shgo_lib/_complex.py
 create mode 100644 scipy/optimize/_shgo_lib/_vertex.py
 delete mode 100644 scipy/optimize/_shgo_lib/triangulation.py
 create mode 100644 scipy/stats/_qmvnt.py
 create mode 100644 scipy/stats/tests/data/_mvt.py
 delete mode 100644 tools/config_utils.py
 create mode 100755 tools/pre-commit-hook.py
 delete mode 100755 tools/pre-commit-hook.sh

(scipydev) C:\Users\matth\scipy>git status
On branch use-boost-math-standalone
Your branch is up to date with 'mckib/use-boost-math-standalone'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   scipy/_lib/boost (new commits)

no changes added to commit (use "git add" and/or "git commit -a")

(scipydev) C:\Users\matth\scipy>git submodule update --init
Submodule path 'scipy/_lib/boost': checked out '9a864d3c520c5fa89043ee7c3edd5ba65fea8cbc'

(scipydev) C:\Users\matth\scipy>git status
On branch use-boost-math-standalone
Your branch is up to date with 'mckib/use-boost-math-standalone'.

nothing to commit, working tree clean

(scipydev) C:\Users\matth\scipy>

Does this sound like it might work for others @rgommers?

If so, I suppose the question then is whether we want to create a scipy fork of boost math instead of fetching from boostorg/math directly?

@rgommers
Copy link
Member

Huh, I have not had trouble with this using git version 2.29.2.windows.2. When I initially checked out this branch and tried git submodule update --init, it failed to find the commit. Nicholas and I talked, and we decided that I should try to fetch from boostorg/math manually, and I haven't had a problem since. (I navigated to scipy/_lib/boost, added boostorg/math as a remote, fetched, and then navigated back to /.) Now, whenever I git submodule update --init, it seems to fetch from the remote automatically and check out the commit it's supposed to.

Exactly - it requires talking to Nicholas and then a bunch of commands. Which means that if we merge this, every contributor who pulls the change into main is going to have these problems.

git fetch on a shallow clone does nothing, and since this PR doesn't include the latest commit in the upstream repo, it required git fetch --unshallow for me. In full:

# switch to this PR's branch, then
git submodule sync
cd scipy/_lib/boost
git fetch --unshallow
cd ../../..
git submodule update

This either has to be well-tested so we're sure it works, and then communicated. Or, better, why not add a new submodule and removing the old one instead of overwriting an existing one with a new upstream? I think that if you'd add scipy/_lib/boost_math/ as a new submodule and then removed scipy/_lib/boost/, things would work a lot more smoothly.

If so, I suppose the question then is whether we want to create a scipy fork of boost math instead of fetching from boostorg/math directly?

That wasn't my point and won't make any difference. So I don't think it's needed.

@mdhaber
Copy link
Contributor

mdhaber commented Feb 22, 2023

Adding this as a new submodule instead does sound easier then.

I just mentioned the repo fork because we use forks for other submodules. I wasn't sure whether we do that as a matter of form or only when there is a particular need.
Is it the latter?

@rgommers
Copy link
Member

I just mentioned the repo fork because we use forks for other submodules. I wasn't sure whether we do that as a matter of form or only when there is a particular need.

We did this for a couple of reasons: ensuring that the repo is stable, because a repo delete or history rewrite action would break our checkouts; and the ability to apply patches, which we can't do directly on upstream. However, for Boost.Math I think we can trust it to be stable, and patch-wise it looks like things will go more smoothly to do that directly upstream, since the project is quite well maintained. So no need for a fork here.

@mdhaber
Copy link
Contributor

mdhaber commented Feb 22, 2023

Got it. That was my guess.
Yes, so far it has been very smooth. Thanks @mborland @jzmaddock!

@mckib2 could you make this a new repo so we can see if the submodule update is easier?

@mckib2
Copy link
Contributor Author

mckib2 commented Feb 26, 2023

Removed _lib/boost and created _lib/boost_math and made corresponding changes (guided by a grep for _lib/boost -- fingers crossed I didn't miss anything)

@mdhaber
Copy link
Contributor

mdhaber commented Feb 26, 2023

Prerelease failure seems unrelated; same failure in gh-18059.

I'm removing gh-17916 because yet another case was added, this time for the NCT distribution. I don't want to delay this further, so after this merges, I can submit a PR that points to the latest commit and adds a test for this case.

@mdhaber
Copy link
Contributor

mdhaber commented Feb 26, 2023

@rgommers Nicholas created a new boost_math submodule and removed the old boost one. When I checked out this branch, git submodule update --init was all it took to get the new submodule ready. I think the only thing developers will need to do is remove the old boost folder manually, and I can send an email to the mailing list to let people know about that. Do we need others to test or anything else before we merge?

@steppi
Copy link
Contributor

steppi commented Feb 27, 2023

This builds without issue on my machine after running git submodule update --init and manually deleting the old boost folder.

@tirthasheshpatel
Copy link
Member

tirthasheshpatel commented Feb 27, 2023

Builds for me on Ubuntu 22.04 using Python 3.11 and Meson. Also, setup.py works with the same setup.

I do get a lot of untracked files with the setup.py build though:

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	scipy/io/_test_fortran-f2pywrappers.f
	scipy/linalg/_interpolative-f2pywrappers.f
	scipy/optimize/__nnls/__nnls-f2pywrappers.f
	scipy/optimize/cobyla/_cobyla-f2pywrappers.f
	scipy/optimize/minpack2/_minpack2-f2pywrappers.f
	scipy/optimize/slsqp/_slsqp-f2pywrappers.f
	scipy/sparse/linalg/_isolve/iterative/_iterative-f2pywrappers.f
	scipy/special/_specfun-f2pywrappers.f
	scipy/stats/_statlib-f2pywrappers.f

We can add that to .gitignore though.

@mdhaber
Copy link
Contributor

mdhaber commented Feb 27, 2023

Thanks for testing this @tirthasheshpatel @steppi!

@rgommers when this merges, I could send to the mailing list:

Hi Team,
#17432 just merged, creating a new boost_math submodule and removing the old boost submodule.

After pulling the latest commits from main, please run
git submodule update --init
before building from source.

git status may reveal some new untracked files in the repo; you're welcome to delete those manually.

That should be all that's needed, but if you have trouble, please let us know in gh-17432. Thanks!
Matt

@rgommers
Copy link
Member

That sounds great to me, thanks! Ready to merge it sounds like, go ahead I'd say.

@lorentzenchr
Copy link
Contributor

lorentzenchr commented Mar 17, 2023

As info

git submodule update --init

resulted in

fatal: transport 'file' not allowed
fatal: Fetched in submodule path 'scipy/_lib/boost_math', but it did not contain 7203fa2def6347b0d5f8fe1e8522d5b0a618db9d. Direct fetching of that commit failed.

I ended up reverting to some old commit on main, removing and adding upstream, and then it worked again.

@mdhaber
Copy link
Contributor

mdhaber commented Mar 17, 2023

Huh. Good to know. When you first posted this, you wrote that
git submodule update --force --recursive --init --remote worked. Did it not end up working?

@lorentzenchr
Copy link
Contributor

Huh. Good to know. When you first posted this, you wrote that git submodule update --force --recursive --init --remote worked. Did it not end up working?

That was a false positive, I wrote it too fast and then deleted this part to limit confusion of possible readers. Sorry for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Build issues Issues with building from source, including different choices of architecture, compilers and OS C/C++ Items related to the internal C/C++ code base enhancement A new feature or improvement
Projects
None yet