Run wheel build on PRs #3867

JelleZijlstra · 2023-09-09T14:35:58Z

So we don't just test this workflow on releases. Maybe every PR is too much, but I want it at least on PRs that prepare releases.

cooperlees

Great idea. Maybe on merge to main only we run this ... but either way this is handy to know we work and I'll be stealing it for other projects.

I'll trust you to fix CI before merge. I guess running on PR is handy there ...

JelleZijlstra · 2023-09-09T14:44:20Z

It's actually not that slow, so we might be able to afford it on PRs. However, it's currently failing and I haven't been able to reproduce that failure mode locally, so I'm going to do some debugging-by-CI here.

hauntsaninja · 2023-09-09T15:02:07Z

try the mypyc_attr serialisable thing?

JelleZijlstra · 2023-09-09T15:04:58Z

try the mypyc_attr serialisable thing?

Sorry didn't see this. I tried moving to a plain tuple, will see if that works and try @mypyc_attr next.

This reverts commit bf18575.

This reverts commit 77c312e.

for more information, see https://pre-commit.ci

JelleZijlstra · 2023-09-09T15:20:08Z

serializable=True doesn't help, and my attempt to use non-named tuples broke the unit tests for caching.

JelleZijlstra · 2023-09-09T15:47:10Z

Looking back at the release builds from last night, the FileData thing actually failed on all platforms there too, except for macos-arm64: https://github.com/psf/black/actions/runs/6128665138/job/16635954268. So downgrading to 1.3.0 won't fix our troubles there.

I have tried and failed to reproduce that issue: when I compile a simple file containing a NamedTuple with mypyc locally, the module is set correctly.

JelleZijlstra · 2023-09-09T15:47:38Z

cc @cdce8p (since you contributed the FileData change)

github-actions · 2023-09-09T15:50:25Z

diff-shades reports zero changes comparing this PR (296841c) to main (add161b).

What is this? | Workflow run | diff-shades documentation

cdce8p · 2023-09-09T17:13:38Z

Just saw it a few moments ago. Anything I can do to help? I could try to see if I'm able to fix it.

JelleZijlstra · 2023-09-09T17:20:58Z

Would be great if you could take a look! I have seen a bunch of different failures and I don't have any real idea of how to fix them for now:

The builds during the 23.9.0 release mostly failed with errors like 'importlib._bootstrap.FileData\'>: attribute lookup FileData on importlib._bootstrap failed. This suggests that mypyc sets the __module__ attribute wrong on FileData (a NamedTuple in cache.py which was added in your PR), but I haven't been able to reproduce this in a smaller example.
I tried switching to a plain tuple instead of a NamedTuple (that's the current version of this PR), but some of the tests fail, possibly because the mocking doesn't work properly. (The tests pass without mypyc.)
If I compile locally on macOS, I get errors about a decorator in linegen.py, like AttributeError: 'split_wrapper_dont_increase_indentation_obj' object has no attribute '__dict__'. Did you mean: '__dir__'?. This suggests some bug with nested functions, but it's also hard to reproduce in a smaller example.

Using mypy 1.3.0 (which black was using until yesterday) or 1.5.1 doesn't seem to make a difference.

AlexWaygood · 2023-09-09T18:32:24Z

If I compile locally on macOS, I get errors about a decorator in linegen.py, like AttributeError: 'split_wrapper_dont_increase_indentation_obj' object has no attribute '__dict__'. Did you mean: '__dir__'?. This suggests some bug with nested functions, but it's also hard to reproduce in a smaller example.

I managed to find a minimal repro for this one locally using Python 3.12. Filed this mypy issue:

functools.update_wrapper and functools.wraps cannot be used in mypycified files on py312 python/mypy#16077

cdce8p · 2023-09-09T21:08:25Z

I tried switching to a plain tuple instead of a NamedTuple (that's the current version of this PR), but some of the tests fail, possibly because the mocking doesn't work properly. (The tests pass without mypyc.)

It's somewhat surprising that the NamedTuple could be the issue. It works fine for mypy, that's what I originally used as inspiration. https://github.com/python/mypy/blob/v1.5.1/mypy/fswatcher.py#L10-L13

Will need to take another look at it tomorrow.

cdce8p · 2023-09-10T19:38:49Z

This was interesting 😅 Quite a few surprising issues but I mostly got it working now, except for Python 3.12.
Feel free to cherry-pick my commits from: main...cdce8p:ci-wheel-build

As for the specific issues:

'importlib._bootstrap.FileData\'>: attribute lookup FileData on importlib._bootstrap failed this turned out to be a Python 3.8 only thing. It seems mypyc doesn't like it, when pickle.dump is used with a compiled NamedTuple. It's fine for 3.9 and up though.
mocking doesn't work properly: mypyc doesn't support patching compiled classes, so I had to add @pytest.mark.incompatible_with_mypyc for those tests.
For Python 3.12, I got a few new errors. Some of them I managed to fix already:
- ImportError: cannot import name 'Mode' from 'black' and there like. Those seem to occur if a name is imported from a module that doesn't explicitly reexports it. E.g. Mode is defined in black.mode originally. Not sure those why it works for 3.11.
- One new DeprecationWarning from aiohttp. It's already fixed upstream but not released yet. Added an ignore for that.

Open issues (Python 3.12):

AttributeError: 'split_wrapper_dont_increase_indentation_obj' object has no attribute '__dict__'. Did you mean: '__dir__'? which @AlexWaygood already managed to debug.
functools.update_wrapper and functools.wraps cannot be used in mypycified files on py312 python/mypy#16077
RuntimeError: Please install Black with the 'd' extra: That's a bit surprising as the d extra is already installed. Not quite sure what's up with that.

Although nice, I don't think it's necessary to ship wheels for Python 3.12 just yet. The sdist already works. 23.9.0 would have been to first time those would have been built anyway. Maybe let's skip those for now.

Full cibuildwheel Python 3.12 testing log

Testing wheel...
  
      + pip install virtualenv -c /constraints.txt
  Collecting virtualenv
    Obtaining dependency information for virtualenv from https://files.pythonhosted.org/packages/14/19/e266f07cf55155d5f45170bbe08c486d8a9a9ae17bc8983acb1c019a8dd4/virtualenv-20.24.2-py3-none-any.whl.metadata
    Downloading virtualenv-20.24.2-py3-none-any.whl.metadata (4.5 kB)
  Collecting distlib<1,>=0.3.7 (from virtualenv)
    Obtaining dependency information for distlib<1,>=0.3.7 from https://files.pythonhosted.org/packages/43/a0/9ba967fdbd55293bacfc1507f58e316f740a3b231fc00e3d86dc39bc185a/distlib-0.3.7-py2.py3-none-any.whl.metadata
    Downloading distlib-0.3.7-py2.py3-none-any.whl.metadata (5.1 kB)
  Collecting filelock<4,>=3.12.2 (from virtualenv)
    Obtaining dependency information for filelock<4,>=3.12.2 from https://files.pythonhosted.org/packages/00/45/ec3407adf6f6b5bf867a4462b2b0af27597a26bd3cd6e2534cb6ab029938/filelock-3.12.2-py3-none-any.whl.metadata
    Downloading filelock-3.12.2-py3-none-any.whl.metadata (2.7 kB)
  Collecting platformdirs<4,>=3.9.1 (from virtualenv)
    Obtaining dependency information for platformdirs<4,>=3.9.1 from https://files.pythonhosted.org/packages/14/51/fe5a0d6ea589f0d4a1b97824fb518962ad48b27cd346dcdfa2405187997a/platformdirs-3.10.0-py3-none-any.whl.metadata
    Using cached platformdirs-3.10.0-py3-none-any.whl.metadata (11 kB)
  Downloading virtualenv-20.24.2-py3-none-any.whl (3.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 65.0 MB/s eta 0:00:00
  Downloading distlib-0.3.7-py2.py3-none-any.whl (468 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.9/468.9 kB 48.0 MB/s eta 0:00:00
  Downloading filelock-3.12.2-py3-none-any.whl (10 kB)
  Using cached platformdirs-3.10.0-py3-none-any.whl (17 kB)
  Installing collected packages: distlib, platformdirs, filelock, virtualenv
  Successfully installed distlib-0.3.7 filelock-3.12.2 platformdirs-3.10.0 virtualenv-20.24.2
      + mktemp -d
      + python -m virtualenv --no-download /tmp/tmp.e1YtjY9tWt/venv
  created virtual environment CPython3.12.0.candidate.1-64 in 350ms
    creator CPython3Posix(dest=/tmp/tmp.e1YtjY9tWt/venv, clear=False, no_vcs_ignore=False, global=False)
    seeder FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
      added seed packages: pip==23.2.1
    activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
      + pip install '/tmp/cibuildwheel/repaired_wheel/black-0.1.dev1+ga862d8d-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl[d, jupyter]'
  Processing /tmp/cibuildwheel/repaired_wheel/black-0.1.dev1+ga862d8d-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  Collecting click>=8.0.0 (from black==0.1.dev1+ga862d8d)
    Obtaining dependency information for click>=8.0.0 from https://files.pythonhosted.org/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl.metadata
    Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
  Collecting mypy-extensions>=0.4.3 (from black==0.1.dev1+ga862d8d)
    Using cached mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
  Collecting packaging>=22.0 (from black==0.1.dev1+ga862d8d)
    Using cached packaging-23.1-py3-none-any.whl (48 kB)
  Collecting pathspec>=0.9.0 (from black==0.1.dev1+ga862d8d)
    Obtaining dependency information for pathspec>=0.9.0 from https://files.pythonhosted.org/packages/b4/2a/9b1be29146139ef459188f5e420a66e835dda921208db600b7037093891f/pathspec-0.11.2-py3-none-any.whl.metadata
    Using cached pathspec-0.11.2-py3-none-any.whl.metadata (19 kB)
  Collecting platformdirs>=2 (from black==0.1.dev1+ga862d8d)
    Obtaining dependency information for platformdirs>=2 from https://files.pythonhosted.org/packages/14/51/fe5a0d6ea589f0d4a1b97824fb518962ad48b27cd346dcdfa2405187997a/platformdirs-3.10.0-py3-none-any.whl.metadata
    Using cached platformdirs-3.10.0-py3-none-any.whl.metadata (11 kB)
  Collecting aiohttp>=3.7.4 (from black==0.1.dev1+ga862d8d)
    Downloading aiohttp-3.8.5.tar.gz (7.4 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.4/7.4 MB 46.2 MB/s eta 0:00:00
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting ipython>=7.8.0 (from black==0.1.dev1+ga862d8d)
    Obtaining dependency information for ipython>=7.8.0 from https://files.pythonhosted.org/packages/7f/d0/c3eb7b17b013da59925aed7b2e7c55f8f1c9209249316812fe8cb758b337/ipython-8.15.0-py3-none-any.whl.metadata
    Downloading ipython-8.15.0-py3-none-any.whl.metadata (5.9 kB)
  Collecting tokenize-rt>=3.2.0 (from black==0.1.dev1+ga862d8d)
    Obtaining dependency information for tokenize-rt>=3.2.0 from https://files.pythonhosted.org/packages/8d/35/78f03aa48cfebd13646707f64477bc7eacf1081edcdcd1b4d57cb1b5d0a8/tokenize_rt-5.2.0-py2.py3-none-any.whl.metadata
    Downloading tokenize_rt-5.2.0-py2.py3-none-any.whl.metadata (4.1 kB)
  Collecting attrs>=17.3.0 (from aiohttp>=3.7.4->black==0.1.dev1+ga862d8d)
    Downloading attrs-23.1.0-py3-none-any.whl (61 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 15.0 MB/s eta 0:00:00
  Collecting charset-normalizer<4.0,>=2.0 (from aiohttp>=3.7.4->black==0.1.dev1+ga862d8d)
    Obtaining dependency information for charset-normalizer<4.0,>=2.0 from https://files.pythonhosted.org/packages/bf/a0/188f223c7d8b924fb9b554b9d27e0e7506fd5bf9cfb6dbacb2dfd5832b53/charset_normalizer-3.2.0-py3-none-any.whl.metadata
    Downloading charset_normalizer-3.2.0-py3-none-any.whl.metadata (31 kB)
  Collecting multidict<7.0,>=4.5 (from aiohttp>=3.7.4->black==0.1.dev1+ga862d8d)
    Downloading multidict-6.0.4.tar.gz (51 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.3/51.3 kB 12.5 MB/s eta 0:00:00
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting async-timeout<5.0,>=4.0.0a3 (from aiohttp>=3.7.4->black==0.1.dev1+ga862d8d)
    Obtaining dependency information for async-timeout<5.0,>=4.0.0a3 from https://files.pythonhosted.org/packages/a7/fa/e01228c2938de91d47b307831c62ab9e4001e747789d0b05baf779a6488c/async_timeout-4.0.3-py3-none-any.whl.metadata
    Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)
  Collecting yarl<2.0,>=1.0 (from aiohttp>=3.7.4->black==0.1.dev1+ga862d8d)
    Downloading yarl-1.9.2.tar.gz (184 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 184.7/184.7 kB 36.6 MB/s eta 0:00:00
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting frozenlist>=1.1.1 (from aiohttp>=3.7.4->black==0.1.dev1+ga862d8d)
    Downloading frozenlist-1.4.0.tar.gz (90 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.7/90.7 kB 25.8 MB/s eta 0:00:00
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting aiosignal>=1.1.2 (from aiohttp>=3.7.4->black==0.1.dev1+ga862d8d)
    Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
  Collecting backcall (from ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading backcall-0.2.0-py2.py3-none-any.whl (11 kB)
  Collecting decorator (from ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)
  Collecting jedi>=0.16 (from ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Obtaining dependency information for jedi>=0.16 from https://files.pythonhosted.org/packages/8e/46/7e3ae3aa2dcfcffc5138c6cef5448523218658411c84a2000bf75c8d3ec1/jedi-0.19.0-py2.py3-none-any.whl.metadata
    Downloading jedi-0.19.0-py2.py3-none-any.whl.metadata (22 kB)
  Collecting matplotlib-inline (from ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading matplotlib_inline-0.1.6-py3-none-any.whl (9.4 kB)
  Collecting pickleshare (from ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
  Collecting prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 (from ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Obtaining dependency information for prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 from https://files.pythonhosted.org/packages/a9/b4/ba77c84edf499877317225d7b7bc047a81f7c2eed9628eeb6bab0ac2e6c9/prompt_toolkit-3.0.39-py3-none-any.whl.metadata
    Downloading prompt_toolkit-3.0.39-py3-none-any.whl.metadata (6.4 kB)
  Collecting pygments>=2.4.0 (from ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Obtaining dependency information for pygments>=2.4.0 from https://files.pythonhosted.org/packages/43/88/29adf0b44ba6ac85045e63734ae0997d3c58d8b1a91c914d240828d0d73d/Pygments-2.16.1-py3-none-any.whl.metadata
    Downloading Pygments-2.16.1-py3-none-any.whl.metadata (2.5 kB)
  Collecting stack-data (from ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading stack_data-0.6.2-py3-none-any.whl (24 kB)
  Collecting traitlets>=5 (from ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading traitlets-5.9.0-py3-none-any.whl (117 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.4/117.4 kB 32.1 MB/s eta 0:00:00
  Collecting pexpect>4.3 (from ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.0/59.0 kB 14.3 MB/s eta 0:00:00
  Collecting parso<0.9.0,>=0.8.3 (from jedi>=0.16->ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading parso-0.8.3-py2.py3-none-any.whl (100 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.8/100.8 kB 30.1 MB/s eta 0:00:00
  Collecting ptyprocess>=0.5 (from pexpect>4.3->ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
  Collecting wcwidth (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading wcwidth-0.2.6-py2.py3-none-any.whl (29 kB)
  Collecting idna>=2.0 (from yarl<2.0,>=1.0->aiohttp>=3.7.4->black==0.1.dev1+ga862d8d)
    Downloading idna-3.4-py3-none-any.whl (61 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 11.7 MB/s eta 0:00:00
  Collecting executing>=1.2.0 (from stack-data->ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading executing-1.2.0-py2.py3-none-any.whl (24 kB)
  Collecting asttokens>=2.1.0 (from stack-data->ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Obtaining dependency information for asttokens>=2.1.0 from https://files.pythonhosted.org/packages/4f/25/adda9979586d9606300415c89ad0e4c5b53d72b92d2747a3c634701a6a02/asttokens-2.4.0-py2.py3-none-any.whl.metadata
    Downloading asttokens-2.4.0-py2.py3-none-any.whl.metadata (4.9 kB)
  Collecting pure-eval (from stack-data->ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading pure_eval-0.2.2-py3-none-any.whl (11 kB)
  Collecting six>=1.12.0 (from asttokens>=2.1.0->stack-data->ipython>=7.8.0->black==0.1.dev1+ga862d8d)
    Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
  Downloading click-8.1.7-py3-none-any.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 27.6 MB/s eta 0:00:00
  Downloading ipython-8.15.0-py3-none-any.whl (806 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 806.6/806.6 kB 21.8 MB/s eta 0:00:00
  Using cached pathspec-0.11.2-py3-none-any.whl (29 kB)
  Using cached platformdirs-3.10.0-py3-none-any.whl (17 kB)
  Downloading tokenize_rt-5.2.0-py2.py3-none-any.whl (5.8 kB)
  Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
  Downloading charset_normalizer-3.2.0-py3-none-any.whl (46 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.7/46.7 kB 12.7 MB/s eta 0:00:00
  Downloading jedi-0.19.0-py2.py3-none-any.whl (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 71.7 MB/s eta 0:00:00
  Downloading prompt_toolkit-3.0.39-py3-none-any.whl (385 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 385.2/385.2 kB 66.0 MB/s eta 0:00:00
  Downloading Pygments-2.16.1-py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 76.8 MB/s eta 0:00:00
  Downloading asttokens-2.4.0-py2.py3-none-any.whl (27 kB)
  Building wheels for collected packages: aiohttp, frozenlist, multidict, yarl
    Building wheel for aiohttp (pyproject.toml): started
    Building wheel for aiohttp (pyproject.toml): finished with status 'done'
    Created wheel for aiohttp: filename=aiohttp-3.8.5-py3-none-any.whl size=183808 sha256=b7be60007dc0f52e5a40c1d16bf88d9bfcfa9c4ae6a625145e19ca8947dc0c51
    Stored in directory: /root/.cache/pip/wheels/4e/1f/3c/1fc4887461a8dd9985985cf53a164baa336bbc6e787ebde965
    Building wheel for frozenlist (pyproject.toml): started
    Building wheel for frozenlist (pyproject.toml): finished with status 'done'
    Created wheel for frozenlist: filename=frozenlist-1.4.0-cp312-cp312-linux_x86_64.whl size=112431 sha256=c89a1f88d2adec8063fd721e108e9798e49c788dffa1484bb7aad7819ccce747
    Stored in directory: /root/.cache/pip/wheels/f1/9c/94/9386cb0ea511a93226456388d41d35f1c24ba15a62ffd7b1ef
    Building wheel for multidict (pyproject.toml): started
    Building wheel for multidict (pyproject.toml): finished with status 'done'
    Created wheel for multidict: filename=multidict-6.0.4-cp312-cp312-linux_x86_64.whl size=69166 sha256=601b4ab8773fcce1ed7dd8f898281aa05412c7abfa0ffe0b5c8c4c9298c4cd45
    Stored in directory: /root/.cache/pip/wheels/f6/d8/ff/3c14a64b8f2ab1aa94ba2888f5a988be6ab446ec5c8d1a82da
    Building wheel for yarl (pyproject.toml): started
    Building wheel for yarl (pyproject.toml): finished with status 'done'
    Created wheel for yarl: filename=yarl-1.9.2-cp312-cp312-linux_x86_64.whl size=128640 sha256=92ab658a09a84d258ca86094314377cb64bda00304ca102e5facb089e050d470
    Stored in directory: /root/.cache/pip/wheels/84/e3/6a/7d0fa1abee8e4aa39922b5bd54689b4b5e4269b2821f482a32
  Successfully built aiohttp frozenlist multidict yarl
  Installing collected packages: wcwidth, pure-eval, ptyprocess, pickleshare, executing, backcall, traitlets, tokenize-rt, six, pygments, prompt-toolkit, platformdirs, pexpect, pathspec, parso, packaging, mypy-extensions, multidict, idna, frozenlist, decorator, click, charset-normalizer, attrs, async-timeout, yarl, matplotlib-inline, jedi, black, asttokens, aiosignal, stack-data, aiohttp, ipython
  Successfully installed aiohttp-3.8.5 aiosignal-1.3.1 asttokens-2.4.0 async-timeout-4.0.3 attrs-23.1.0 backcall-0.2.0 black-0.1.dev1+ga862d8d charset-normalizer-3.2.0 click-8.1.7 decorator-5.1.1 executing-1.2.0 frozenlist-1.4.0 idna-3.4 ipython-8.15.0 jedi-0.19.0 matplotlib-inline-0.1.6 multidict-6.0.4 mypy-extensions-1.0.0 packaging-23.1 parso-0.8.3 pathspec-0.11.2 pexpect-4.8.0 pickleshare-0.7.5 platformdirs-3.10.0 prompt-toolkit-3.0.39 ptyprocess-0.7.0 pure-eval-0.2.2 pygments-2.16.1 six-1.16.0 stack-data-0.6.2 tokenize-rt-5.2.0 traitlets-5.9.0 wcwidth-0.2.6 yarl-1.9.2
      + pip install 'pytest>=6.1.1'
  Collecting pytest>=6.1.1
    Obtaining dependency information for pytest>=6.1.1 from https://files.pythonhosted.org/packages/df/d0/e192c4275aecabf74faa1aacd75ef700091913236ec78b1a98f62a2412ee/pytest-7.4.2-py3-none-any.whl.metadata
    Downloading pytest-7.4.2-py3-none-any.whl.metadata (7.9 kB)
  Collecting iniconfig (from pytest>=6.1.1)
    Downloading iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
  Requirement already satisfied: packaging in /tmp/tmp.e1YtjY9tWt/venv/lib/python3.12/site-packages (from pytest>=6.1.1) (23.1)
  Collecting pluggy<2.0,>=0.12 (from pytest>=6.1.1)
    Obtaining dependency information for pluggy<2.0,>=0.12 from https://files.pythonhosted.org/packages/05/b8/42ed91898d4784546c5f06c60506400548db3f7a4b3fb441cba4e5c17952/pluggy-1.3.0-py3-none-any.whl.metadata
    Using cached pluggy-1.3.0-py3-none-any.whl.metadata (4.3 kB)
  Downloading pytest-7.4.2-py3-none-any.whl (324 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 324.5/324.5 kB 26.8 MB/s eta 0:00:00
  Using cached pluggy-1.3.0-py3-none-any.whl (18 kB)
  Installing collected packages: pluggy, iniconfig, pytest
  Successfully installed iniconfig-2.0.0 pluggy-1.3.0 pytest-7.4.2
      + mkdir -p /tmp/tmp.e1YtjY9tWt/test_cwd
      + sh -c 'pytest /project -k "not incompatible_with_mypyc"'
  ============================= test session starts ==============================
  platform linux -- Python 3.12.0rc1, pytest-7.4.2, pluggy-1.3.0
  rootdir: /project
  configfile: pyproject.toml
  collected 198 items / 2 errors
  
  ==================================== ERRORS ====================================
  _____________________ ERROR collecting tests/test_black.py _____________________
  ../venv/lib/python3.12/site-packages/_pytest/runner.py:341: in from_call
      result: Optional[TResult] = func()
  ../venv/lib/python3.12/site-packages/_pytest/runner.py:372: in <lambda>
      call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
  ../venv/lib/python3.12/site-packages/_pytest/python.py:531: in collect
      self._inject_setup_module_fixture()
  ../venv/lib/python3.12/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
      self.obj, ("setUpModule", "setup_module")
  ../venv/lib/python3.12/site-packages/_pytest/python.py:310: in obj
      self._obj = obj = self._getobj()
  ../venv/lib/python3.12/site-packages/_pytest/python.py:528: in _getobj
      return self._importtestmodule()
  ../venv/lib/python3.12/site-packages/_pytest/python.py:617: in _importtestmodule
      mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
  ../venv/lib/python3.12/site-packages/_pytest/pathlib.py:567: in import_path
      importlib.import_module(module_name)
  /opt/python/cp312-cp312/lib/python3.12/importlib/__init__.py:90: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  <frozen importlib._bootstrap>:1293: in _gcd_import
      ???
  <frozen importlib._bootstrap>:1266: in _find_and_load
      ???
  <frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
      ???
  <frozen importlib._bootstrap>:841: in _load_unlocked
      ???
  ../venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
      exec(co, module.__dict__)
  /project/tests/test_black.py:40: in <module>
      import black
  src/black/__init__.py:64: in <module>
      ???
  src/black/linegen.py:1000: in <module>
      ???
  src/black/linegen.py:971: in dont_increase_indentation
      ???
  /opt/python/cp312-cp312/lib/python3.12/functools.py:58: in update_wrapper
      getattr(wrapper, attr).update(getattr(wrapped, attr, {}))
  E   AttributeError: 'split_wrapper_dont_increase_indentation_obj' object has no attribute '__dict__'. Did you mean: '__dir__'?
  ____________________ ERROR collecting tests/test_blackd.py _____________________
  /project/tests/test_blackd.py:14: in <module>
      import blackd
  ../venv/lib/python3.12/site-packages/blackd/__init__.py:24: in <module>
      from black.concurrency import maybe_install_uvloop
  ../venv/lib/python3.12/site-packages/black/concurrency.py:19: in <module>
      from black import WriteBack, format_file_in_place
  E   ImportError: cannot import name 'WriteBack' from 'black' (/tmp/tmp.e1YtjY9tWt/venv/lib/python3.12/site-packages/black/__init__.cpython-312-x86_64-linux-gnu.so). Did you mean: 'traceback'?
  
  The above exception was the direct cause of the following exception:
  ../venv/lib/python3.12/site-packages/_pytest/runner.py:341: in from_call
      result: Optional[TResult] = func()
  ../venv/lib/python3.12/site-packages/_pytest/runner.py:372: in <lambda>
      call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
  ../venv/lib/python3.12/site-packages/_pytest/python.py:531: in collect
      self._inject_setup_module_fixture()
  ../venv/lib/python3.12/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
      self.obj, ("setUpModule", "setup_module")
  ../venv/lib/python3.12/site-packages/_pytest/python.py:310: in obj
      self._obj = obj = self._getobj()
  ../venv/lib/python3.12/site-packages/_pytest/python.py:528: in _getobj
      return self._importtestmodule()
  ../venv/lib/python3.12/site-packages/_pytest/python.py:617: in _importtestmodule
      mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
  ../venv/lib/python3.12/site-packages/_pytest/pathlib.py:567: in import_path
      importlib.import_module(module_name)
  /opt/python/cp312-cp312/lib/python3.12/importlib/__init__.py:90: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  <frozen importlib._bootstrap>:1293: in _gcd_import
      ???
  <frozen importlib._bootstrap>:1266: in _find_and_load
      ???
  <frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
      ???
  <frozen importlib._bootstrap>:841: in _load_unlocked
      ???
  ../venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
      exec(co, module.__dict__)
  /project/tests/test_blackd.py:16: in <module>
      raise RuntimeError("Please install Black with the 'd' extra") from e
  E   RuntimeError: Please install Black with the 'd' extra
  =========================== short test summary info ============================
  ERROR ../../../project/tests/test_black.py - AttributeError: 'split_wrapper_d...
  ERROR ../../../project/tests/test_blackd.py - RuntimeError: Please install Bl...
  !!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!

Edit
Just saw that @hauntsaninja opened #3871 already to add the @pytest.mark.incompatible_with_mypyc decorators 👍🏻

JelleZijlstra · 2023-09-10T22:11:22Z

Thanks, that's really helpful! Good catch on the NamedTuple thing being an issue only in 3.8.

I wonder if it's worth always pickling the raw tuples, as it makes the file size a lot smaller:

In [78]: len(pickle.dumps(FileData(1, "x", 1.0)))
Out[78]: 92

In [79]: len(pickle.dumps((1, "x", 1.0)))
Out[79]: 29

Though it gets amortized a bit if there are many objects:

In [80]: len(pickle.dumps([FileData(i, "x", 1.0) for i in range(100)]))
Out[80]: 2571

In [81]: len(pickle.dumps([(i, "x", 1.0) for i in range(100)]))
Out[81]: 1518

I thought this might be costly because we'd have to convert to FileData at unpickling time and that would be slower, but it's actually faster!

In [82]: p1 = pickle.dumps([FileData(i, "x", 1.0) for i in range(100)])

In [83]: p2 = pickle.dumps([(i, "x", 1.0) for i in range(100)])

In [84]: %timeit pickle.loads(p1)
66.5 µs ± 101 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

In [85]: %timeit [FileData(*t) for t in pickle.loads(p2)]
37.3 µs ± 95.6 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Presumably pickle is faster at unpickling tuples than NamedTuples.

For the Mode thing, I think I'd prefer to explicitly export it from __init__.py, since some users are using it: https://grep.app/search?q=black.Mode&case=true. We don't yet define a public API, but if we did Mode would probably be in it. I'll do that in a separate PR.

cdce8p · 2023-09-10T22:34:52Z

There is no specific need to use FileData with pickle, so I would be fine with using a raw tuple there. Just though it might not be worth the additional complexity.

Although, a normal tuple will certainly make it easier to manually unpickle these files if one should ever want to inspect it.

JelleZijlstra · 2023-09-10T23:48:21Z

Everything this PR does has now been done better in other PRs (#3870, #3877).

hauntsaninja · 2023-09-10T23:50:48Z

Thank you @cdce8p!

Run wheel build on PRs

93d8484

JelleZijlstra added the skip news Pull requests that don't need a changelog entry. label Sep 9, 2023

cooperlees approved these changes Sep 9, 2023

View reviewed changes

try this

88ddb95

JelleZijlstra added 2 commits September 9, 2023 07:49

this could not possibly be a bad idea

7610db8

Deeper hole

501c535

JelleZijlstra added 2 commits September 9, 2023 08:03

Try non-named tuples

77c312e

Living in the past

bf18575

JelleZijlstra and others added 4 commits September 9, 2023 08:10

Revert "Living in the past"

164fcb7

This reverts commit bf18575.

Revert "Try non-named tuples"

2d849fd

This reverts commit 77c312e.

mypyc_attr

6508490

[pre-commit.ci] auto fixes from pre-commit.com hooks

b232a01

for more information, see https://pre-commit.ci

JelleZijlstra added 2 commits September 9, 2023 08:21

older mypy

df25acd

even older

b1674e1

JelleZijlstra added 4 commits September 9, 2023 09:50

try this again

d08f8ed

not untyped in CI

ffe6cea

patchable

2675f77

change order

296841c

AlexWaygood mentioned this pull request Sep 9, 2023

functools.update_wrapper and functools.wraps cannot be used in mypycified files on py312 python/mypy#16077

Closed

JelleZijlstra mentioned this pull request Sep 10, 2023

No mypyc wheels for release 23.9.0 #3865

Closed

JelleZijlstra mentioned this pull request Sep 10, 2023

Ignore aiohttp DeprecationWarning for 3.12 #3876

Merged

This was referenced Sep 10, 2023

Re-export black.Mode #3875

Merged

Pickle raw tuples in FileData cache #3877

Merged

JelleZijlstra closed this Sep 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run wheel build on PRs #3867

Run wheel build on PRs #3867

JelleZijlstra commented Sep 9, 2023

cooperlees left a comment •

edited

JelleZijlstra commented Sep 9, 2023

hauntsaninja commented Sep 9, 2023

JelleZijlstra commented Sep 9, 2023

JelleZijlstra commented Sep 9, 2023

JelleZijlstra commented Sep 9, 2023

JelleZijlstra commented Sep 9, 2023

github-actions bot commented Sep 9, 2023 •

edited

cdce8p commented Sep 9, 2023 •

edited

JelleZijlstra commented Sep 9, 2023

AlexWaygood commented Sep 9, 2023 •

edited

cdce8p commented Sep 9, 2023

cdce8p commented Sep 10, 2023 •

edited

JelleZijlstra commented Sep 10, 2023

cdce8p commented Sep 10, 2023

JelleZijlstra commented Sep 10, 2023

hauntsaninja commented Sep 10, 2023

Run wheel build on PRs #3867

Run wheel build on PRs #3867

Conversation

JelleZijlstra commented Sep 9, 2023

cooperlees left a comment • edited

Choose a reason for hiding this comment

JelleZijlstra commented Sep 9, 2023

hauntsaninja commented Sep 9, 2023

JelleZijlstra commented Sep 9, 2023

JelleZijlstra commented Sep 9, 2023

JelleZijlstra commented Sep 9, 2023

JelleZijlstra commented Sep 9, 2023

github-actions bot commented Sep 9, 2023 • edited

cdce8p commented Sep 9, 2023 • edited

JelleZijlstra commented Sep 9, 2023

AlexWaygood commented Sep 9, 2023 • edited

cdce8p commented Sep 9, 2023

cdce8p commented Sep 10, 2023 • edited

JelleZijlstra commented Sep 10, 2023

cdce8p commented Sep 10, 2023

JelleZijlstra commented Sep 10, 2023

hauntsaninja commented Sep 10, 2023

cooperlees left a comment •

edited

github-actions bot commented Sep 9, 2023 •

edited

cdce8p commented Sep 9, 2023 •

edited

AlexWaygood commented Sep 9, 2023 •

edited

cdce8p commented Sep 10, 2023 •

edited