Build Windows wheels using cibuildwheel #7580

nulano · 2023-11-27T18:51:38Z

removed wheel building on every push to simplify test-windows.yml so libimagequant can be included there,
moved fribidi artifact to the wheels workflow so that both x86 and x64 versions are provided,
moved CIBW_CONFIG_SETTINGS to pyproject.toml and added build-verbosity = 1 so that build warnings are visible,
moved modules/codecs/features check from wheels-test.sh to Tests\check_wheel.py so that it can also be used on Windows,
added a Windows build to wheels.yml, using winbuild\build_prepare.py to build dependencies same as before, building with cibuildwheel, and testing in Docker similarly to Check available features in Windows wheels #6847.

I've included x86 also for #7443 (comment); there have been several issues opened about installing on win32. Perhaps it is not yet the time to stop providing these.

I've tested the arm64 wheels on an M2 MacBook in a Windows ARM VM using this batch script:

rem params: path\to\python.exe path\to\wheel.whl venv_dir_name

%1 -m pip install virtualenv
%1 -m virtualenv %3
cmd /c %3\Scripts\activate.bat ^&^& python -m pip install %2[tests]
powershell .github\workflows\wheels-test.ps1 %3 && echo Success!

Although the VM did have VC++ redistributable DLLs installed, I checked the list of imported dlls by the pyd files match between arm64 and amd64 with dumpbin.

nulano · 2023-11-27T18:55:06Z

Tests/helper.py

+    if shutil.which("djpeg"):
+        try:
+            subprocess.check_call(["djpeg", "-version"])
+            return True
+        except subprocess.CalledProcessError:
+            return False


djpeg.exe and cjpeg.exe require VC++ redistributable packages, but the whole point of testing in Docker is to not install those. Running these during CPython tests therefore crashes with a message about missing DLLs. PyPy tests do require VC++, so they are still tested there.

nulano · 2023-11-27T19:00:50Z

.github/workflows/wheels.yml

+            docker run --rm
+            -v {project}:C:\pillow
+            -v C:\cibw:C:\cibw
+            -v %CD%\..\venv-test:%CD%\..\venv-test
+            -e CI -e GITHUB_ACTIONS
+            mcr.microsoft.com/windows/servercore:ltsc2022
+            powershell C:\pillow\.github\workflows\wheels-test.ps1 %CD%\..\venv-test


cibuildwheel downloads Python to CIBW_CACHE_PATH (set to C:\cibw above) and sets up a venv for testing at %temp%\cibw-run-<random>\<python-tag>\venv-test. It then runs the test command in the directory %temp%\cibw-run-<random>\<python-tag>\test_cwd.

The CIBW_CACHE_PATH and venv-test directories are bind-mounted here so that the venv can be used from the Docker container. The only way I could find to refer to the venv path was %CD%\..\venv-test; it might be worth opening an issue in cibuildwheel for a better solution?

Could try it!

hugovk

Thanks for this! A quick review.

Tests/test_imagegrab.py

.github/workflows/wheels.yml

Tests/test_imagegrab.py

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

radarhere · 2023-11-29T23:45:05Z

Tests/check_wheel.py

+    expected_modules = {"pil", "tkinter", "freetype2", "littlecms2", "webp"}
+
+    # tkinter is not available in cibuildwheel installed CPython on Windows
+    if not importlib.util.find_spec("tkinter"):


I expect there's a reason, but why is it not simpler to test for tkinter by attempting to import it normally?

Lint error, https://github.com/nulano/Pillow/actions/runs/6998411214/job/19036420674#step:7:21:

Tests/check_wheel.py:10:16: F401 `tkinter` imported but unused; consider using `importlib.util.find_spec` to test for availability

Ah, assert tkinter should silence the warning. We've done that before.

Yes, I remember seeing that before, but figured the linter suggestion would work as well.

I've pushed a commit to try: import tkinter instead for consistency with other optional imports (the only one I could find with a simple grep was in Image.preinit).

Yeah, the others were replaced with pytest.importorskip (#4436) which doesn't fit here.

radarhere · 2023-11-30T23:14:57Z

.github/workflows/wheels.yml

+
+      - name: Upload fribidi.dll
+        if: "matrix.arch != 'ARM64'"
+        uses: actions/upload-artifact@v3


Previously, this was only run for one Python version.

Pillow/.github/workflows/test-windows.yml

Lines 246 to 248 in 63bec07

- name: Upload fribidi.dll

if: "github.event_name != 'pull_request' && matrix.python-version == 3.11"

uses: actions/upload-artifact@v3

Any particular reason it's now being run for every version?

This is now in the wheels.yml workflow which only runs a single job for all Python versions of a single architecture.
Previously it was in test-windows.yml which has a separate job for each Python version.

nulano commented Nov 27, 2023

View reviewed changes

This was referenced Nov 27, 2023

win32 wheels missing on PyPI #7443

Closed

Automate build and release to PyPI #7390

Closed

Check available features in Windows wheels #6847

Closed

radarhere added Windows Build labels Nov 27, 2023

hugovk reviewed Nov 27, 2023

View reviewed changes

Tests/test_imagegrab.py Outdated Show resolved Hide resolved

.github/workflows/wheels.yml Outdated Show resolved Hide resolved

.github/workflows/wheels.yml Outdated Show resolved Hide resolved

.github/workflows/wheels.yml Outdated Show resolved Hide resolved

hugovk mentioned this pull request Nov 27, 2023

Install cibuildwheel from requirements file #7581

Merged

hugovk reviewed Nov 27, 2023

View reviewed changes

Tests/test_imagegrab.py Outdated Show resolved Hide resolved

nulano force-pushed the cibuildwheel-docker branch 3 times, most recently from c10446f to f55f7ce Compare November 28, 2023 13:19

nulano and others added 7 commits November 28, 2023 14:20

use cibuildwheel on windows

4f939a1

enable heap verification

bf51d71

specify build config settings in pyproject.toml

3d49244

test cibuildwheel wheels in Docker on Windows

1fdb066

do not build Windows wheels on every push

7dc3a8f

use verbose flag when building wheels

d88ab8a

Apply suggestions from code review

6fe42bd

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

nulano force-pushed the cibuildwheel-docker branch from f55f7ce to 6fe42bd Compare November 28, 2023 13:21

hugovk approved these changes Nov 29, 2023

View reviewed changes

radarhere reviewed Nov 29, 2023

View reviewed changes

replace importlib.util.find_spec with try import except ImportError

e105976

radarhere reviewed Nov 30, 2023

View reviewed changes

radarhere merged commit 316f397 into python-pillow:main Nov 30, 2023
65 checks passed

This was referenced Nov 30, 2023

Add support for Python 3.12 #6941

Closed

10.0.0 refused to build on Windows with 32-bit Python #7251

Closed

Update Windows wheels info #7590

Merged

nulano deleted the cibuildwheel-docker branch December 21, 2023 00:42

hugovk mentioned this pull request Jan 4, 2024

Skip non-wheel CI runs for tags: Windows #7692

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build Windows wheels using cibuildwheel #7580

Build Windows wheels using cibuildwheel #7580

nulano commented Nov 27, 2023 •

edited

nulano Nov 27, 2023 •

edited

nulano Nov 27, 2023 •

edited

hugovk Nov 29, 2023

hugovk left a comment

radarhere Nov 29, 2023

nulano Nov 29, 2023 •

edited

hugovk Nov 30, 2023

nulano Nov 30, 2023

hugovk Dec 1, 2023

radarhere Nov 30, 2023

nulano Nov 30, 2023

	- name: Upload fribidi.dll
	if: "github.event_name != 'pull_request' && matrix.python-version == 3.11"
	uses: actions/upload-artifact@v3

Build Windows wheels using cibuildwheel #7580

Build Windows wheels using cibuildwheel #7580

Conversation

nulano commented Nov 27, 2023 • edited

nulano Nov 27, 2023 • edited

Choose a reason for hiding this comment

nulano Nov 27, 2023 • edited

Choose a reason for hiding this comment

hugovk Nov 29, 2023

Choose a reason for hiding this comment

hugovk left a comment

Choose a reason for hiding this comment

radarhere Nov 29, 2023

Choose a reason for hiding this comment

nulano Nov 29, 2023 • edited

Choose a reason for hiding this comment

hugovk Nov 30, 2023

Choose a reason for hiding this comment

nulano Nov 30, 2023

Choose a reason for hiding this comment

hugovk Dec 1, 2023

Choose a reason for hiding this comment

radarhere Nov 30, 2023

Choose a reason for hiding this comment

nulano Nov 30, 2023

Choose a reason for hiding this comment

nulano commented Nov 27, 2023 •

edited

nulano Nov 27, 2023 •

edited

nulano Nov 27, 2023 •

edited

nulano Nov 29, 2023 •

edited