Set SYSTEM_VERSION_COMPAT=0 during pip install on macos

[laggykiller]

Fixes #1767

[joerick]

I think you're modifying the wrong environment here, it should be the previous invocation labelled 'install the wheel'. Having said that it's probably worth setting the env var for this invocation too.

[laggykiller]

Done, please check!

[joerick]

It would also be good to limit this to CPython 3.8, it's not required to do this on CPython 3.9+ and on 3.6 and 3.7 this is an error that users will also get, so it's worth alerting packagers to.

[laggykiller]

on 3.6 and 3.7 this is an error that users will also get, so it's worth alerting packagers to.

Sorry can you clarify? Do you mean on python 3.6 and 3.7, we also have to set SYSTEM_VERSION_COMPAT=0 as user would also encounter the same problem (e.g. On arm64 runner building x86_64 wheels, triggering the problem when they install wheel with target >=11.0)? Or only set SYSETEM_VERSION_COMPAT=0 on python 3.8?

[henryiii]

Python 3.6 and 3.7 were never ported to ARM on macOS. (3.8 was only partially ported).

[laggykiller]

@henryiii theoretically, can someone crosscompile cp36/cp37 x86_64 wheel on arm64 macos runner, and trigger the bug we are discussing? (Probably nobody would actually do this irl though)

Edit: Looks like cibuildwheel straight out skip cp36/cp37 x86_64 build on arm64, even though it is technically possible by installing x86_64 python on it and run with arch -x86_64. Anyway I am side tracking... (My project has minimum python requirement of 3.8, so it appeared that cibuildwheel skipped cp36/37, please ignore this edit)

[laggykiller]

Done adding the additional check, please review.

[laggykiller]

Turns out the bug can indeed occur if someone crosscompile cp36/37 x86_64 wheels on arm64 macos runner: https://github.com/laggykiller/rlottie-python/actions/runs/8086369591/job/22095975016

I have changed to checking if python version <= 3.8, please review.

[joerick]

It's a complex area, but here's my understanding:

Sorry can you clarify? Do you mean on python 3.6 and 3.7, we also have to set SYSTEM_VERSION_COMPAT=0 as user would also encounter the same problem

Actually this issue isn't to do with arm64 at all, it's to do with the version numbers. macOS was 10.x for many many years, so when they made the transition to 11.0, Apple was concerned that some old software wouldn't bother checking the major version and do checks like if version.minor >= 10: # after 10.10, which would make the new 11.0 look like an old OS to badly-written software.

So they coded the OS with a special rule, if the software was built with an SDK for a previous version of macOS, instead of returning the real version number for the OS, (something like 12.1), they return a fake version, 10.16. 10.16 doesn't exist.

CPython 3.6 and 3.7 were released before this SDK was released, so they are running in this compatibility mode by default (i.e. they default to SYSTEM_VERSION_COMPAT=1). CPython 3.9+ were built with new SDKs, so they default to SYSTEM_VERSION_COMPAT=0. CPython 3.8 straddled the gap. Most distributions were built using an older SDK such that they get the 10.16 version number. A single release of 3.8 was rushed out just at the end of the 3.8 maintenance window to include arm64 support. This build would then have SYSTEM_VERSION_COMPAT=0 because it was built with the newer SDK.

Users of CPython 3.8 can make pip install a package targeting macOS 11+ by updating their Python. Users of 3.6 and 3.7 cannot - they'd have to set SYSTEM_VERSION_COMPAT=0 to do so.

Turns out the bug can indeed occur if someone crosscompile cp36/37 x86_64 wheels on arm64 macos runner

If my reasoning above is correct, the same issue would happen on an x86_64 runner too - the issue is the MACOSX_DEPLOYMENT_TARGET is set too high, so we hit this pip bug.

[laggykiller]

the same issue would happen on an x86_64 runner too

Can confirm about this

[laggykiller]

All agreed and committed, please check!