Raspi OS 64-bit: pio remote initial dependency installation error on cryptography part

[ramadhanafif]

Configuration

Operating system: Rasberry Pi OS 64-bit version

PlatformIO Version (platformio --version):
PlatformIO Core, version 6.1.4
The CLI version of PIO core.

Description of problem

In pio remote initial start, during cryptography library installation, it invokes some command that uses x64 gcc instead of arm gcc. Perhaps problem on the cryptography library side but manual pip by install seems to not doing anything.

--------------------------  ----------------------------------------
PlatformIO Core             6.1.4
Python                      3.9.2-final.0
System Type                 linux_aarch64
Platform                    Linux-5.15.70-v8+-aarch64-with-glibc2.31
File System Encoding        utf-8
Locale Encoding             UTF-8
PlatformIO Core Directory   /home/pi/.platformio
PlatformIO Core Executable  /home/pi/.local/bin/platformio
Python Executable           /usr/bin/python3
Global Libraries            0
Development Platforms       1
Tools & Toolchains          7
--------------------------  ----------------------------------------

➜  ~ pio remote agent start
Tool Manager: Installing platformio/contrib-pysite @ ~2.39.0
Unpacking  [####################################]  100%          
Tool Manager: contrib-pysite@2.39.201020 has been installed!
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting twisted[tls]==22.1.0
  Using cached Twisted-22.1.0.tar.gz (3.7 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting hyperlink>=17.1.1
  Using cached hyperlink-21.0.0.tar.gz (140 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting Automat>=0.8.0
  Using cached Automat-20.2.0.tar.gz (61 kB)
Collecting constantly>=15.1
  Using cached constantly-15.1.0.tar.gz (21 kB)
Collecting typing-extensions>=3.6.5
  Using cached typing_extensions-4.3.0.tar.gz (47 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting zope.interface>=4.4.2
  Using cached zope.interface-5.4.0.tar.gz (249 kB)
Collecting incremental>=21.3.0
  Using cached incremental-21.3.0.tar.gz (17 kB)
Collecting attrs>=19.2.0
  Using cached attrs-22.1.0.tar.gz (201 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting idna>=2.4
  Using cached idna-3.4.tar.gz (183 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pyopenssl>=16.0.0
  Using cached pyOpenSSL-22.1.0.tar.gz (181 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting service-identity>=18.1.0
  Using cached service-identity-21.1.0.tar.gz (40 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting six
  Using cached six-1.16.0.tar.gz (34 kB)
Collecting cryptography<39,>=38.0.0
  Using cached cryptography-38.0.1.tar.gz (599 kB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/lib/python3/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-n4fb2hg5/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --extra-index-url https://www.piwheels.org/simple -- 'setuptools>=40.6.0,!=60.9.0' wheel 'cffi>=1.12; platform_python_implementation != '"'"'PyPy'"'"'' 'setuptools-rust>=0.11.4'
       cwd: None
  Complete output (124 lines):
  Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
  Collecting setuptools!=60.9.0,>=40.6.0
    Using cached setuptools-65.4.0.tar.gz (2.6 MB)
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting wheel
    Using cached wheel-0.37.1.tar.gz (66 kB)
  Collecting cffi>=1.12
    Using cached cffi-1.15.1.tar.gz (508 kB)
  Collecting setuptools-rust>=0.11.4
    Using cached setuptools-rust-1.5.2.tar.gz (295 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting pycparser
    Using cached pycparser-2.21.tar.gz (170 kB)
  Collecting typing-extensions>=3.7.4.3
    Using cached typing_extensions-4.3.0.tar.gz (47 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting semantic-version<3,>=2.8.2
    Using cached semantic_version-2.10.0.tar.gz (52 kB)
  Skipping wheel build for cffi, due to binaries being disabled for it.
  Skipping wheel build for semantic-version, due to binaries being disabled for it.
  Skipping wheel build for wheel, due to binaries being disabled for it.
  Skipping wheel build for pycparser, due to binaries being disabled for it.
  Building wheels for collected packages: setuptools-rust, typing-extensions, setuptools
    Building wheel for setuptools-rust (PEP 517): started
    Building wheel for setuptools-rust (PEP 517): finished with status 'done'
    Created wheel for setuptools-rust: filename=setuptools_rust-1.5.2-py3-none-any.whl size=23542 sha256=153498fa6923fba5df17487b96b9fc59ca461baa2af35552f103bd4bb452c3c2
    Stored in directory: /home/pi/.cache/pip/wheels/07/f9/89/589dff1ee1d26e3d3afb3ddc390222c05fb10861cb1e31cfd5
    Building wheel for typing-extensions (PEP 517): started
    Building wheel for typing-extensions (PEP 517): finished with status 'done'
    Created wheel for typing-extensions: filename=typing_extensions-4.3.0-py3-none-any.whl size=25596 sha256=804454722564b33d850b74f628871f3a928399e0c71b17fc8c5d073aa34d3753
    Stored in directory: /home/pi/.cache/pip/wheels/2e/8a/02/c858135462f99278e4bcc9bc8b1cca39737b1a7459323d67e4
    Building wheel for setuptools (PEP 517): started
    Building wheel for setuptools (PEP 517): finished with status 'done'
    Created wheel for setuptools: filename=setuptools-65.4.0-py3-none-any.whl size=1232562 sha256=44753eaaa354c9ebc82824fa95ed47ac7049d2a99cd69599dc28b3ba2577217e
    Stored in directory: /home/pi/.cache/pip/wheels/b4/59/12/0bbc5e4ae30c779b53e0001ec9ea354cbb87daac09c7fee06a
  Successfully built setuptools-rust typing-extensions setuptools
  Installing collected packages: typing-extensions, setuptools, semantic-version, pycparser, wheel, setuptools-rust, cffi
      Running setup.py install for semantic-version: started
      Running setup.py install for semantic-version: finished with status 'done'
      Running setup.py install for pycparser: started
      Running setup.py install for pycparser: finished with status 'done'
      Running setup.py install for wheel: started
      Running setup.py install for wheel: finished with status 'done'
      Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      ERROR: Command errored out with exit status 1:
       command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-qitbafkm/cffi_44fb6c57132245cda089e88dd6cf4f98/setup.py'"'"'; __file__='"'"'/tmp/pip-install-qitbafkm/cffi_44fb6c57132245cda089e88dd6cf4f98/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-kkxevqgi/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-n4fb2hg5/overlay --compile --install-headers /tmp/pip-build-env-n4fb2hg5/overlay/include/python3.9/cffi
           cwd: /tmp/pip-install-qitbafkm/cffi_44fb6c57132245cda089e88dd6cf4f98/
      Complete output (56 lines):
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-3.9
      creating build/lib.linux-aarch64-3.9/cffi
      copying cffi/pkgconfig.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/cparser.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/__init__.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/commontypes.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/recompiler.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/verifier.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/model.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/error.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/api.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/lock.py -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/_embedding.h -> build/lib.linux-aarch64-3.9/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-3.9/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-aarch64-3.9
      creating build/temp.linux-aarch64-3.9/c
      aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-PN012d/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-PN012d/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-aarch64-3.9/c/_cffi_backend.o
      c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
         15 | #include <ffi.h>
            |          ^~~~~~~
      compilation terminated.
      error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-qitbafkm/cffi_44fb6c57132245cda089e88dd6cf4f98/setup.py'"'"'; __file__='"'"'/tmp/pip-install-qitbafkm/cffi_44fb6c57132245cda089e88dd6cf4f98/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-kkxevqgi/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-n4fb2hg5/overlay --compile --install-headers /tmp/pip-build-env-n4fb2hg5/overlay/include/python3.9/cffi Check the logs for full command output.
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/6d/0c/5e67831007ba6cd7e52c4095f053cf45c357739b0a7c46a45ddd50049019/cryptography-38.0.1.tar.gz#sha256=1db3d807a14931fa317f96435695d9ec386be7b84b618cc61cfa5d08b0ae33d7 (from https://pypi.org/simple/cryptography/) (requires-python:>=3.6). Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-n4fb2hg5/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple --extra-index-url https://www.piwheels.org/simple -- 'setuptools>=40.6.0,!=60.9.0' wheel 'cffi>=1.12; platform_python_implementation != '"'"'PyPy'"'"'' 'setuptools-rust>=0.11.4' Check the logs for full command output.
  Using cached cryptography-38.0.0.tar.gz (599 kB)

Steps to Reproduce

1. Use raspberry pi OS
2. Install and update pio with pip install -U platformio
3. Login to platformio account
4. pio remote agent start

[ramadhanafif]

This is really strange behavior since I in version 6.1.4, I should be installing cryptography library version 3.3. I know this from this link

def get_contrib_pysite_deps():
    systype = util.get_systype()
    twisted_version = "22.1.0"
    if "linux_arm" in systype:
        result = [
            # twisted[tls], see setup.py for %twisted_version%
            "twisted == %s" % twisted_version,
            # pyopenssl depends on it, use RUST-less version
            "cryptography >= 3.3, < 35.0.0",
            "pyopenssl >= 16.0.0, <= 21.0.0",
            "service_identity >= 18.1.0, <= 21.1.0",
        ]
    else:
        result = ["twisted[tls] == %s" % twisted_version]
    if "windows" in systype:
        result.append("pywin32 != 226")
    return 

[ramadhanafif]

Update: I tried to check the output of get_systype() by running this on python interpreter:

>>> import platform
>>> platform.system().lower()
'linux'
>>> platform.machine().lower()
'aarch64'
>>> 

As a comparison, I tried to do it on my other pi that uses 32-bit raspi os

>>> import platform
>>> platform.system().lower()
'linux'
>>> platform.machine().lower()
'armv6l'
>>> 

In conclusion, get_systype() function does not returns 'linux_armvxx' in raspi os 64-bit, therefore the architecture check that I referenced in my previous comment aren't working as intended. Can anyone suggest any fixes or workarounds? It seems very similar to #4353 .

[ivankravets]

Now dependencies will be built on the target side. Please re-test with pio upgrade --dev. Does it work now?