spack install of astropy fails in extension-helpers

[lgarrison]

When installing astropy in spack, a curious situation comes up:

    File "setup.py", line 68, in <module>
      setup(ext_modules=get_extensions())
    File "/mnt/sw/nix/store/bcdmjrxn9z9zaa04k2gvgijic57vahjw-py-extension-helpers-1.0.0/lib/python3.9/site-packages/extension_helpers/_setup_helpers.py", line 97, in get_extensions
      shutil.copy(os.path.join(src_path, 'compiler.c'),
    File "/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/lib/python3.9/shutil.py", line 427, in copy
      copyfile(src, dst, follow_symlinks=follow_symlinks)
    File "/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/lib/python3.9/shutil.py", line 266, in copyfile
      with open(dst, 'wb') as fdst:
  PermissionError: [Errno 13] Permission denied: './astropy/_compiler.c'
  error: subprocess-exited-with-error

My diagnosis is this: first, extension-helpers is installed and spack decides to make it read-only; no problem. Then, astropy is built from source, calling extension-helpers' get_extensions() function twice: once in pip's "Preparing metadata" phase, and once to actually build it. The first time, extension-helpers/src/_compiler.c is copied to ./astropy/_compiler.c without issue. The second time, ./astropy/_compiler.c already exists and is read-only, because it was copied along with its 0444 permissions the first time. So the copy fails the second time.

I'm not 100% sure where the permissions on _compiler.c are being modified; I'm not even 100% sure it's spack. The build system is pretty complicated. But regardless, I think this issue would arise any time an installation of extension-helpers is made read-only, which seems reasonable, if uncommon.

I worked around this issue by modifying extension-helpers to simply remove the _compiler.c file before attempting to overwrite it. os.remove() doesn't care about read-only (like rm), but shutil.copy() does. Go figure!

This seems like a pretty safe, general change, so I wanted to see if (1) you agree this is a reasonable modification, and (2) you would be interested in a PR. The diff is simply:

diff --git a/extension_helpers/_setup_helpers.py b/extension_helpers/_setup_helpers.py
index 7e766da..8636873 100644
--- a/extension_helpers/_setup_helpers.py
+++ b/extension_helpers/_setup_helpers.py
@@ -94,8 +94,13 @@ def get_extensions(srcdir='.'):
     if len(ext_modules) > 0:
         main_package_dir = min(packages, key=len)
         src_path = os.path.join(os.path.dirname(__file__), 'src')
-        shutil.copy(os.path.join(src_path, 'compiler.c'),
-                    os.path.join(srcdir, main_package_dir, '_compiler.c'))
+        dst_file = os.path.join(srcdir, main_package_dir, '_compiler.c')
+        try:
+            # remove dst_file in case it exists but is read-only
+            os.remove(dst_file)
+        except FileNotFoundError:
+            pass
+        shutil.copy(os.path.join(src_path, 'compiler.c'), dst_file)
         ext = Extension(main_package_dir + '.compiler_version',
                         [os.path.join(main_package_dir, '_compiler.c')])
         ext_modules.append(ext)

More build log details

==> py-astropy: Executing phase: 'install'
==> [2023-02-06-18:19:11.359931] '/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/bin/python3.9' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/mnt/sw/nix/store/p16j2flby89r9m29p2bnd7cagqp780sf-py-astropy-5.1' '--install-option=--use-system-libraries' '--install-option=--use-system-erfa' '--install-option=--use-system-wcslib' '--install-option=--use-system-cfitsio' '--install-option=--use-system-expat' '.'
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Using pip 22.2.2 from /mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip (python 3.9)
Non-user install due to --prefix or --target option
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-ephem-wheel-cache-jjoo0f0a
Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg
Initialized build tracking at /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg
Created build tracker: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg
Entered build tracker: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg
Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-install-tjs3ee_h
Processing /dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src
  Added file:///dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src to build tracker '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg'
  Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  /mnt/sw/nix/store/d9ksrdh7n65r9f6l2yfk9bh9scnxy03m-py-setuptools-62.6.0/lib/python3.9/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
    warnings.warn(msg, warning_class)
  running dist_info
  creating /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info
  writing /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/PKG-INFO
  writing dependency_links to /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/dependency_links.txt
  writing entry points to /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/entry_points.txt
  writing requirements to /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/requires.txt
  writing top-level names to /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/top_level.txt
  writing manifest file '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/SOURCES.txt'
  listing git files failed - pretending there aren't any
  reading manifest file '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching '*.templ' under directory 'astropy'
  warning: no files found matching 'astropy/astropy.cfg'
  warning: no files found matching '*' under directory 'scripts'
  warning: no files found matching '*' under directory 'static'
  no previously-included directories found matching 'astropy/_dev'
  no previously-included directories found matching 'docs/_build'
  no previously-included directories found matching 'build'
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '*.o' found anywhere in distribution
  adding license file 'LICENSE.rst'
  writing manifest file '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy.egg-info/SOURCES.txt'
  creating '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-modern-metadata-xuibowpl/astropy-5.1.dist-info'
  Preparing metadata (pyproject.toml): finished with status 'done'
  Source in /dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src has version 5.1, which satisfies requirement astropy==5.1 from file:///dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src
  Removed astropy==5.1 from file:///dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src from build tracker '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg'
Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-unpack-h8qbi0x4
Building wheels for collected packages: astropy
  Created temporary directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-wheel-y1fm63d2
  Destination directory: /dev/shm/nix-build-py-astropy-5.1.drv-1/pip-wheel-y1fm63d2
  Building wheel for astropy (pyproject.toml): started
  Running command Building wheel for astropy (pyproject.toml)
  Traceback (most recent call last):
    File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/mnt/sw/nix/store/d9ksrdh7n65r9f6l2yfk9bh9scnxy03m-py-setuptools-62.6.0/lib/python3.9/site-packages/setuptools/build_meta.py", line 244, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/mnt/sw/nix/store/d9ksrdh7n65r9f6l2yfk9bh9scnxy03m-py-setuptools-62.6.0/lib/python3.9/site-packages/setuptools/build_meta.py", line 229, in _build_with_temp_dir
      self.run_setup()
    File "/mnt/sw/nix/store/d9ksrdh7n65r9f6l2yfk9bh9scnxy03m-py-setuptools-62.6.0/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 68, in <module>
      setup(ext_modules=get_extensions())
    File "/mnt/sw/nix/store/bcdmjrxn9z9zaa04k2gvgijic57vahjw-py-extension-helpers-1.0.0/lib/python3.9/site-packages/extension_helpers/_setup_helpers.py", line 97, in get_extensions
      shutil.copy(os.path.join(src_path, 'compiler.c'),
    File "/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/lib/python3.9/shutil.py", line 427, in copy
      copyfile(src, dst, follow_symlinks=follow_symlinks)
    File "/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/lib/python3.9/shutil.py", line 266, in copyfile
      with open(dst, 'wb') as fdst:
  PermissionError: [Errno 13] Permission denied: './astropy/_compiler.c'
  error: subprocess-exited-with-error

  × Building wheel for astropy (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/bin/python3.9 /mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /dev/shm/nix-build-py-astropy-5.1.drv-1/tmpudc3nck7
  cwd: /dev/shm/nix-build-py-astropy-5.1.drv-1/nixbld1/spack-stage-py-astropy-5.1-p16j2flby89r9m29p2bnd7cagqp780sf/spack-src
  Building wheel for astropy (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for astropy
Failed to build astropy
ERROR: Could not build wheels for astropy, which is required to install pyproject.toml-based projects
Exception information:
Traceback (most recent call last):
  File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/mnt/sw/nix/store/hmyviwx9ydli42d7al14dl0a644jr17f-py-pip-22.2.2/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 431, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for astropy, which is required to install pyproject.toml-based projects
Removed build tracker: '/dev/shm/nix-build-py-astropy-5.1.drv-1/pip-build-tracker-gnsrljxg'
Traceback (most recent call last):
  File "/mnt/sw/nix/store/x791w7gixpcfhqs0bdhgq9q33w8z6lvv-builder.py", line 75, in <module>
    spack.installer.build_process(pkg, opts)
  File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/installer.py", line 2077, in build_process
    return installer.run()
  File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/installer.py", line 1941, in run
    self._real_install()
  File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/installer.py", line 2037, in _real_install
    phase_fn.execute()
  File "/mnt/sw/nix/store/x791w7gixpcfhqs0bdhgq9q33w8z6lvv-builder.py", line 61, in wrapPhase
    return f(*args)
  File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/builder.py", line 430, in execute
    self.phase_fn(pkg, pkg.spec, pkg.prefix)
  File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/builder.py", line 306, in _adapter
    return phase_fn(spec, prefix)
  File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/builder.py", line 150, in __forward
    return self.__getattr__(fn_name)(*args, **kwargs)
  File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/builder.py", line 72, in __call__
    return self.phase_fn(self.builder.pkg, spec, prefix)
  File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/build_systems/python.py", line 463, in install
    pip(*args)
  File "/mnt/sw/nix/store/rz0ypy576hbkx883b12fj9v1sd4nfxs9-spack/lib/spack/spack/util/executable.py", line 238, in __call__
    raise ProcessError("Command exited with status %d:" % proc.returncode, long_msg)
spack.util.executable.ProcessError: Command exited with status 1:
    '/mnt/sw/nix/store/y80v9c1vpcqjlvxrkfxw5hyvrhjiixjr-python-3.9.15/bin/python3.9' '-m' 'pip' '-vvv' '--no-input' '--no-cache-dir' '--disable-pip-version-check' 'install' '--no-deps' '--ignore-installed' '--no-build-isolation' '--no-warn-script-location' '--no-index' '--prefix=/mnt/sw/nix/store/p16j2flby89r9m29p2bnd7cagqp780sf-py-astropy-5.1' '--install-option=--use-system-libraries' '--install-option=--use-system-erfa' '--install-option=--use-system-wcslib' '--install-option=--use-system-cfitsio' '--install-option=--use-system-expat' '.'
builder for '/mnt/sw/nix/store/hzdw358lnnl132lvpcsamgiwsfxzil3s-py-astropy-5.1.drv' failed with exit code 1
error: build of '/mnt/sw/nix/store/hzdw358lnnl132lvpcsamgiwsfxzil3s-py-astropy-5.1.drv' failed

Thanks!

[astrofrog]

Thanks for the report! Another possible fix that would avoid having to call os.remove would be to check if the destination file is the same as the file being copied and to skip the copy if so?

[lgarrison]

Re-reading my own bug report, I'm now wondering if an even easier fix is to do the copy in a way that doesn't preserve the read-only permissions, which could be accomplished by using shutil.copyfile() instead of shutil.copy(). What would you think about that?