Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No such file or directory: '/github/home/.cache/pre-commit/repo4mrvfeou/rbenv-system/.install_state_v1staging' #2799

Closed
lmilbaum opened this issue Mar 6, 2023 · 33 comments · Fixed by #2905

Comments

@lmilbaum
Copy link

lmilbaum commented Mar 6, 2023

search you tried in the issue tracker

Found this one #1658

describe your issue

Running pre-commit GitHub action in a custom container.

Workflow is broken:

[INFO] Installing environment for https://github.com/markdownlint/markdownlint.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: FileNotFoundError: [Errno 2] No such file or directory: '/github/home/.cache/pre-commit/repo4mrvfeou/rbenv-system/.install_state_v1staging'

pre-commit --version

pre-commit 2.20.0

.pre-commit-config.yaml

---
ci:
  skip: [terraform_fmt, terraform_validate, terragrunt_validate]

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: end-of-file-fixer
      - id: trailing-whitespace
        args:
          - --markdown-linebreak-ext=md
      - id: check-docstring-first
      - id: requirements-txt-fixer
      - id: check-merge-conflict
      - id: no-commit-to-branch
        args:
          - "--branch"
          - "main"
      - id: check-symlinks
      - id: detect-private-key
      - id: detect-aws-credentials
        args:
          - --allow-missing-credentials
      - id: check-json
  - repo: https://github.com/markdownlint/markdownlint
    rev: v0.12.0
    hooks:
      - id: markdownlint
        additional_dependencies: [rake]
  - repo: https://github.com/maxbrunet/pre-commit-renovate
    rev: 34.157.1
    hooks:
      - id: renovate-config-validator
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.77.1
    hooks:
      - id: terraform_providers_lock
        args:
          - --tf-init-args=-upgrade
      - id: terraform_fmt
      - id: terraform_validate
      - id: terragrunt_validate

~/.cache/pre-commit/pre-commit.log (if present)

No response

@asottile
Copy link
Member

asottile commented Mar 6, 2023

you found an exact duplicate, why did you make an issue?

@asottile asottile closed this as completed Mar 6, 2023
@lmilbaum
Copy link
Author

lmilbaum commented Mar 6, 2023

Because I didn't understand how to fix my problem with the information provided on the other closed issue

@asottile
Copy link
Member

asottile commented Mar 6, 2023

you've not shown what you've tried so I don't know how to help you. right now it looks like you've tried nothing

@lmilbaum
Copy link
Author

lmilbaum commented Mar 6, 2023

I've tried to check rbenv is installed in the container. It is not.
I've checked the ruby version. It is 3.0.4p208 (2022-04-12 revision 3fa771dded) [x86_64-linux].
What else can I try?

@asottile
Copy link
Member

asottile commented Mar 6, 2023

clearly that container is broken -- why do you need that container? also looks like your CI is running both pre-commit.ci and github actions pre-commit -- which is kind of wasteful

@lmilbaum
Copy link
Author

lmilbaum commented Mar 6, 2023

clearly that container is broken -- why do you need that container? also looks like your CI is running both pre-commit.ci and github actions pre-commit -- which is kind of wasteful

Agree. It is broken but I don't have enough ruby expriance to figure out what is broken and fix it. The container is built by me and it used for baking all my team tooling in a container image instead of installing/upgrading them on our laptops.

@asottile
Copy link
Member

asottile commented Mar 6, 2023

if your company wants paid support I'm happy to provide it -- but this seems like you've already got several solutions which don't involve that container

@lmilbaum
Copy link
Author

lmilbaum commented Mar 7, 2023

I am pocking the pre-commit code. It seems like the language ruby ENVIRONMENT_DIR is set to rbenv while the container image doesn't have it. The error is persistent even if rbenv is installed.

@lmilbaum
Copy link
Author

lmilbaum commented Mar 7, 2023

In repository.py line 108, it tries to write a file in a directory which doesn't exist. Could you provide me a hint where it is created?

@pbrkr
Copy link

pbrkr commented Apr 16, 2023

@asottile I'm seeing the same issue described here, and do not have rbenv installed so #1658 doesn't give a solution.

I've looked in ~/.cache/pre-commit/pre-commit.log but I don't see any useful information.

Looking at the code:

def install_environment(

This seems to be responsible for creating the environment directory. If the version is system, we don't run rbenv init -. So are we expecting gem build ... to create the rbenv-system directory?

If the output of gem build ... contains some useful information on what's going wrong, it's being dropped:

def cmd_output_b(

This collects the stdout & stderr of the called process and returns them.

def setup_cmd(prefix: Prefix, cmd: tuple[str, ...], **kwargs: Any) -> None:

This calls `cmd_output_b() but ignores the return values. Perhaps more verbose output is needed in the case of an error.

Contents of ~/.cache/pre-commit/pre-commit.log

version information

pre-commit version: 3.2.2
git --version: git version 2.39.2
sys.version:
    3.10.10 (main, Mar 26 2023, 19:47:56) [GCC 12.2.1 20230121]
sys.executable: /home/pbarker/Projects/pbarker.dev/venv/bin/python
os.name: posix
sys.platform: linux

error information

An unexpected error has occurred: FileNotFoundError: [Errno 2] No such file or directory: '/home/pbarker/.cache/pre-commit/repoiuzm_cni/rbenv-system/.install_state_v1staging'
Traceback (most recent call last):
  File "/home/pbarker/Projects/pbarker.dev/venv/lib/python3.10/site-packages/pre_commit/error_handler.py", line 73, in error_handler
    yield
  File "/home/pbarker/Projects/pbarker.dev/venv/lib/python3.10/site-packages/pre_commit/main.py", line 409, in main
    return run(args.config, store, args)
  File "/home/pbarker/Projects/pbarker.dev/venv/lib/python3.10/site-packages/pre_commit/commands/run.py", line 442, in run
    install_hook_envs(to_install, store)
  File "/home/pbarker/Projects/pbarker.dev/venv/lib/python3.10/site-packages/pre_commit/repository.py", line 248, in install_hook_envs
    _hook_install(hook)
  File "/home/pbarker/Projects/pbarker.dev/venv/lib/python3.10/site-packages/pre_commit/repository.py", line 111, in _hook_install
    with open(staging, 'w') as state_file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/pbarker/.cache/pre-commit/repoiuzm_cni/rbenv-system/.install_state_v1staging'

@seanharmer
Copy link

I'm getting the same error on Windows, ruby 3.2.2, no rbenv.
pre-commit.log

@jaysoffian
Copy link
Contributor

jaysoffian commented Jun 7, 2023

Seeing same thing as #2799 (comment)

Occurs during pre-commit install-hooks with this hook:

repos:
  - repo: https://github.com/rubocop/rubocop
    rev: v1.52.0
    hooks:
      - id: rubocop

version information

pre-commit version: 3.3.2
git --version: git version 2.39.3
sys.version:
    3.11.2 (main, Mar  9 2023, 21:49:05) [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)]
sys.executable: .../bin/python3.11
os.name: posix
sys.platform: linux

error information

An unexpected error has occurred: FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/pre-commit/repo238a8i_c/rbenv-system/.install_state_v1staging'
Traceback (most recent call last):
  File ".../lib/python3.11/site-packages/pre_commit/error_handler.py", line 73, in error_handler
    yield
  File ".../lib/python3.11/site-packages/pre_commit/main.py", line 410, in main
    return install_hooks(args.config, store)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/pre_commit/commands/install_uninstall.py", line 146, in install_hooks
    install_hook_envs(all_hooks(load_config(config_file), store), store)
  File ".../lib/python3.11/site-packages/pre_commit/repository.py", line 248, in install_hook_envs
    _hook_install(hook)
  File ".../lib/python3.11/site-packages/pre_commit/repository.py", line 111, in _hook_install
    with open(staging, 'w') as state_file:
         ^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/pre-commit/repo238a8i_c/rbenv-system/.install_state_v1staging'

(I've partially elided the Python paths above.)

There directory /root/.cache/pre-commit/repo238a8i_c/rbenv-system does NOT exist. I'm not sure what's supposed to create it.

Ruby is installed via dnf -y -q install ruby ruby-devel:

Installed:
  annobin-11.05-1.el9.x86_64
  dwz-0.14-3.el9.x86_64
  efi-srpm-macros-6-2.el9_0.0.1.noarch
  file-5.39-12.el9.x86_64
  fonts-srpm-macros-1:2.0.5-7.el9.1.noarch
  gcc-plugin-annobin-11.3.1-4.3.el9.alma.x86_64
  ghc-srpm-macros-1.5.0-6.el9.noarch
  glibc-gconv-extra-2.34-60.el9.x86_64
  go-srpm-macros-3.2.0-1.el9.noarch
  kernel-srpm-macros-1.0-12.el9.noarch
  lua-srpm-macros-1-6.el9.noarch
  ocaml-srpm-macros-6-6.el9.noarch
  openblas-srpm-macros-2-11.el9.noarch
  perl-srpm-macros-1-41.el9.noarch
  pyproject-srpm-macros-1.6.2-1.el9.noarch
  python-srpm-macros-3.9-52.el9.noarch
  qt5-srpm-macros-5.15.3-1.el9.noarch
  redhat-rpm-config-199-1.el9.alma.noarch
  ruby-3.0.4-160.el9_0.x86_64
  ruby-default-gems-3.0.4-160.el9_0.noarch
  ruby-devel-3.0.4-160.el9_0.x86_64
  ruby-libs-3.0.4-160.el9_0.x86_64
  rubygem-bigdecimal-3.0.0-160.el9_0.x86_64
  rubygem-bundler-2.2.33-160.el9_0.noarch
  rubygem-io-console-0.5.7-160.el9_0.x86_64
  rubygem-json-2.5.1-160.el9_0.x86_64
  rubygem-psych-3.3.2-160.el9_0.x86_64
  rubygem-rdoc-6.3.3-160.el9_0.noarch
  rubygems-3.2.33-160.el9_0.noarch
  rust-srpm-macros-17-4.el9.noarch
  zip-3.0-35.el9.x86_64

By editing the source code to get the gem output I'm seeing this:

[root@e736e09e3d9d foo]# pre-commit install-hooks
cmd=('git', 'rev-parse', '--show-cdup')
cmd=('git', 'rev-parse', '--is-inside-git-dir')
[INFO] Installing environment for https://github.com/rubocop/rubocop.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
cmd=('gem', 'build', 'rubocop.gemspec')
stdout:
  Successfully built RubyGem
  Name: rubocop
  Version: 1.52.0
  File: rubocop-1.52.0.gem

stderr:
WARNING:  open-ended dependency on parser (>= 3.2.0.0) is not recommended
  if parser is semantically versioned, use:
    add_runtime_dependency 'parser', '~> 3.2', '>= 3.2.0.0'
WARNING:  See https://guides.rubygems.org/specification-reference/ for help

cmd=('gem', 'install', '--no-document', '--no-format-executable', '--no-user-install', 'rubocop-1.52.0.gem')
stdout:
Successfully installed rubocop-1.52.0
1 gem installed

stderr:

An unexpected error has occurred: FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/pre-commit/repo98m3m8cs/rbenv-system/.install_state_v1staging'
cmd=('git', '--version')
Check the log at /root/.cache/pre-commit/pre-commit.log

Gem env:

RubyGems Environment:
  - RUBYGEMS VERSION: 3.2.33
  - RUBY VERSION: 3.0.4 (2022-04-12 patchlevel 208) [x86_64-linux]
  - INSTALLATION DIRECTORY: /usr/share/gems
  - USER INSTALLATION DIRECTORY: /root/.local/share/gem/ruby
  - RUBY EXECUTABLE: /usr/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /usr/bin
  - SPEC CACHE DIRECTORY: /root/.local/share/gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-linux
  - GEM PATHS:
     - /usr/share/gems
     - /root/.local/share/gem/ruby
     - /usr/local/share/gems
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - "gem" => "--install-dir=/usr/local/share/gems --bindir /usr/local/bin"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /root/.local/bin
     - /root/bin
     - /opt/python/bin
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin

@asottile
Copy link
Member

asottile commented Jun 7, 2023

can you share your environment and gem and Ruby rc files?

@jaysoffian
Copy link
Contributor

@asottile I edited my comment above yours to add more information, including the stdout/stderr of the gem build commands and the output of gem env.

@asottile
Copy link
Member

asottile commented Jun 8, 2023

your --install-dir option appears to be breaking gem -- any idea where that's coming from? we use GEM_HOME to specify the target installation for gems

@jaysoffian
Copy link
Contributor

your --install-dir option appears to be breaking gem -- any idea where that's coming from? we use GEM_HOME to specify the target installation for gems

Hmm, either Red Hat's packaging or possibly it's something with newer versions of gem?

@jaysoffian
Copy link
Contributor

In my experience, you have to set both GEM_HOME and GEM_PATH.

@asottile
Copy link
Member

asottile commented Jun 8, 2023

GEM_PATH is for lookup, GEM_HOME is installation target -- only the latter is necessary for pre-commit since it intends to isolate from others (instead of cascading gem PATH lookup)

@jaysoffian
Copy link
Contributor

To be clear, the gem env output I shared is running it directly from the shell, not with GEM_HOME as set by pre-commit.

@asottile
Copy link
Member

asottile commented Jun 8, 2023

yes of course -- the useful part from gem env appears to be this configuration override:

     - "gem" => "--install-dir=/usr/local/share/gems --bindir /usr/local/bin"

afaik that's coming from some config file -- are you sure this is an otherwise empty image?

@jaysoffian
Copy link
Contributor

jaysoffian commented Jun 8, 2023

Here's what gem env looks like when I run it from pre-commit:

cmd=('gem', 'env')
stdout:
RubyGems Environment:
  - RUBYGEMS VERSION: 3.2.33
  - RUBY VERSION: 3.0.4 (2022-04-12 patchlevel 208) [x86_64-linux]
  - INSTALLATION DIRECTORY: /root/.cache/pre-commit/repo98m3m8cs/rbenv-system/gems
  - USER INSTALLATION DIRECTORY: /root/.local/share/gem/ruby
  - RUBY EXECUTABLE: /usr/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /root/.cache/pre-commit/repo98m3m8cs/rbenv-system/gems/bin
  - SPEC CACHE DIRECTORY: /root/.local/share/gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-linux
  - GEM PATHS:
     - /root/.cache/pre-commit/repo98m3m8cs/rbenv-system/gems
     - /root/.local/share/gem/ruby
     - /usr/share/gems
     - /usr/local/share/gems
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
     - "gem" => "--install-dir=/usr/local/share/gems --bindir /usr/local/bin"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /root/.cache/pre-commit/repo98m3m8cs/rbenv-system/gems/bin
     - /root/.local/bin
     - /root/bin
     - /opt/python/bin
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/sbin
     - /usr/bin
     - /sbin
     - /bin

Is that not what we expect?

This image is based on Alma 9 and is customized for Python. Ruby is installed via dnf.

@jaysoffian
Copy link
Contributor

Hmm, you're right, rubocop is installing to /usr/local/share/gems/gems/rubocop-1.52.. I gather we want it to be installing to /root/.cache/pre-commit/repo98m3m8cs/rbenv-system/gems.

Let me see if I can track that down.

@jaysoffian
Copy link
Contributor

This is apparently an issue with Alma Linux (aka Red Hat) 9.2. You can repro it with:

$ docker run -it almalinux:9.2 bash
# dnf -yq install git-core ruby-devel rubygems gcc make pip
# pip install pre-commit
# git init foo
# cd foo
# cat >.pre-commit-config.yaml<<EOF
repos:
  - repo: https://github.com/rubocop/rubocop
    rev: v1.52.0
    hooks:
      - id: rubocop
fail_fast: true
EOF
# pre-commit install-hooks

And I gather it's an issue with how Alma Linux packages up rubygems:

     - "gem" => "--install-dir=/usr/local/share/gems --bindir /usr/local/bin"

@asottile
Copy link
Member

asottile commented Jun 8, 2023

can you report it to them? unconditionally changing --install-dir seems wrong especially when the user is requesting a different one

@jaysoffian
Copy link
Contributor

jaysoffian commented Jun 8, 2023

I'm really not a fan of Ruby, but I don't think this is something RedHat is likely to fix. FWIW:

[root@f4dbfd9c2ee3 /]# irb
irb(main):001:0> Gem.operating_system_defaults
=> {"gem"=>"--install-dir=/usr/local/share/gems --bindir /usr/local/bin"}

That's where it comes from. RedHat's answer will probably be: "Pound sand. You shouldn't be using the system ruby. Install your own ruby."

RubyGems accommodated this issue:

rubygems/rubygems#5566

Can pre-commit use --install-dir instead of GEM_HOME?

@jaysoffian
Copy link
Contributor

Work-around:

$ rm -f /usr/share/rubygems/rubygems/defaults/operating_system.rb

Blech. No more time for shaving yaks so that's the solution I'm going to live with.

@jaysoffian
Copy link
Contributor

This looks to be an intended use-case by RubyGems:

rubygems/rubygems#2116

This means that pre-commit is not compatible with any packaged gem that uses this feature.

If you don't want to accommodate it I understand. Would you accept a PR that at least causes pre-commit to emit a more helpful error message?

@jaysoffian
Copy link
Contributor

@pbrkr
Copy link

pbrkr commented Jun 8, 2023

Note that Gentoo has similar customisation: https://github.com/gentoo/gentoo/blob/master/dev-ruby/rubygems/files/gentoo-defaults-5.rb

@asottile
Copy link
Member

asottile commented Jun 8, 2023

@jaysoffian did you want to try your idea from #2799 (comment) ?

@jaysoffian
Copy link
Contributor

jaysoffian commented Jun 12, 2023

@jaysoffian did you want to try your idea from #2799 (comment) ?

@asottile this patch works for me under the environment described in #2799 (comment)

diff --git i/pre_commit/languages/ruby.py w/pre_commit/languages/ruby.py
index 76631f2532..13cc628d80 100644
--- i/pre_commit/languages/ruby.py
+++ w/pre_commit/languages/ruby.py
@@ -125,6 +125,7 @@ def install_environment(
             # Need to call this after installing to set up the shims
             lang_base.setup_cmd(prefix, ('rbenv', 'rehash'))
 
+    envdir = lang_base.environment_dir(prefix, ENVIRONMENT_DIR, version)
     with in_env(prefix, version):
         lang_base.setup_cmd(
             prefix, ('gem', 'build', *prefix.star('.gemspec')),
@@ -135,6 +136,8 @@ def install_environment(
                 'gem', 'install',
                 '--no-document', '--no-format-executable',
                 '--no-user-install',
+                '--install-dir', os.path.join(envdir, 'gems'),
+                '--bindir',  os.path.join(envdir, 'bin'),
                 *prefix.star('.gem'), *additional_dependencies,
             ),
         )

Would you like a PR?

@asottile
Copy link
Member

do you want to send a pr?

@jaysoffian
Copy link
Contributor

jaysoffian commented Jun 12, 2023

do you want to send a pr?

Will do. Edit: #2905

jaysoffian added a commit to jaysoffian/pre-commit that referenced this issue Jun 13, 2023
RubyGems allows OS packagers to specify defaults for `--install-dir`
and `--bindir` and these take precedence over `GEM_HOME`. The only way
to override the defaults is to explicitly specify the options ourselves
when running `gem install`.

Examples of OSes where this is the case are RedHat 9.2 and Gentoo.

Fixes pre-commit#2799.
jaysoffian added a commit to jaysoffian/pre-commit that referenced this issue Jun 13, 2023
RubyGems allows OS packagers to specify defaults for `--install-dir`
and `--bindir` and these take precedence over `GEM_HOME`. The only way
to override the defaults is to explicitly specify the options ourselves
when running `gem install`.

Examples of OSes where this is the case are RedHat 9.2 and Gentoo.

Fixes pre-commit#2799.
saravankrish pushed a commit to saravankrish/pre-commit that referenced this issue Jul 7, 2023
RubyGems allows OS packagers to specify defaults for `--install-dir`
and `--bindir` and these take precedence over `GEM_HOME`. The only way
to override the defaults is to explicitly specify the options ourselves
when running `gem install`.

Examples of OSes where this is the case are RedHat 9.2 and Gentoo.

Fixes pre-commit#2799.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

5 participants