Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow partial cache keys #627

Closed
toadjaune opened this issue Mar 15, 2023 · 4 comments
Closed

Allow partial cache keys #627

toadjaune opened this issue Mar 15, 2023 · 4 comments
Labels
feature request New feature or request to improve the current logic

Comments

@toadjaune
Copy link

Description:

When restoring cache, the current cache management system only uses the primary cache key.

GitHub Actions cache allows specifying multiple prefixes, and restoring older caches, to speed up setup even when there has been a dependency change. Cf corresponding documentation

I'd recommend setting this as an opt-in behaviour, toggleable with a new input, because :

  • That way, this would not be a breaking change
  • Never rebuilding from scratch means stale data could stay forever in cache, which can be dangerous (e.g. a former dependency still present in a virtualenv)

Here's an example of what the configuration could look like (for poetry) :

    - uses: actions/setup-python@v4
      id: setup-python
      with:
        python-version: "3.10"
        cache: "poetry"
        allow-partial-restores: true

Here's a configuration roughly equivalent to what I'm proposing here (the cache key is slightly different, but the idea is the same)

    - uses: actions/setup-python@v4
      id: setup-python
      with:
        python-version: "3.10"
    - uses: actions/cache@v3
      with:
        path: /home/runner/.cache/pypoetry/virtualenvs
        key: poetry-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('poetry.lock') }}
        restore-keys: |
          poetry-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('poetry.lock') }}
          poetry-${{ steps.setup-python.outputs.python-version }}-

We could also expose cache key management to the users through an input parameter, but that seems much more complex and brittle.

Justification:

This change can represent a significant performance improvement on lockfile change, easily several minutes, depending on the project.

Are you willing to submit a PR?
Not at the time, as I have a working workaround (presented above).

@toadjaune toadjaune added feature request New feature or request to improve the current logic needs triage labels Mar 15, 2023
@dmitry-shibanov
Copy link
Contributor

Hello @toadjaune. Thank you for your report. We'll investigate the feature request.

@dmitry-shibanov
Copy link
Contributor

Hello @toadjaune. The purpose of caching is to cover the majority of use cases. If you need more customization we advise to use actions/cache

For now I'm going to close the issue.

@dmitry-shibanov dmitry-shibanov closed this as not planned Won't fix, can't repro, duplicate, stale Apr 5, 2023
@tragiclifestories
Copy link

@dmitry-shibanov can this be reopened? Partial cache keys are not some obscure use case but an important optimisation widely used in the CI ecosystem. I would guess that 90+% of jobs that use setup-python would benefit from it.

It's built into setup-ruby, coming in setup-node, and indeed is actually implemented for some package managers in this code base, just not all of them (eg poetry).

@iamstarkov
Copy link

@dmitry-shibanov can I ask you to reevaluate your stance and incorporate partial keys how to speed up CI builds and be consistent with the rest of setup-{tool} actions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request to improve the current logic
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants