Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't use temp credentials returned from assume role if they have special characters #599

Closed
AvivPeledTalon opened this issue Dec 7, 2022 · 8 comments · Fixed by #795
Closed
Labels
bug Something isn't working effort/medium This issue will take a few days of effort to fix p1

Comments

@AvivPeledTalon
Copy link

AvivPeledTalon commented Dec 7, 2022

Describe the bug

My team moved our to using assume-role instead of const access keys recently, and since then we've been seeing a lot of SignatureDoesNotMatch errors.

Digging deeper into this issue, we discovered that the AWS_SECRET_ACCESS_KEY in all of the runs that failed contained a special character ('/'), and that it might cause this kind of issues (see https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-troubleshooting.html, aws/aws-cli#2665, aws/aws-cli#602)

Our temporary solution is to try and generate another set of credentials in case we get this kind of error (run configure-aws-credentials with assume role again) and hope for the best.

This reduces the chances for our workflows to fail because of this issue, but it still might happen.

Since this issue appears in the awscli troubleshooting guide, I'm guessing they're not going to change the client to support that, and won't stop issuing credentials with special characters.
My hope is that at least this action itself could identify special characters in the temporary credentials it gets and regenerate them if necessary

Expected Behavior

I expect to be able to use AWS after running configure-aws-credentials successfully.

Current Behavior

Statistically, running configure-aws-credentials with assume role succeeds, but the temp credentials have special characters in them which is known to cause issues.

Reproduction Steps

If you run this on 200 Windows instances (using matrix for example) you will see some of them failing due to Signature Mismatch.

If you check the credentials you got on the failed runs they will have special characters in them.

steps:
  - name: Configure AWS credentials
    uses: aws-actions/configure-aws-credentials@v1-node16
    with:
      role-to-assume: <YOUR_ROLE>
      aws-region: <YOUR_REGION>

  - name: identity check
    run: |
      aws sts get-caller-identity

Possible Solution

  1. Retry assumerole until you get credentials without special characters. You can control this behavior using an action input 'allow-special-characters'
  2. Even easier to implement: Fail the action if you get temp credentials with special characters so we could retry it ourselves until it works. This behavior can be controlled using an action input 'fail-special-characters'
  3. awscli could deal with the special characters issue?
  4. AWS backend could stop issuing credentials with special characters on AssumeRole

Additional Information/Context

No response

@AvivPeledTalon AvivPeledTalon added bug Something isn't working needs-triage This issue still needs to be triaged labels Dec 7, 2022
@ironstas1k
Copy link

Experiencing the same issue

@AvivPeledTalon
Copy link
Author

I have opened a PR with a workaround for this issue, regenerating the credentials until we get ones with no special characters.

#613

In my experiments, it's even enough to only make sure that the secret access key does not start with a special character, this way we cause way less retries.

@paulmowat
Copy link

Some of my team are encountering this as well.

@AvivPeledTalon
Copy link
Author

Some of my team are encountering this as well.

No one's taken a look at my PR, but we've been using it ever since and it works really well for us, so consider trying something similar

@peterwoodworth peterwoodworth added p1 effort/medium This issue will take a few days of effort to fix and removed needs-triage This issue still needs to be triaged labels Feb 21, 2023
@peterwoodworth
Copy link
Contributor

From the CLI docs:

Possible cause: Your operating system is mishandling AWS secret keys that contain certain special characters
If your AWS secret key includes certain special characters, such as -, +, /, or %, some operating system variants process the string improperly and cause the secret key string to be interpreted incorrectly.
If you process your access keys and secret keys using other tools or scripts, such as tools that build the credentials file on a new instance as part of its creation, those tools and scripts might have their own handling of special characters that causes them to be transformed into something that AWS no longer recognizes.
We suggest regenerating the secret key to get one that does not include the special character causing issues.

Given this last line, I'd lean towards the best solution for us being regenerating the credentials if a special character is detected

@AvivPeledTalon
Copy link
Author

From the CLI docs:

Possible cause: Your operating system is mishandling AWS secret keys that contain certain special characters
If your AWS secret key includes certain special characters, such as -, +, /, or %, some operating system variants process the string improperly and cause the secret key string to be interpreted incorrectly.
If you process your access keys and secret keys using other tools or scripts, such as tools that build the credentials file on a new instance as part of its creation, those tools and scripts might have their own handling of special characters that causes them to be transformed into something that AWS no longer recognizes.
We suggest regenerating the secret key to get one that does not include the special character causing issues.

Given this last line, I'd lean towards the best solution for us being regenerating the credentials if a special character is detected

That is exactly what I did in my (closed without any discussion or comment) PR here #613 which my team's has been using for quite a while now without any issues

@peterwoodworth
Copy link
Contributor

Thanks for the previous PR @AvivPeledTalon, and sorry we closed this with little communication.

We're currently in active development in the next version of this action. That makes reviewing PRs for the current version tricky to review especially with the limited dev time we can commit to this project, since we're making a few big changes in the next version (converting the project to ts, and using js v3 instead of v2). If this feature is important to you, I would recommend sticking with your own solution for now, and look forward to our next major version

@github-actions
Copy link

** Note **
Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working effort/medium This issue will take a few days of effort to fix p1
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants