NoCredentialProviders: no valid providers in chain when adding a certain aws profile #4989

HenryXie1 · 2023-09-14T21:52:02Z

Describe the bug

We have blow code to describe eks cluster

sess, err := session.NewSessionWithOptions(session.Options{
		Config:            aws.Config{Region: aws.String("ap-southeast-2"), CredentialsChainVerboseErrors: aws.Bool(true)},
		SharedConfigFiles: []string{awsConfigPath},
		SharedConfigState: session.SharedConfigEnable,
		Profile:           clusterName,
	})
	if err != nil {
		fmt.Println("clustername: " + clusterName)
		return err
	}

	fmt.Println("new cluster session good: " + clusterName)
	eksClient := eks.New(sess)
	cluster, err := eksClient.DescribeCluster(&eks.DescribeClusterInput{Name: &clusterName})

	if err != nil {
		return err
	}

it works well before.
after we add below line in the ./aws/config , it starts error out

[profile e02-admin]
sso_start_url = ******
role_arn = arn:aws:iam::******:role/AdministratorRole
source_profile = e02
sso_role_name = PlatformAdmin
region = ap-southeast-2

The error is

new cluster session good: e02
 NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, e02.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "[http://169.254.169.254/latest/meta-data/iam/security-credentials/ ](http://169.254.169.254/latest/meta-data/iam/security-credentials/)": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Expected Behavior

it was working well before we add e02-admin profile.

Current Behavior

it error out after we add this profile in the .aws/config

[profile e02-admin]
sso_start_url = ******
role_arn = arn:aws:iam::******:role/AdministratorRole
source_profile = e02
sso_role_name = PlatformAdmin
region = ap-southeast-2

profile e02 exists before this issue happens

[profile e02]
sso_start_url = ***
sso_region = ap-southeast-2
sso_account_id = ***
sso_role_name = PlatformAdmin
region = ap-southeast-2

Reproduction Steps

Add e02-admin profile in .aws/config
then the issue is reproduced

Possible Solution

change source_profile in e02-admin from e02 to others (ie d02)
or
remove e02-admin from .aws/config
the issue is gone

Additional Information/Context

Go verison

go 1.19

require (
	github.com/aws/aws-sdk-go v1.45.9
	github.com/awslabs/goformation/v4 v4.19.5
	github.com/buger/jsonparser v1.1.1
	github.com/urfave/cli v1.22.5
	gopkg.in/yaml.v2 v2.4.0
	k8s.io/api v0.25.3
	k8s.io/apimachinery v0.25.3
	k8s.io/client-go v0.25.3
)

SDK version used

1.45.9

Environment details (Version of Go (`go version`)? OS name and version, etc.)

go version go1.19.1 darwin/amd64

The text was updated successfully, but these errors were encountered:

HenryXie1 · 2023-09-14T21:54:42Z

the aws cli is working well with the e02-admin and e02 profiles

❯ aws eks update-kubeconfig --profile e02-admin --name e02
Updated context arn:aws:eks:ap-southeast-2:***:cluster/e02 in /Users/xieho/.kube/config

~/Documents
❯ aws s3 ls --profile e02-admin
2023-07-06 08:46:41 test
2023-08-23 14:45:16 test1
2022-08-26 15:00:20 test2

aws eks describe-cluster --name e02  --profile e02-admin
...
aws eks describe-cluster --name e02  --profile e02
...

HenryXie1 · 2023-09-17T21:17:51Z

I enabled LogLevel: aws.LogLevel(aws.LogDebugWithHTTPBody) in the code, find below
trace

2023/09/18 07:02:01 DEBUG: Request ec2metadata/GetToken Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /latest/api/token HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.45.9 (go1.19.1; darwin; amd64)
Content-Length: 0
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
Accept-Encoding: gzip
....

After I implemented workaround, change source_profile from e02 to d02
the issue is gone. The trace is like

2023/09/18 07:03:02 DEBUG: Request SSO/GetRoleCredentials Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /federation/credentials?account_id=1234***&role_name=PlatformAdmin HTTP/1.1
Host: portal.sso.ap-southeast-2.amazonaws.com
User-Agent: aws-sdk-go/1.45.9 (go1.19.1; darwin; amd64)
X-Amz-Sso_bearer_token: *****
Accept-Encoding: gzip

Because I use sso federation in aws profile, I wonder why sdk is looking for ec2metadata/GetToken in this situation.

HenryXie1 · 2023-09-17T21:21:13Z

We tested on d02 profile which has same content as e02, it is working well.

In this case, it seems that the SDK is not correctly recognizing the SSO federation credentials only from e02 profile, causing it to fall back to the EC2 metadata provider.

but AWS CLI is working fine with the same profiles, this might be a bug in the SDK
Wonder anything special with the key word e02 in aws sdk go?

RanVaknin · 2023-09-20T18:38:37Z

Hi @HenryXie1 ,

Thanks for all the details, this is indeed an interesting behavior.

If IRSA credentials fail, the logs would typically show a failed request to the OIDC identity provider and then to the STS AssumeRoleWithWebIdentity operation. Errors could include token validation issues, service errors, etc. Are you able to see any logs of errors with that credentials provider?

Another thought I have was for you to set the desired profile explicitly in the SDK and see if that helps with resolving the issue, or at least would surface some more helpful errors. You can check out this blogpost .

Let us know what you find.
Thanks,
Ran~

HenryXie1 · 2023-09-21T06:12:11Z

Hey @RanVaknin
Thanks for your reply
Here are some input I have

Are you able to see any logs of errors with that credentials provider?

no, do you know how to enable some debug trace for it?

Another thought I have was for you to set the desired profile explicitly in the SDK and see if that helps with resolving the issue, or at least would surface some more helpful errors. You can check out this blogpost .

I add below code to debug, but i don't see any output from sts client and error is also nil

client := sts.New(sess)
	fmt.Println("sts client info2:")
	fmt.Println("sts client:", *client)
	identity, err := client.GetCallerIdentity(&sts.GetCallerIdentityInput{})
	fmt.Println("sts client info3:")
	fmt.Println("sts identity:", *identity)

    if err != nil {
        fmt.Println("error:", err)
        os.Exit(1)
    }

    fmt.Printf(
        "Account: %s\nUserID: %s\nARN: %s\n",
        aws.StringValue(identity.Account),
        aws.StringValue(identity.UserId),
        aws.StringValue(identity.Arn),
    )

sts client info1:
sts client info2:
sts client: {0xc000423c00}
2023/09/21 16:03:10 DEBUG: Request ec2metadata/GetToken Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /latest/api/token HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.45.9 (go1.19.1; darwin; amd64)
Content-Length: 0
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
Accept-Encoding: gzip

After I change source_profile in e02-admin from e02 to d02(workaround)
The output is

❯ ~/github-cba/pksctl/bin/pksctl generate kube-config
sts client info1:
sts client info2:
sts client: {0xc000411c00}
2023/09/21 16:05:49 DEBUG: Request SSO/GetRoleCredentials Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /federation/credentials?account_id=****&role_name=PlatformAdmin HTTP/1.1
Host: portal.sso.ap-southeast-2.amazonaws.com
User-Agent: aws-sdk-go/1.45.9 (go1.19.1; darwin; amd64)
X-Amz-Sso_bearer_token: a****
Accept-Encoding: gzip

It is weird that i don't see println for sts client info3

RanVaknin · 2023-09-21T20:45:09Z

Hi @HenryXie1,

We think this is an issue with how the SDK's INI loader parses numbers. @lucix-aws has raised a PR regarding a similar issue we saw on v2, and tested the ini parser with d02 as an argument without an issue, and then e02 and we saw an issue. It might be related.
Can you please wait for tomorrow's release, pull the latest version of the SDK, and see if this solves your issue?

Thanks,
Ran~

lucix-aws · 2023-09-22T19:48:00Z

@HenryXie1 --

Pursuant to @RanVaknin's comment above, release 1.45.15 has just been tagged, which should solve your issue.

HenryXie1 · 2023-09-22T20:36:11Z

Great, thanks @lucix-aws @RanVaknin

lucix-aws · 2023-09-23T20:42:29Z

Please let us know if that unblocks you so we can close this out or investigate further.

HenryXie1 · 2023-09-24T19:23:40Z

@lucix-aws I confirm the issue is fixed by upgrading aws go sdk to 1.45.15

github-actions · 2023-09-24T23:39:51Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

HenryXie1 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 14, 2023

RanVaknin self-assigned this Sep 20, 2023

RanVaknin added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Sep 20, 2023

RanVaknin added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Sep 21, 2023

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Sep 23, 2023

lucix-aws closed this as completed Sep 24, 2023

adrianmoisey mentioned this issue Dec 8, 2023

VDS secret sync errors are not reported hashicorp/vault-secrets-operator#507

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NoCredentialProviders: no valid providers in chain when adding a certain aws profile #4989

NoCredentialProviders: no valid providers in chain when adding a certain aws profile #4989

HenryXie1 commented Sep 14, 2023 •

edited

HenryXie1 commented Sep 14, 2023 •

edited

HenryXie1 commented Sep 17, 2023 •

edited

HenryXie1 commented Sep 17, 2023 •

edited

RanVaknin commented Sep 20, 2023

HenryXie1 commented Sep 21, 2023 •

edited

RanVaknin commented Sep 21, 2023 •

edited

lucix-aws commented Sep 22, 2023

HenryXie1 commented Sep 22, 2023

lucix-aws commented Sep 23, 2023

HenryXie1 commented Sep 24, 2023

github-actions bot commented Sep 24, 2023

NoCredentialProviders: no valid providers in chain when adding a certain aws profile #4989

NoCredentialProviders: no valid providers in chain when adding a certain aws profile #4989

Comments

HenryXie1 commented Sep 14, 2023 • edited

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

SDK version used

Environment details (Version of Go (go version)? OS name and version, etc.)

HenryXie1 commented Sep 14, 2023 • edited

HenryXie1 commented Sep 17, 2023 • edited

HenryXie1 commented Sep 17, 2023 • edited

RanVaknin commented Sep 20, 2023

HenryXie1 commented Sep 21, 2023 • edited

RanVaknin commented Sep 21, 2023 • edited

lucix-aws commented Sep 22, 2023

HenryXie1 commented Sep 22, 2023

lucix-aws commented Sep 23, 2023

HenryXie1 commented Sep 24, 2023

github-actions bot commented Sep 24, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

HenryXie1 commented Sep 14, 2023 •

edited

Environment details (Version of Go (`go version`)? OS name and version, etc.)

HenryXie1 commented Sep 14, 2023 •

edited

HenryXie1 commented Sep 17, 2023 •

edited

HenryXie1 commented Sep 17, 2023 •

edited

HenryXie1 commented Sep 21, 2023 •

edited

RanVaknin commented Sep 21, 2023 •

edited