Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDK client overrides region settings for custom s3api endpoint #3935

Closed
3 tasks done
ornokrean opened this issue May 30, 2021 · 12 comments
Closed
3 tasks done

SDK client overrides region settings for custom s3api endpoint #3935

ornokrean opened this issue May 30, 2021 · 12 comments
Labels
p3 This is a minor priority issue workaround-available

Comments

@ornokrean
Copy link

ornokrean commented May 30, 2021

Confirm by changing [ ] to [x] below to ensure that it's a bug:

Describe the bug
When using custom api endpoint, with a set of none-AWS regions, SDK client overrides this regions, blocking us from making the request.

Version of AWS SDK for Go?
v1.38.3

Version of Go (go version)?
go1. 16

To Reproduce (observed behavior)

if r.ParamsFilled() && aws.StringValue(r.Config.Region) != "us-east-1" {

func getS3Client(creds *credentials) (*s3.S3, error) {
        sess, err := getAwsClient(creds)
        if err != nil {
                return nil, fmt.Errorf("failed to create an AWS session: %s", err)
        }
        return s3.New(sess), nil
}
func getAwsClient(creds *credentials) (*session.Session, error) {
        return session.NewSession(&aws.Config{
                Region:           aws.String("my-custom-region"),
                Credentials:      awsCredentials.NewStaticCredentials(creds.awsAccess, creds.awsSecret, ""),
                Endpoint:         aws.String("https://my-endpoint"),
                HTTPClient:       creds.httpClient,
                S3ForcePathStyle: aws.Bool(true),
        })
}
func do() error {
        svcS3, err := getS3Client(creds)
        if err != nil {
                return err
        }
        bucketName := "my-bucket-name"
        // ensure bucket is created
        _, err = svcS3.CreateBucket(&s3.CreateBucketInput{
                Bucket: &bucketName,
        })
        if err != nil {
                return fmt.Errorf("failed to create %q bucket: %v", bucketName, err)
        }
}
error message:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidLocationConstraint
  </Code>
  <Message>The specified location-constraint is not valid
  </Message>
  <LocationConstraint>my-custom-region
  </LocationConstraint>
  <RequestId>A8PASP7TRCNWJ7DJ
  </RequestId>
  <HostId>***masked***
  </HostId>
</Error>

Expected behavior
We want the client not to change our region configuration, meaning he will pass it on without modifying it.

Additional context
As a workaround, we used this code:
if false && r.ParamsFilled() && aws.StringValue(r.Config.Region) != "us-east-1" {
which keeps the client in the same region as it configured.

we opened an issue #3906 and got this answer:

Try setting the CreateBucketConfiguration member of CreateBucketInput.

        _, err = svcS3.CreateBucket(&s3.CreateBucketInput{
                Bucket: &bucketName,
                CreateBucketConfiguration: &CreateBucketConfiguration{
                          Location: aws.String("my-custom-region"), 
                },

        })

and then the issue was closed, but we did try using this solution, even before we got the answer, and it did'nt work, gave us this error:
Error: failed to create "--bucketname--" bucket: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'

@ornokrean ornokrean added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels May 30, 2021
@vudh1 vudh1 self-assigned this Apr 15, 2022
@vudh1
Copy link
Contributor

vudh1 commented Apr 15, 2022

Hi, is this bug still persisting with the latest version of SDK?

@vudh1 vudh1 added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed needs-triage This issue or PR still needs to be triaged. labels Apr 15, 2022
@github-actions
Copy link

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

@github-actions github-actions bot added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Apr 18, 2022
@kayrus
Copy link

kayrus commented Apr 18, 2022

Was it fixed?

@github-actions github-actions bot removed closing-soon This issue will automatically close in 4 days unless further comments are made. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Apr 19, 2022
@kayrus
Copy link

kayrus commented Apr 19, 2022

Our workaround for the issue:

        req, _ := svc.CreateBucketRequest(&s3.CreateBucketInput{
                Bucket:                    &bucketName,
                CreateBucketConfiguration: bucketConf,
        })
        // removes "populateLocationConstraint" handler from aws/aws-sdk-go/service/s3/bucket_location.go
        req.Handlers.Validate.RemoveByName("__anonymous")

@vudh1
Copy link
Contributor

vudh1 commented Jun 1, 2022

Can you confirm if this is still persisting with the latest version of SDK?

@vudh1 vudh1 added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. closing-soon This issue will automatically close in 4 days unless further comments are made. labels Jun 1, 2022
@kayrus
Copy link

kayrus commented Jun 2, 2022

@vudh1 the bug still persists

@github-actions github-actions bot removed closing-soon This issue will automatically close in 4 days unless further comments are made. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Jun 3, 2022
@jasdel
Copy link
Contributor

jasdel commented Jun 27, 2022

Thanks for the update @kayrus. I'm attempting to reproduce the issue described where setting a custom CreateBucketConfiguration isn't being used, but not able to.

In addition removal of the populateLocationConstraint handler shouldn't be needed, because it checks if CreateBucketConfiguration is populated or not. Only setting the default value if nil. Removal of __anonymous is also pretty dangerous as it could apply to any anonymous handler, not the populateLocationConstraint one specifically.

Error: failed to create "--bucketname--" bucket: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'

With regard to this error message you're seeing when explicitly setting CreateBucketConfiguration could you provide more detail, and enable signing logging for this? From the error message it sounds like the request is being sent to us-east-1, but is the LocationConstraint set to us-west-2? The AWS SDK for Go requires all request made to the correct region. If a bucket is to be created in us-west-2 region, the request request must also be signed and made to that region as well.

The following is the code that I'm using to try to reproduce the issue, but I'm seeing the configured LocationConstraint specified being used in the request body.

package main

import (
	"fmt"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {
	sess, err := session.NewSession(&aws.Config{
		Region:           aws.String("my-custom-region"),
		Endpoint:         aws.String("https://my-endpoint"),
		S3ForcePathStyle: aws.Bool(true),
		LogLevel:         aws.LogLevel(aws.LogDebugWithHTTPBody | aws.LogDebugWithSigning),
		MaxRetries:       aws.Int(0),
	})
	if err != nil {
		panic(err)
	}

	client := s3.New(sess)

	result, err := client.CreateBucket(&s3.CreateBucketInput{
		Bucket: aws.String("my-bucket"),
		CreateBucketConfiguration: &s3.CreateBucketConfiguration{
			LocationConstraint: aws.String("us-west-2"),
		},
	})
	fmt.Println(result, err)
}

Example output.

2022/06/27 10:06:29 DEBUG: Request Signature:
---[ CANONICAL STRING  ]-----------------------------
PUT
/my-bucket

content-length:153
host:my-endpoint
x-amz-content-sha256:a4a3d263179bfcbaf88755b14ea9db3048e4179c03bbc94ccf0f13c8c2d3df6a
x-amz-date:20220627T170629Z
x-amz-security-token:<token>

content-length;host;x-amz-content-sha256;x-amz-date;x-amz-security-token
a4a3d263179bfcbaf88755b14ea9db3048e4179c03bbc94ccf0f13c8c2d3df6a
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20220627T170629Z
20220627/my-custom-region/s3/aws4_request
3adf1ff7ae8933a1df796585a400d062fdf0ca67f6d50791884152a8e23745bc
-----------------------------------------------------
2022/06/27 10:06:29 DEBUG: Request s3/CreateBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT /my-bucket HTTP/1.1
Host: my-endpoint
User-Agent: aws-sdk-go/1.44.42 (go1.18.3; darwin; amd64)
Content-Length: 153
Authorization: AWS4-HMAC-SHA256 Credential=<AccessKey>/20220627/my-custom-region/s3/aws4_request, SignedHeaders=content-length;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=4342803553d2278750a45c63775614cd6901aa36990ca48f6f0f916a85917fc1
X-Amz-Content-Sha256: a4a3d263179bfcbaf88755b14ea9db3048e4179c03bbc94ccf0f13c8c2d3df6a
X-Amz-Date: 20220627T170629Z
X-Amz-Security-Token: <token>
Accept-Encoding: gzip

<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>us-west-2</LocationConstraint></CreateBucketConfiguration>

@kayrus
Copy link

kayrus commented Jun 27, 2022

In the original message you can see:

When using custom api endpoint, with a set of none-AWS regions

it also visible in the code:

if r.ParamsFilled() && aws.StringValue(r.Config.Region) != "us-east-1" {
in := r.Params.(*CreateBucketInput)
if in.CreateBucketConfiguration == nil {
r.Params = awsutil.CopyOf(r.Params)
in = r.Params.(*CreateBucketInput)
in.CreateBucketConfiguration = &CreateBucketConfiguration{
LocationConstraint: r.Config.Region,
}
}
}

If the region doesn't correspond to us-east-1 and CreateBucketConfiguration is not set (we don't need it to be set), sdk forces this CreateBucketConfiguration. If we set an empty (not nil) CreateBucketConfiguration, then SDK will send empty constraints, which are not expected on the receiving side and it complains.

@skmcgrail
Copy link
Member

Given that there isn't a reasonably or infallible way to determine what is or isn't a custom API endpoint there isn't really a specific action we can do here without creating a breaking change. If you are using the SDK to talk to a third-party S3 API implementation then you can could remove this request handler on the request like so:

req.Handlers.Unmarshal.Remove(s3.NormalizeBucketLocationHandler)

@skmcgrail skmcgrail added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed bug This issue is a bug. labels Jul 28, 2022
@kayrus
Copy link

kayrus commented Jul 29, 2022

@skmcgrail you're mixing up two things: normalize handler (which actually has a handler name) and populateLocationConstraint handler. The last is added as an anonymous handler and removing it from the request is not reliable, because there could be other anonymous handlers. In order to keep the backward compatibility you can adjust AWS SDK and add the populateLocationConstraint handler to a request using a request.NamedHandler struct.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jul 30, 2022
@vudh1 vudh1 removed their assignment Aug 25, 2022
@RanVaknin RanVaknin added the p3 This is a minor priority issue label Mar 27, 2023
@RanVaknin
Copy link
Contributor

Hi there,

This is quite old and will not get fixed in v1. If this is still an issue on AWS SDK Go v2, please feel free to open a separate issue on the v2 repo

Thanks,
Ran~

@RanVaknin RanVaknin closed this as not planned Won't fix, can't repro, duplicate, stale May 23, 2024
Copy link

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
p3 This is a minor priority issue workaround-available
Projects
None yet
Development

No branches or pull requests

6 participants