Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to aws-sdk-sqs="1.23.0" causing timeouts #1148

Closed
Smotrov opened this issue May 14, 2024 · 10 comments
Closed

Upgrade to aws-sdk-sqs="1.23.0" causing timeouts #1148

Smotrov opened this issue May 14, 2024 · 10 comments
Labels
bug This issue is a bug. p1 This is a high priority issue

Comments

@Smotrov
Copy link

Smotrov commented May 14, 2024

Describe the bug

After updating to aws-sdk-sqs = "1.23.0" facing massive timeouts in the log.

Expected Behavior

Would be good if there will be no timeouts.

Current Behavior

Vast majority of calls to SQS are time outed with following error

[ERROR] SQS processor failed: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: ThroughputBelowMinimum { expected: Throughput { bytes_read: 1, per_time_elapsed: 1s }, actual: Throughput { bytes_read: 0, per_time_elapsed: 1s } }, connection: Unknown } })

Reproduction Steps

Upgrade to aws-sdk-sqs = "1.23.0"
Try to work with AWS SQS

Possible Solution

No response

Additional Information/Context

aws-sdk-sqs = "1.24.0" does not solving the bug

Version

➜  cargo_workspace git:(main) ✗ cargo tree | grep aws-
├── aws-config v1.3.0
│   ├── aws-credential-types v1.2.0
│   │   ├── aws-smithy-async v1.2.1
│   │   ├── aws-smithy-runtime-api v1.6.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-types v1.1.9
│   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-runtime v1.2.2
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-sigv4 v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-eventstream v0.60.4
│   │   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   │   ├── aws-smithy-http v0.60.8
│   │   │   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-eventstream v0.60.4 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1
│   │   │   ├── aws-credential-types v1.2.0 (*)
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-sdk-sso v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7
│   │   │   └── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime v1.5.0
│   │   │   ├── aws-smithy-async v1.2.1 (*)
│   │   │   ├── aws-smithy-http v0.60.8 (*)
│   │   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-ssooidc v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-sdk-sts v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-query v0.60.7
│   │   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-smithy-xml v0.60.8
│   │   ├── aws-types v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-apigateway v1.22.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-athena v1.23.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-cloudwatchlogs v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-dynamodb v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-s3 v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-sigv4 v1.2.1 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-checksums v0.60.7
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-eventstream v0.60.4 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-xml v0.60.8 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-sns v1.23.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-query v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-smithy-xml v0.60.8 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-sqs v1.24.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-sdk-ssm v1.25.0
│   ├── aws-credential-types v1.2.0 (*)
│   ├── aws-runtime v1.2.2 (*)
│   ├── aws-smithy-async v1.2.1 (*)
│   ├── aws-smithy-http v0.60.8 (*)
│   ├── aws-smithy-json v0.60.7 (*)
│   ├── aws-smithy-runtime v1.5.0 (*)
│   ├── aws-smithy-runtime-api v1.6.0 (*)
│   ├── aws-smithy-types v1.1.9 (*)
│   ├── aws-types v1.2.1 (*)
├── aws-smithy-types v1.1.9 (*)
│   ├── aws-sdk-dynamodb v1.25.0 (*)
│   ├── aws-sdk-dynamodbstreams v1.22.0
│   │   ├── aws-credential-types v1.2.0 (*)
│   │   ├── aws-runtime v1.2.2 (*)
│   │   ├── aws-smithy-async v1.2.1 (*)
│   │   ├── aws-smithy-http v0.60.8 (*)
│   │   ├── aws-smithy-json v0.60.7 (*)
│   │   ├── aws-smithy-runtime v1.5.0 (*)
│   │   ├── aws-smithy-runtime-api v1.6.0 (*)
│   │   ├── aws-smithy-types v1.1.9 (*)
│   │   ├── aws-types v1.2.1 (*)


### Environment details (OS name and version, etc.)

Linux

### Logs

_No response_
@Smotrov Smotrov added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels May 14, 2024
@aajtodd aajtodd added p1 This is a high priority issue and removed needs-triage This issue or PR still needs to be triaged. labels May 14, 2024
@Turbo87
Copy link

Turbo87 commented May 14, 2024

FWIW after merging rust-lang/crates.io#8610 we noticed similar issues on the crates.io staging environment:

Failed to receive SQS queue message: dispatch failure: timeout: minimum throughput was specified at 1 B/s, but throughput of 0 B/s was observed

Turbo87 added a commit to Turbo87/crates.io that referenced this issue May 14, 2024
…-sdk-rust-monorepo"

This reverts commit 34bd67e, reversing
changes made to b6b4386.

see awslabs/aws-sdk-rust#1148
Turbo87 added a commit to rust-lang/crates.io that referenced this issue May 14, 2024
@Smotrov Smotrov changed the title Upgrade to aws-sdk-sqs = "1.23.0" causing timeouts Upgrade to aws-sdk-sqs="1.23.0" causing timeouts May 14, 2024
@aajtodd
Copy link
Contributor

aajtodd commented May 14, 2024

Thanks for the issue, we are looking into this now and how best to address it.


As a workaround you can disable upload throughput protection:

    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .stalled_stream_protection(
            StalledStreamProtectionConfig::enabled()
                .upload_enabled(false)
                .build(),
        )
        .load()
        .await;

This will keep stalled stream protection for downloads enabled and only disable uploads (which was the default prior to BehaviorVersion::v2024_03_28()). If you want to disable it completely for uploads and downloads:

    let config = aws_config::defaults(aws_config::BehaviorVersion::latest())
        .stalled_stream_protection(StalledStreamProtectionConfig::disabled())
        .load()
        .await;

@kolloch
Copy link

kolloch commented May 15, 2024

Explicitly setting a deprecated behavior version fixes this as well:

BehaviorVersion::v2023_11_09() instead of BehaviorVersion::v2024_03_28()

github-merge-queue bot pushed a commit to smithy-lang/smithy-rs that referenced this issue May 17, 2024
…lete (#3644)

## Motivation and Context
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here -->
* awslabs/aws-sdk-rust#1141
* awslabs/aws-sdk-rust#1146
* awslabs/aws-sdk-rust#1148


## Description
* Disables stalled stream upload protection for requests with an
empty/zero length body.
* Disables stalled stream upload throughput checking once the request
body has been read and handed off to the HTTP layer.


## Testing
Additional integration tests added covering empty bodies and completed
uploads.

Tested SQS issue against latest runtime and can see it works now. The S3
`CopyObject` issue is related to downloads and will need a different
solution.

## Checklist
<!--- If a checkbox below is not applicable, then please DELETE it
rather than leaving it unchecked -->
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the
smithy-rs codegen or runtime crates
- [x] I have updated `CHANGELOG.next.toml` if I made changes to the AWS
SDK, generated SDK code, or SDK runtime crates

----

_By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice._

---------

Co-authored-by: Zelda Hessler <zhessler@amazon.com>
Co-authored-by: ysaito1001 <awsaito@amazon.com>
@Smotrov
Copy link
Author

Smotrov commented May 21, 2024

It seams like the fix is already merged into main? Which crate version includes it?

@Velfi
Copy link
Contributor

Velfi commented May 21, 2024

We just released the runtime crate updates: https://github.com/smithy-lang/smithy-rs/releases/tag/release-2024-05-21
Once a release for the SDK crates runs later today (in about two hours) they'll include this fix.

@Smotrov
Copy link
Author

Smotrov commented May 22, 2024

New release brought new errors. Unfortunately unable so far...
When getting S3 metadata:

ResponseError(ResponseError { source: ContentLengthError { expected: 54736384, received: 0 }, raw: Response { status: StatusCode(200), headers: Headers { headers: {"x-amz-id-2": HeaderValue { _private: H0("uxo9Lk8l/KkksF3/QH5tFj1Ec4WiCO1yUKj/LrEdZPxeQ925zZFxTfJVqtDXhXnFou62IvgyKZWJH9QejfvyePXmf5qnXx/4") }, "x-amz-request-id": HeaderValue { _private: H0("S5Q8M1H1YPBS44KG") }, "date": HeaderValue { _private: H0("Wed, 22 May 2024 07:16:57 GMT") }, "last-modified": HeaderValue { _private: H0("Wed, 22 May 2024 07:16:53 GMT") }, "x-amz-expiration": HeaderValue { _private: H0("expiry-date=\"Fri, 24 May 2024 00:00:00 GMT\", rule-id=\"delete-unprocessed-files\"") }, "etag": HeaderValue { _private: H0("\"749e7afbd0311a630201cb61d439730e\"") }, "x-amz-server-side-encryption": HeaderValue { _private: H0("AES256") }, "x-amz-meta-pcc": HeaderValue { _private: H0("YYY") }, "x-amz-meta-pos": HeaderValue { _private: H0("US") }, "x-amz-meta-partition": HeaderValue { _private: H0("filtered") }, "accept-ranges": HeaderValue { _private: H0("bytes") }, "content-type": HeaderValue { _private: H0("application/octet-stream") }, "server": HeaderValue { _private: H0("AmazonS3") }, "content-length": HeaderValue { _private: H0("54736384") }} }, body: SdkBody { inner: Taken, retryable: false }, extensions: Extensions { extensions_02x: Extensions, extensions_1x: Extensions } } }) raw response Some(Response { status: StatusCode(200), headers: Headers { headers: {"x-amz-id-2": HeaderValue { _private: H0("uxo9Lk8l/KkksF3/QH5tFj1Ec4WiCO1yUKj/LrEdZPxeQ925zZFxTfJVqtDXhXnFou62IvgyKZWJH9QejfvyePXmf5qnXx/4") }, "x-amz-request-id": HeaderValue { _private: H0("S5Q8M1H1YPBS44KG") }, "date": HeaderValue { _private: H0("Wed, 22 May 2024 07:16:57 GMT") }, "last-modified": HeaderValue { _private: H0("Wed, 22 May 2024 07:16:53 GMT") }, "x-amz-expiration": HeaderValue { _private: H0("expiry-date=\"Fri, 24 May 2024 00:00:00 GMT\", rule-id=\"delete-unprocessed-files\"") }, "etag": HeaderValue { _private: H0("\"749e7afbd0311a630201cb61d439730e\"") }, "x-amz-server-side-encryption": HeaderValue { _private: H0("AES256") }, "x-amz-meta-pcc": HeaderValue { _private: H0("YYY") }, "x-amz-meta-pos": HeaderValue { _private: H0("US") }, "x-amz-meta-partition": HeaderValue { _private: H0("filtered") }, "accept-ranges": HeaderValue { _private: H0("bytes") }, "content-type": HeaderValue { _private: H0("application/octet-stream") }, "server": HeaderValue { _private: H0("AmazonS3") }, "content-length": HeaderValue { _private: H0("54736384") }} }, body: SdkBody { inner: Taken, retryable: false }, extensions: Extensions { extensions_02x: Extensions, extensions_1x: Extensions } })

@Velfi
Copy link
Contributor

Velfi commented May 22, 2024

I'm looking into a fix right now. Hopefully we can get it out this week.

@ysaito1001
Copy link
Collaborator

@Smotrov, We have just released a new version in the smithy-rs repository. Curious to know whether the issue can be resolved if you use aws-smithy-runtime version 1.5.4.

@Smotrov
Copy link
Author

Smotrov commented May 23, 2024

Dear @Velfi, @aajtodd than you for you personal involvement.
It seams like the issue is solved now 🎉

@Smotrov Smotrov closed this as completed May 23, 2024
Copy link

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. p1 This is a high priority issue
Projects
None yet
Development

No branches or pull requests

6 participants