Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: large message in SQS and SNS #1310

Merged
merged 36 commits into from
Aug 2, 2023
Merged

feat: large message in SQS and SNS #1310

merged 36 commits into from
Aug 2, 2023
+3,650 −89

Conversation

jeromevdl
Copy link
Contributor

@jeromevdl jeromevdl commented Jul 25, 2023 •
edited
Loading

Issue #, if available: #1259

Description of changes:

New module to handle large messages in SQS & SNS offloaded to S3.

Checklist

Breaking change checklist

Deprecate part of the SQS module @SqsLargeMessage (still available in v1 but will be removed in v2).

RFC issue #:

  • Migration process documented
  • Implement warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Sorry, something went wrong.

@jeromevdl jeromevdl changed the title Feat: large message in SQS and SNS feat: large message in SQS and SNS Jul 25, 2023
@codecov-commenter
Copy link

codecov-commenter commented Jul 25, 2023 •
edited
Loading

Codecov Report

Patch and project coverage have no change.

Comparison is base (3a106ca) 79.35% compared to head (03ee552) 79.35%.

❗ Current head 03ee552 differs from pull request most recent head 1c4aca2. Consider uploading reports for the commit 1c4aca2 to get more accurate results

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files 
@@            Coverage Diff            @@
##               main    #1310   +/-   ##
=========================================
  Coverage     79.35%   79.35%           
  Complexity      641      641           
=========================================
  Files            73       73           
  Lines          2446     2446           
  Branches        253      253           
=========================================
  Hits           1941     1941           
  Misses          425      425           
  Partials         80       80
Files Changed Coverage Δ
...ertools/idempotency/internal/IdempotentAspect.java 87.09% <ø> (ø)
...oftware/amazon/lambda/powertools/sqs/SqsUtils.java 80.76% <ø> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jeromevdl
Copy link
Contributor Author

I still have the doc and migration guide to write. But want your first feedback on the code.

scottgerring
scottgerring reviewed Jul 26, 2023
View reviewed changes
Copy link
Contributor

@scottgerring scottgerring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some initial comments! I think the API is solid and as we discussed on the RFC.

@jeromevdl jeromevdl force-pushed the feat/large_message branch from bea9374 to 7ea0088 Compare July 27, 2023 09:59
jeromevdl and others added 17 commits July 27, 2023 21:00
@jeromevdl
initial version of large message handling

Verified

This commit was signed with the committer’s verified signature.
nlf nlf
SSH Key Fingerprint: BJwQLfDy+6IM7VyCh65kB/gIPFFn29cppiLur87xad0
Verified
Learn about vigilant mode
45c2c02
@jeromevdl
Improve module documentation
3bfeaab
@jeromevdl
deprecate sqs module
f4308ca
@jeromevdl
cleanup and use payloadoffloading library instead of reinventing wheel
9b39cbb
@jeromevdl
add some behavior from the sns/sqs extended client lib
70e6233
@jeromevdl
add md5 logic and logs
fa23718
@jeromevdl
fix tests
c298e4f
@jeromevdl
e2e tests for large messages + improve Infra
8959083
@jeromevdl
fix spotbug and sonar most important issues
439be98
@jeromevdl
fix spotbug
3d7e075
@jeromevdl
fix e2e bucket deletion
0aab25b
@jeromevdl
fix spotbugs
c7a5a59
@jeromevdl
stop e2e tests after 1 failure to optimize cost/time
515344c
@jeromevdl @scottgerring
Apply suggestions from code review
220e591 
Co-authored-by: Scott Gerring <scottgerring@users.noreply.github.com>
@jeromevdl
add copyright to files
b40a29e
@jeromevdl
remarks from review
d3683b2
@jeromevdl
documentation + migration guide
dfbe15e
@jeromevdl jeromevdl force-pushed the feat/large_message branch from f92f309 to dfbe15e Compare July 27, 2023 21:14
scottgerring
scottgerring suggested changes Jul 28, 2023
View reviewed changes
Copy link
Contributor

@scottgerring scottgerring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small doco nitpicks!

kozub and others added 2 commits July 28, 2023 13:25
@kozub @jeromevdl
docs: versioning - fix typo (#1322)
d66e1ca 
* docs: add support for docs versioning (#1239)

* docs: change configuration to release docs only during the release or by hand. Rollback config for building doc for PR. (#1239)

* docs: HotFix - workflow path(#1239)

* docs: HotFix - workflow path(#1239)

* docs: typo. (#1239)
@jeromevdl
remove auto-format which breaks doc and add mermaid diagram
3d5629e
jeromevdl and others added 11 commits July 28, 2023 13:36
@jeromevdl @scottgerring
Apply suggestions from code review
0311f61 
Co-authored-by: Scott Gerring <scottgerring@users.noreply.github.com>
@jeromevdl
add logs to e2e test
abbc403
@jeromevdl
force jackson core to avoid taking the transitive one from payloadoff…
399898c 
…loading (1.14.2)
@jeromevdl
annotations order (idempotent before large message)
93441a9
@jeromevdl
end-to-end tests
17af703
@jeromevdl
update documentation (idempotency)
ddeaf0a
@jeromevdl
add (c) header
714db87
@scottgerring @jeromevdl
fix: Rollback doc changes (#1323)
9aa9a58 
* Revert "docs: versioning - fix typo (#1322)"

This reverts commit a0c7d38.

* Revert "docs: add support for docs versioning (#1239) (#1293)"

This reverts commit 3dc8405.
@dependabot @jeromevdl
build(deps): bump aws.sdk.version from 2.20.111 to 2.20.114 (#1324)
df02854 
Bumps `aws.sdk.version` from 2.20.111 to 2.20.114.

Updates `software.amazon.awssdk:bom` from 2.20.111 to 2.20.114

Updates `software.amazon.awssdk:http-client-spi` from 2.20.111 to 2.20.114

Updates `software.amazon.awssdk:url-connection-client` from 2.20.111 to 2.20.114

Updates `software.amazon.awssdk:s3` from 2.20.111 to 2.20.114

Updates `software.amazon.awssdk:lambda` from 2.20.111 to 2.20.114

Updates `software.amazon.awssdk:cloudwatch` from 2.20.111 to 2.20.114

Updates `software.amazon.awssdk:xray` from 2.20.111 to 2.20.114

Updates `software.amazon.awssdk:cloudformation` from 2.20.111 to 2.20.114

Updates `software.amazon.awssdk:sts` from 2.20.111 to 2.20.114

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:http-client-spi
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:url-connection-client
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:s3
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:lambda
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:cloudwatch
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:xray
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:cloudformation
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: software.amazon.awssdk:sts
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@jeromevdl
Merge branch 'main' into feat/large_message
d24bb15
@jeromevdl
fix junit
cfd4fa0
scottgerring
scottgerring reviewed Jul 31, 2023
View reviewed changes
jeromevdl and others added 2 commits July 31, 2023 11:28
@jeromevdl @scottgerring
Apply suggestions from code review
78da8a7 
Co-authored-by: Scott Gerring <scottgerring@users.noreply.github.com>
@jeromevdl
logs and doc
57bdd77
@jeromevdl jeromevdl self-assigned this Aug 1, 2023
@sonarqubecloud
Copy link

sonarqubecloud bot commented Aug 2, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 35 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

scottgerring
scottgerring approved these changes Aug 2, 2023
View reviewed changes
Copy link
Contributor

@scottgerring scottgerring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jeromevdl jeromevdl merged commit a05823b into main Aug 2, 2023
@jeromevdl jeromevdl deleted the feat/large_message branch August 2, 2023 09:53
@scottgerring scottgerring mentioned this pull request Sep 28, 2023
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@scottgerring scottgerring

@mriccia mriccia

Assignees

@jeromevdl jeromevdl

Labels
size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@jeromevdl @codecov-commenter @scottgerring @kozub