Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cache): dynamodb implementation #953

Merged
merged 1 commit into from
Nov 22, 2022

Conversation

chofnar
Copy link
Contributor

@chofnar chofnar commented Nov 3, 2022

What type of PR is this?

enhancement

Which issue does this PR fix:

Part of #564

What does this PR do / Why do we need it:

Added AWS DynamoDB driver.

Will this break upgrades or downgrades?

If a client uses DynamoDB and decides to downgrade, they must make sure that the database is migrated to BoltDB to avoid data loss.

New validations that might break stuff:
local storage driver, but explicit RemoteCache=true, this is not valid, so error out
remote storage driver, but no "cacheDriver" config section, error out

Defaults:
local storage driver -> RemoteCache=false
remote storage driver (s3) -> RemoteCache=true

Does this PR introduce any user-facing change?:

Adds the possibility to use AWS DynamoDB instead of BoltDB

Problems

  • need to specify protocol in front of dynamodb endpoint (known issue)

To do

  • fix lint problems
  • remove comment from controller.go line 461
  • fix misleading example config (local storage with remote cache, is invalid)
  • check that the example config actually works
  • add negative tests
  • previous tests pass
  • add coverage tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@chofnar chofnar self-assigned this Nov 3, 2022
@chofnar chofnar added the feature New feature or request label Nov 3, 2022
@chofnar chofnar force-pushed the dynamodbReimplementation branch 4 times, most recently from 7e053e8 to 7a515b1 Compare November 4, 2022 13:22
@chofnar chofnar changed the title feature(cache): dynamodb implementation feat(cache): dynamodb implementation Nov 4, 2022
@chofnar chofnar force-pushed the dynamodbReimplementation branch 2 times, most recently from 3c1a9e4 to aa9db8a Compare November 7, 2022 08:22
@codecov
Copy link

codecov bot commented Nov 7, 2022

Codecov Report

Merging #953 (f9b575e) into main (94d073c) will not change coverage.
The diff coverage is n/a.

❗ Current head f9b575e differs from pull request most recent head ebb0c99. Consider uploading reports for the commit ebb0c99 to get more accurate results

@@           Coverage Diff           @@
##             main     #953   +/-   ##
=======================================
  Coverage   88.39%   88.39%           
=======================================
  Files          74       74           
  Lines       14128    14128           
=======================================
  Hits        12488    12488           
  Misses       1277     1277           
  Partials      363      363           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@chofnar chofnar force-pushed the dynamodbReimplementation branch 4 times, most recently from 6540e04 to 863e6f0 Compare November 7, 2022 15:53
@chofnar chofnar marked this pull request as ready for review November 7, 2022 16:47
@chofnar chofnar force-pushed the dynamodbReimplementation branch 6 times, most recently from 72a5ac3 to ebb0c99 Compare November 9, 2022 08:10
@andaaron andaaron linked an issue Nov 9, 2022 that may be closed by this pull request
2 tasks
@andaaron andaaron added this to the v1.4.3 milestone Nov 9, 2022
@peusebiu
Copy link
Collaborator

peusebiu commented Nov 9, 2022

let's add a new testCase in TestS3Dedupe https://github.com/project-zot/zot/blob/main/pkg/storage/s3/s3_test.go#L935

Currently TestS3Dedupe it's using boltdb, let's use both boltdb and DynamoDB, this should be a small change.

@rchincha
Copy link
Contributor

Thanks for tracking your PR with task items, makes it easier to follow.

@chofnar chofnar force-pushed the dynamodbReimplementation branch 4 times, most recently from 2c0890c to c9768ff Compare November 16, 2022 13:27
@chofnar chofnar force-pushed the dynamodbReimplementation branch 17 times, most recently from d2ab7c3 to 07d5cea Compare November 22, 2022 14:16
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
@rchincha rchincha merged commit 31b9481 into project-zot:main Nov 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

storage: root directory directly on cloud storage
4 participants