Unable to backup to Google Cloud bucket using S3 backend on restic v0.15.2

[joelnb]

Output of restic version

% restic version
restic 0.15.2 compiled with go1.20.3 on linux/amd6

How did you run restic exactly?

Originally I was trying to backup but I have found it more useful to test with a snapshots command so that it does not add any new snapshots and just reads data from the repo. The issue seems to be communication with the backend so I assume it would affect any command that accesses the repo.

Example snapshots command:

restic -r s3:https://storage.googleapis.com/backup-bucket/restic-repo snapshots

Example backup command:

restic -r s3:https://storage.googleapis.com/backup-bucket/restic-repo backup /some/path

This accesses a restic repo at the path /restic-repo (this is the real path I use) inside the bucket backup-bucket (a made up bucket name).

Before running these commands I am exporting required environment variables with credentials that worked on restic v0.15.1 (and still do if I go back to that version):

export AWS_ACCESS_KEY_ID=GOOG...
export AWS_SECRET_ACCESS_KEY=...

What backend/server/service did you use to store the repository?

Using the S3 backend to connect to a Google Cloud Platform bucket. In the "Steps to reproduce the behavior" I linked a guide which can be used to set this up the same way using the "Interoperability" settings in Google Cloud Storage.

Expected behavior

Correctly list snapshots or do the backup to the bucket as it did in v0.15.1

Actual behavior

The error I get is:

Fatal: unable to open config file: Stat: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
Is there a repository at the following location?
s3:https://storage.googleapis.com/backup-bucket/restic-repo

I am familiar with this error if I forget to export the required environment variables before running but I have carefully checked that is not the case here & running the exact same steps with v0.15.1 works okay.

Steps to reproduce the behavior

As far as I can tell any backup to a GCP bucket using the S3 backend & credentials provided in this way will have the issue. I do not have access to other cloud providers which I can test using the S3 backend so cannot say if only Google Cloud buckets are affected.

If you have a GCP account you can follow this guide to create the access/secret key in the same way as I have done.

Do you have any idea what may have caused this?

I was able to git bisect & find that it seems to have started with 48e065d so I assume it is linked to an updated dependency.

Since then I was able to do a similar bisect inside https://github.com/minio/minio-go and find that it seems to have been caused by minio/minio-go#1785 - if I revert that commit (and one small later change which requires it) I get a working restic binary just as I had in v0.15.1.

I have opened this here rather than on the minio repo because restic is the software I actually use - I am not familiar with minio. I am happy to open a related issue on the minio repo if that would help get this sorted.

Do you have an idea how to solve the issue?

It works for me if I do go mod edit --replace=github.com/minio/minio-go/v7=../../minio/minio-go and then make restic build against a minio repo with 24cdd7fc940230d35edf1a4d4824c0e7b5a02828 and 0bf84b37b1553b5233d14c4d04ce2480b2cf5264 reverted but I do not know of a proper fix as I assume reverting those commits is unlikely to be the real solution going forward.

Did restic help you today? Did it make you happy in any way?

Not today specifically but I have been using restic for > 6 years and I am very happy with it. I have nothing bad to say about the performance and I tend to recommend it to people needing a solid backup solution.

[joelnb]

Just to be fully clear this is what works for me in my testing:

• Restic v0.15.1
• Restic master or v0.15.2 with minio v7.0.47
• Restic master or v0.15.2 with the 2 commits reverted in minio (on minio master)

These do not work:

• Restic v0.15.2
• Restic master with default minio version
• Restic master with minio master

[MichaelEischer]

This might be caused by minio/minio-go#1800 . From the change in minio/minio-go#1785 it looks like the bucket location lookup changed from path to virtual host style. My guess would be that if you set AWS_DEFAULT_REGION to the correct region, then 0.15.2 should also work.

Could you compare the initial requests to GCS in the debug log (by setting DEBUG_LOG=debug.log)?

[joelnb]

Yes that's correct @MichaelEischer it does work with that set (not something I had tried before). Thanks!

Just so I am clear when you say to compare the initial requests in the debug log do you mean to compare between restic v0.15.2 with the variable & without or to compare between 2 restic versions? I should be able to do that but want to be sure I check the right thing.

[MichaelEischer]

Just so I am clear when you say to compare the initial requests in the debug log do you mean to compare between restic v0.15.2 with the variable & without or to compare between 2 restic versions? I should be able to do that but want to be sure I check the right thing.

@joelnb I'm interested in what has changed between restic 0.15.1 and 0.15.2.

[joelnb]

Difference seems to be in the initial HTTP request - the first response in 0.15.2 is a 403. In 0.15.1 it looks like this:

2023/05/21 08:28:50 layout/layout.go:139        layout.ParseLayout      1       parse layout string "" for backend at restic-repo
2023/05/21 08:28:50 layout/layout.go:99 layout.DetectLayout     1       detect layout at restic-repo
2023/05/21 08:28:50 s3/s3.go:200        s3.(*Backend).ReadDir   1       ReadDir(restic-repo/keys)
2023/05/21 08:28:50 s3/s3.go:210        s3.(*Backend).ReadDir   1       using ListObjectsV1(false)
2023/05/21 08:28:50 debug/round_tripper.go:93   debug.loggingRoundTripper.RoundTrip     34      ------------  HTTP REQUEST -----------
GET /backup-bucket/?location= HTTP/1.1
Host: storage.googleapis.com
User-Agent: MinIO (linux; amd64) minio-go/v7.0.47
Authorization: **redacted**
Date: Sun, 21 May 2023 07:28:50 GMT
Accept-Encoding: gzip


2023/05/21 08:28:50 debug/round_tripper.go:110  debug.loggingRoundTripper.RoundTrip     34      ------------  HTTP RESPONSE ----------
HTTP/2.0 200 OK
Content-Length: 91
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private, max-age=0
Content-Type: application/xml; charset=UTF-8
Date: Sun, 21 May 2023 07:28:50 GMT
Expires: Sun, 21 May 2023 07:28:50 GMT
Server: UploadServer
X-Goog-Metageneration: 3
X-Guploader-Uploadid: ADPycdu-MM2b9ppBE-Pvsg473e1zrGuEoV6QRUpUVkcB7AaSM9bcCnQkBevjNHvOI3vmo1wRxKgqUZrdP4H6BtLSghtDvw

While with 0.15.2 it looks like this:

2023/05/21 08:28:26 layout/layout.go:139        layout.ParseLayout      1       parse layout string "" for backend at restic-repo
2023/05/21 08:28:26 layout/layout.go:99 layout.DetectLayout     1       detect layout at restic-repo
2023/05/21 08:28:26 s3/s3.go:200        s3.(*Backend).ReadDir   1       ReadDir(restic-repo/keys)
2023/05/21 08:28:26 s3/s3.go:210        s3.(*Backend).ReadDir   1       using ListObjectsV1(false)
2023/05/21 08:28:26 debug/round_tripper.go:93   debug.loggingRoundTripper.RoundTrip     19      ------------  HTTP REQUEST -----------
GET /?location HTTP/1.1
Host: backup-bucket.storage.googleapis.com
User-Agent: MinIO (linux; amd64) minio-go/v7.0.50
Authorization: **redacted**
Date: Sun, 21 May 2023 07:28:26 GMT
Accept-Encoding: gzip


2023/05/21 08:28:26 debug/round_tripper.go:110  debug.loggingRoundTripper.RoundTrip     19      ------------  HTTP RESPONSE ----------
HTTP/2.0 403 Forbidden
Content-Length: 323
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private, max-age=0
Content-Type: application/xml; charset=UTF-8
Date: Sun, 21 May 2023 07:28:26 GMT
Expires: Sun, 21 May 2023 07:28:26 GMT
Server: UploadServer
X-Guploader-Uploadid: ADPycdvbJBSNxOiS_LDbx3w_uOMYXHwoIj-PqZ2X8gKyhhVY6iNGzDLCEs2BGxXnujnq7GwlLL7l8fc6RDwV0oDg_HJ8NQ

[joelnb]

In case it's helpful this is how it looks in 0.15.2 with the extra AWS_DEFAULT_REGION variable you suggested set (when it is working):

2023/05/21 09:29:41 layout/layout.go:139        layout.ParseLayout      1       parse layout string "" for backend at restic-repo
2023/05/21 09:29:41 layout/layout.go:99 layout.DetectLayout     1       detect layout at restic-repo
2023/05/21 09:29:41 s3/s3.go:200        s3.(*Backend).ReadDir   1       ReadDir(restic-repo/keys)
2023/05/21 09:29:41 s3/s3.go:210        s3.(*Backend).ReadDir   1       using ListObjectsV1(false)
2023/05/21 09:29:41 debug/round_tripper.go:93   debug.loggingRoundTripper.RoundTrip     33      ------------  HTTP REQUEST -----------
GET /?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=restic-repo%2Fkeys%2F HTTP/1.1
Host: backup-bucket.storage.googleapis.com
User-Agent: MinIO (linux; amd64) minio-go/v7.0.50
Authorization: **redacted**
Date: Sun, 21 May 2023 08:29:41 GMT
Accept-Encoding: gzip


2023/05/21 09:29:41 debug/round_tripper.go:110  debug.loggingRoundTripper.RoundTrip     33      ------------  HTTP RESPONSE ----------
HTTP/2.0 200 OK
Content-Length: 293
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private, max-age=0
Content-Type: application/xml; charset=UTF-8
Date: Sun, 21 May 2023 08:29:41 GMT
Expires: Sun, 21 May 2023 08:29:41 GMT
Server: UploadServer
X-Goog-Metageneration: 3
X-Guploader-Uploadid: ADPycdtsgcrnDfChWwb0JNtXc0iF-wM6t-QAep4bSKCC-Z3kIMLayT4ZyVUjc0KDAKsHF5ZyHnEI-R1oVGAbYj5M0iStQQ

[MichaelEischer]

The main difference between the first two debug log snippets is the following (0.15.1)

GET /backup-bucket/?location= HTTP/1.1
Host: storage.googleapis.com

(0.15.2)

GET /?location HTTP/1.1
Host: backup-bucket.storage.googleapis.com

So the culprit indeed seems to be the from path to virtual hosted style. Although I'd expect GCS to support both variants...

[MichaelEischer]

Is this still an issue with restic 0.16.1?

[joelnb]

I'm actually on 0.16.0 but can confirm it works there without the need to set AWS_DEFAULT_REGION at all.

% restic version
restic 0.16.0 compiled with go1.20.7 on linux/amd64

I hadn't noticed this until now because I had changed my backup script to set this after your helpful suggestion.

[MichaelEischer]

Thanks for the reply, then let's close this issue as resolved. Or did I miss something?