Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: determine what to do on chunksize=0 [GRPC] #7798

Closed
BrennaEpp opened this issue Apr 20, 2023 · 1 comment · Fixed by #8348
Closed

storage: determine what to do on chunksize=0 [GRPC] #7798

BrennaEpp opened this issue Apr 20, 2023 · 1 comment · Fixed by #8348
Assignees
@tritone
Labels
api: storage Issues related to the Cloud Storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: process A process-related concern. May include testing, release, or the like.

Comments

@BrennaEpp
Copy link
Contributor

BrennaEpp commented Apr 20, 2023 •
edited

Address TODOs around L1472 in grpc_client.go

Should we actually use the minimum of 256 KB here [chunksize=0] when the user indicates they want minimal memory usage? We cannot do a zero-copy, bufferless upload like HTTP/JSON can.

Must make a note to update documentation accordingly.
@BrennaEpp BrennaEpp added the triage me I really want to be triaged. label Apr 20, 2023
@product-auto-label product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Apr 20, 2023
@BrennaEpp BrennaEpp added priority: p2 Moderately-important priority. Fix may not be included in next release. type: process A process-related concern. May include testing, release, or the like. and removed api: storage Issues related to the Cloud Storage API. triage me I really want to be triaged. labels Apr 20, 2023
@BrennaEpp BrennaEpp mentioned this issue Apr 20, 2023
Merged
@product-auto-label product-auto-label bot added the api: storage Issues related to the Cloud Storage API. label Apr 20, 2023
BrennaEpp added a commit that referenced this issue May 1, 2023
@BrennaEpp
chore(storage): round up Chunksize to nearest multiple of 256kib [GRP…
ca738ab 
…C] (#7799)

Fixes #7754

GRPC skipped due to #7798 affecting the last test chunksize 0 uploads everything

Also cleans up a typo in ProgressFunc docs
tritone added a commit to tritone/google-cloud-go that referenced this issue May 2, 2023
@tritone
chore(storage): fix progress call in gRPC Writer
01c0197 
Fixes googleapis#7798
@tritone tritone mentioned this issue May 2, 2023
Merged
tritone added a commit to tritone/google-cloud-go that referenced this issue Jul 28, 2023
@tritone
feat(storage): support single-shot uploads in gRPC
68abdd7 
When ChunkSize is zero, use a single shot upload (leaving the
stream open between messages) rather than a resumable session.
This matches better with what we do in JSON. Also reduce the
buffer size to the minimum (256k).

Fixes googleapis#7798
@tritone tritone mentioned this issue Jul 28, 2023
Merged
gcf-merge-on-green bot pushed a commit that referenced this issue Aug 1, 2023
@tritone
feat(storage): support single-shot uploads in gRPC (#8348)
7de4a7d 
When ChunkSize is zero, use a single shot upload (leaving the stream open between messages) rather than a resumable session. This matches better with what we do in JSON. Also reduce the buffer size to the minimum (256k).

Fixes #7798
@tritone
Copy link
Contributor

tritone commented Aug 1, 2023

Changed the chunkSize=0 logic to do the following:

  1. Use a single-shot upload rather than resumable
  2. Allocate 256k for the buffer (which is the min possible)

@release-please release-please bot mentioned this issue Aug 1, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tritone tritone

Labels
api: storage Issues related to the Cloud Storage API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: process A process-related concern. May include testing, release, or the like.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(storage): support single-shot uploads in gRPC tritone/google-cloud-go
2 participants
@tritone @BrennaEpp