The issue encountered when uploading large size file to Pikpak.

[cj-neo]

What is the problem you are having with rclone?
I encountered a few days ago while uploading a large file. The file is 134GB, but I encountered the following error before even reaching 100GB:

2024/05/08 20:56:55 ERROR : /Marvels.Avengers.zip: Failed to copy: failed to upload: MultipartUpload: upload multipart failed
        upload id: C278C4F1FAD2486782772961F3CA663A
caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit

It mentions that adjusting the PartSize parameter can allow the upload. Is the corresponding parameter max-upload-parts? However, I couldn't find this parameter in Pikpak. Currently, I only see two specific parameters for cloud storage services: --oos-max-upload-parts and --s3-max-upload-parts, but neither seems applicable to Pikpak.

I'm not sure what the maximum file size limit for Pikpak is, but currently, uploading large files to Pikpak using rclone seems to be problematic.

If possible, I think rclone could handle large file uploads better. For instance, it could first check if the file size fits within the default PartSize before starting the upload. If it doesn't fit, the upload shouldn't start because uploading large files takes a lot of time. Encountering a PartSize error halfway through, requiring a settings adjustment and re-upload, wastes all the previous time.

Additionally, maybe rclone could automatically adjust the PartSize based on the file size, completing the upload without any errors. I think it's challenging for the average user to know how large the file needs to be before adjusting the PartSize for a successful upload, Of course, these are just my humble suggestions and might be a bit demanding. I'm just offering some possible directions for improvement.

Thank you for your help.

What is your rclone version (output from rclone version)
1.66

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)
Win10 & Ubuntu 22.04, 64 bit

Which cloud storage system are you using? (e.g. Google Drive)
local or google drive upload to pikpak

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)
rclone copy & sync

How to use GitHub
Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

[ncw]

caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit

We don't currently have a way of adjusting this.

However we do have an internal library which can calculate part sizes for you

rclone/fs/chunksize/chunksize.go

Line 20 in 96e39ea

func Calculator(o interface{}, size int64, maxParts int, defaultChunkSize fs.SizeSuffix) fs.SizeSuffix {

This needs to go in about here

rclone/backend/pikpak/pikpak.go

Line 1170 in 96e39ea

// Perform upload with options different than the those in the Uploader.

@wiserain have you got time to take a look at this?

[wiserain]

@wiserain have you got time to take a look at this?

Sure I will make a PR for this after #7852 merged.