6

The gsutil command has options to optimize upload/download speed for large files. For example

GSUtil:parallel_composite_upload_threshold=150M
GSUtil:sliced_object_download_max_components=8

see this page for reference.

What is the equivalence in the google.cloud.storage python API? I didn't find the relevant parameters in this document.

In general, does the client API and gsutil have one to one correspondence in terms of functionalities?

2 Answers 2

9

I think it's not natively supported.

However (!) if you're willing to decompose files then use threading or multiprocessing, there is a compose method that should help you assemble the parts into one GCS object.

Ironically, gsutil is written in Python but it uses a library gslib to implement parallel uploads. You may be able to use gslib as a template.

Sign up to request clarification or add additional context in comments.

3 Comments

Here's an open issue to support parallel operations for copying objects.
@DonnaldCucharo the issue you linked to is for copying objects in parallel, while composite upload is about chunking large files to speed up the operations using composite objects (cloud.google.com/storage/docs/composite-objects).
I opened an issue to add support for parallel composite upload: github.com/googleapis/python-storage/issues/1202
0

Check out the transfer_manager module of the Python Client for Google Cloud Storage or in this sample code. It has methods for file up/download where you pass it the object to copy, set the chunk size and number of workers, and the module takes care of the rest.

It uses the Multipart XML API which allows even higher parallelization than gcloud storage of gsutil, and avoids potential bucket feature interop issues too. I wrote about this and more in my recent blog post "High throughput file transfers with Google Cloud Storage (GCS).

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.