Google Cloud Storage streaming upload from Python generator

Question

I have a Python generator that will yield a large and unknown amount of byte data. I'd like to stream the output to GCS, without buffering to a file on disk first.

While I'm sure this is possible (e.g., I can create a subprocess of gsutil cp - <...> and just write my bytes into its stdin), I'm not sure what's a recommended/supported way and the documentation gives the example of uploading a local file.

How should I do this right?

The magic is to convert your generator into a stream that yields each time a read is performed. The Python example in your reference link demonstrates how to read the stream. This article will help you create a stream backed by a generator: coderscat.com/python-generator-and-yield — John Hanley
– John Hanley, Commented Sep 6, 2022 at 0:18

David · Accepted Answer · 2022-09-09 17:02:14Z

6

The BlobWriter class makes this a bit easier:

bucket = storage_client.bucket('my_bucket')
blob = bucket.blob('my_object')
writer = BlobWriter(blob)

for d in your_generator:
  writer.write(d)

writer.close()

answered Sep 9, 2022 at 17:02

David

9,7811 gold badge22 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Mohamed Noordin Apr 23 at 10:10

So, is each stream write counted as a GCS write operation or is the entire stream counted as a single write operation when we flush or close the stream? There could be millions of writes to the stream which of course will exceed any quotas and that would break the bank, literally.

Collectives™ on Stack Overflow

Google Cloud Storage streaming upload from Python generator

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related