Google Storage parallel upload

[pditommaso]

Does the Google Storage Java SDK provides an API for parallel multi-parts upload? Is there any example out there?

[garrettjonesgoogle]

I will take a look but I won't be able to get to this immediately.

[pditommaso]

Any update on this? Tx

[garrettjonesgoogle]

@lesv or @tswast do either of you know?

[tswast]

I don't know.

Only sample for uploading I'm aware of is https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/google-cloud-examples/src/main/java/com/google/cloud/examples/storage/snippets/BucketSnippets.java#L227 linked from https://cloud.google.com/storage/docs/object-basics#storage-upload-object-java which doesn't explicitly do a multi-part upload.

[pditommaso]

Plain upload it's quite easy. The documentation suggests that parallel upload can be implemented just splitting the original object in chunks and upload them in parallel, then compose all the chunks in a single object.

It's not clear how to perform the compose operation by using the Java API.

[lesv]

Looking at Storage.writer, it doesn't appear that there is a way to get more than one WriteChannel. But I didn't work on that API, @tswast might know more.

[tswast]

We don't have any code samples at https://cloud.google.com/storage/docs/composite-objects so my guess is the client libraries don't support it in any language. I'll follow up with Omar.

[lesv]

@tswast This seems like it should be a FR, rather than block GA -- thoughts?

[tswast]

Yeah, definitely a new feature for the client libraries to support the compose operation. I don't know enough about how people tend to use Cloud Storage to know if it should be a GA blocker.

Seems like it would be a useful feature for large files.