Right now we're using httplib2 because of oauth2client working really nicely with that, however httplib2 doesn't support a streaming download.
This means, a .request() will load the entire response content into memory, which won't work at all for a big file.
To get around this, we chunk the file up using the Range HTTP header and request pieces of it until the end, however the better way to handle this is with a streaming HTTP library:
stream = <send http request>
with open('output.txt', 'wb') as output:
data = stream.read(CHUNK_SIZE)
while data:
output.write(data)
data = stream.read(CHUNK_SIZE)To make this work we need to:
Right now we're using
httplib2because ofoauth2clientworking really nicely with that, howeverhttplib2doesn't support a streaming download.This means, a
.request()will load the entire response content into memory, which won't work at all for a big file.To get around this, we chunk the file up using the Range HTTP header and request pieces of it until the end, however the better way to handle this is with a streaming HTTP library:
To make this work we need to:
oauth2clientto work with that library