Storage: TypeError: request() takes at least 3 arguments (3 given)

[kipi78]

Hi,

I'm having trouble while using google.cloud for Storage. I start by trying to list blobs in a bucket.
I'm trying to get threw the proxy of my organization. So I try to create my own http connection in order to specify the proxy, but even with a simple urllib3.PoolManager(), I keep getting the error and stacktrace bellow:

Error:
TypeError: request() takes at least 3 arguments (3 given)

Stacktrace:

Traceback (most recent call last)
<ipython-input-9-a7bd2d0fcc13> in <module>()
      3 authed_http = AuthorizedHttp(credentials, http)
      4 storage_client = storage.Client(_http=http)
----> 5 bucket = storage_client.get_bucket('mybucket')
      6 blobs = bucket.list_blobs()
      7 

/gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/storage/client.pyc in get_bucket(self, bucket_name)
    171         """
    172         bucket = Bucket(self, name=bucket_name)
--> 173         bucket.reload(client=self)
    174         return bucket
    175 

/gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/storage/_helpers.pyc in reload(self, client)
     97         api_response = client._connection.api_request(
     98             method='GET', path=self.path, query_params=query_params,
---> 99             _target_object=self)
    100         self._set_properties(api_response)
    101 

/gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/_http.pyc in api_request(self, method, path, query_params, data, content_type, headers, api_base_url, api_version, expect_json, _target_object)
    297         response, content = self._make_request(
    298             method=method, url=url, data=data, content_type=content_type,
--> 299             headers=headers, target_object=_target_object)
    300 
    301         if not 200 <= response.status < 300:

/gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/_http.pyc in _make_request(self, method, url, data, content_type, headers, target_object)
    191         headers['User-Agent'] = self.USER_AGENT
    192 
--> 193         return self._do_request(method, url, headers, data, target_object)
    194 
    195     def _do_request(self, method, url, headers, data,

/gpfs/user/u515866/env_python2/lib/python2.7/site-packages/google/cloud/_http.pyc in _do_request(self, method, url, headers, data, target_object)
    221         """
    222         return self.http.request(uri=url, method=method, headers=headers,
--> 223                                  body=data)
    224 
    225     def api_request(self, method, path, query_params=None,

TypeError: request() takes at least 3 arguments (3 given)

---

My code is the following:

from google.oauth2 import service_account
from google.cloud import storage
import urllib3
from google.auth.transport.urllib3 import AuthorizedHttp

json="pathtojson"

credentials = service_account.Credentials.from_service_account_file(json)
http = urllib3.PoolManager()
#authed_http = AuthorizedHttp(credentials, http)
storage_client = storage.Client(_http=http)
bucket = storage_client.get_bucket('mybucket')
blobs = bucket.list_blobs()

for blob in blobs:
    print(blob.name)

Python is 2.7.11.
I'm using Jupyter (notebook server is 5.0.0).

pip freeze:

Click to expand

asn1crypto==0.22.0
backports-abc==0.5
backports.shutil-get-terminal-size==1.0.0
bleach==2.0.0
cachetools==2.0.0
certifi==2017.4.17
cffi==1.10.0
chardet==3.0.4
configparser==3.5.0
cx-Oracle==5.3
decorator==4.0.11
dill==0.2.7.1
entrypoints==0.2.3
enum34==1.1.6
functools32==3.2.3.post2
future==0.16.0
futures==3.1.1
gapic-google-cloud-datastore-v1==0.15.3
gapic-google-cloud-error-reporting-v1beta1==0.15.3
gapic-google-cloud-logging-v2==0.91.3
gapic-google-cloud-pubsub-v1==0.15.4
gapic-google-cloud-spanner-admin-database-v1==0.15.3
gapic-google-cloud-spanner-admin-instance-v1==0.15.3
gapic-google-cloud-spanner-v1==0.15.3
gapic-google-cloud-speech-v1==0.15.3
gapic-google-logging-v2==0.10.1
gapic-google-pubsub-v1==0.10.1
google-api-python-client==1.6.2
google-auth==1.0.1
google-auth-httplib2==0.0.2
google-cloud==0.26.1
google-cloud-bigquery==0.25.0
google-cloud-bigtable==0.25.0
google-cloud-core==0.25.0
google-cloud-datastore==1.1.0
google-cloud-dns==0.25.0
google-cloud-error-reporting==0.25.1
google-cloud-happybase==0.21.0
google-cloud-language==0.25.0
google-cloud-logging==1.1.0
google-cloud-monitoring==0.25.0
google-cloud-pubsub==0.26.0
google-cloud-resource-manager==0.25.0
google-cloud-runtimeconfig==0.25.0
google-cloud-spanner==0.25.0
google-cloud-speech==0.26.0
google-cloud-storage==1.2.0
google-cloud-translate==0.25.0
google-cloud-videointelligence==0.25.0
google-cloud-vision==0.25.0
google-gax==0.15.13
google-resumable-media==0.2.1
googleapis-common-protos==1.5.2
grpc-google-iam-v1==0.11.1
grpc-google-logging-v2==0.10.1
grpc-google-pubsub-v1==0.10.1
grpcio==1.4.0
html5lib==0.999999999
httplib2==0.10.3
idna==2.5
ipaddress==1.0.18
ipykernel==4.6.1
ipython==5.4.1
ipython-genutils==0.2.0
ipywidgets==6.0.0
Jinja2==2.9.6
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.1.0
jupyter-console==5.1.0
jupyter-core==4.3.0
MarkupSafe==1.0
mistune==0.7.4
nbconvert==5.2.1
nbformat==4.3.0
notebook==5.0.0
numpy==1.13.1
oauth2client==3.0.0
oauthclient==1.0.3
pandas==0.20.3
pandocfilters==1.4.1
pathlib2==2.3.0
pexpect==4.2.1
pickleshare==0.7.4
ply==3.8
prompt-toolkit==1.0.14
proto-google-cloud-datastore-v1==0.90.4
proto-google-cloud-error-reporting-v1beta1==0.15.3
proto-google-cloud-logging-v2==0.91.3
proto-google-cloud-pubsub-v1==0.15.4
proto-google-cloud-spanner-admin-database-v1==0.15.3
proto-google-cloud-spanner-admin-instance-v1==0.15.3
proto-google-cloud-spanner-v1==0.15.3
proto-google-cloud-speech-v1==0.15.3
protobuf==3.3.0
ptyprocess==0.5.2
pyasn1==0.2.3
pyasn1-modules==0.0.9
pycparser==2.18
Pygments==2.2.0
pysqlite==2.8.3
python-dateutil==2.6.1
pytz==2017.2
pyzmq==16.0.2
qtconsole==4.3.0
requests==2.18.1
rsa==3.4.2
scandir==1.5
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.10.0
SQLAlchemy==1.1.11
terminado==0.6
testpath==0.3.1
tornado==4.5.1
traitlets==4.3.2
uritemplate==3.0.0
urllib3==1.21.1
wcwidth==0.1.7
webencodings==0.5.1
widgetsnbextension==2.0.0

Please feel free to reformat this issue if that could help.

Many thanks.

[dhermes]

@kipi78 The issue is your transport:

http = urllib3.PoolManager()
#authed_http = AuthorizedHttp(credentials, http)
storage_client = storage.Client(_http=http)

Until #1998 is closed, you must use an httplib2.Http transport. From the docstring

Can be any object that defines request() with the same interface as httplib2.Http.request

---

Is there any reason you passed an explicit _http to the Client constructor? The leading underscore is a sign that _http isn't part of the public API.

[kipi78]

Thank you for your response.

I wrote _http according to this doc:
http://google-cloud-python.readthedocs.io/en/latest/storage/client.html

Okay, I read that it was not an httplib2 object but an urllib3 object, but I tried with a httplib2.Http.
As I said I want to get threw my proxy. Here is my code with both "simple" http transport and one with proxy settings:

httpStorage = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP,proxyUrl, int(proxyPort), True,proxyUser,proxyPassword),ca_certs = pathCertificat)
#httpStorage = httplib2.Http()
authed_http = AuthorizedHttp(credentials, httpStorage)
storage_client = storage.Client(credentials=credentials, _http=httpStorage)

How am I supposed to use that httplib2.Http transport if not passed with the explicit _http arg ?

[dhermes]

It seems to me that you should be passing _http=authed_http.

I am pre-emptively closing this out, it seems you've figure out how to configure against your proxy?

While waiting on #3674 / #1998, you may also want to check out https://github.com/GoogleCloudPlatform/httplib2shim

[dhermes]

Let me know if there are still concerns and we can re-open and discuss if need be.

[kipi78]

No, my comments meant what I tried to simplify the case. I already tried with _http=authed_http which leads to the TypeError: request() takes at least 3 arguments (3 given) error. With the bellow code:

credentials = service_account.Credentials.from_service_account_file(json)
httpStorage = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP,proxyUrl, int(proxyPort), True,proxyUser,proxyPassword),ca_certs = pathCertificat)
authed_http = AuthorizedHttp(credentials, httpStorage)
storage_client = storage.Client(credentials=credentials, _http=authed_http)
bucket = storage_client.get_bucket('mybucket')

I will have a try with httplib2shim, thank you.

[kipi78]

So, with httplib2shim, with the sample code from the doc which uses GoogleCredentials from oauth2client.client, the problem is that google-cloud-python Client only supports credentials from google-auth-library-python. But authorize is not a method of credentials from google-cloud-python Client.

My code:

credentials = GoogleCredentials.get_application_default()
httpStorage = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP,proxyUrl, int(proxyPort), True,proxyUser,proxyPassword),ca_certs = pathCertificat)
httpStorage = credentials.authorize(httpStorage)
storage_client = storage.Client(credentials=credentials, _http=httpStorage)
bucket = storage_client.get_bucket('mybucket')

As I need to use ca_certs, I tried to implement this:

Instead, pass a urllib3.Pool instance http = httplib2shim.Http(pool=my_pool)

with the bellow code:

credentials = service_account.Credentials.from_service_account_file(json)
http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat)
http = httplib2shim.Http(pool=http)
authed_http = AuthorizedHttp(credentials, http)
storage_client = storage.Client(_http=authed_http, credentials=credentials)
bucket = storage_client.get_bucket('mybucket')

Which still leads to the same error, whether I try with a simple PoolManager or with a ProxyManager.

[dhermes]

@jonparrott Can you weigh in here?

[theacodes]

I'm not able to reproduce, here's the code I used:

>>> from google.cloud import storage
>>> import google.auth
>>> import google.auth.transport.urllib3
>>> import urllib3
>>> import httplib2shim
>>>
>>> pool = urllib3.PoolManager()
>>> credentials, project = google.auth.default()
>>> authed_pool = google.auth.transport.urllib3.AuthorizedHttp(credentials, http=pool)
>>> http = httplib2shim.Http(pool=authed_pool)
>>>
>>> client = storage.Client(project=project, _http=http)
>>> list(client.list_buckets())
[<Bucket: ...>, ...]
>>>

I would strongly encourage waiting until #1998 is closed as this will all be significantly easier then.

[kipi78]

Okay, the thing is that you passed an AuthorizedHttp to httplib2shim.Http(), while I passed the ProxyManager and applied the AuthorizedHttp on httplib2shim.Http(). So, I don't have this error anymore with your code :)

The thing is now to use the ProxyManager, it looks like I don't manage to do it. I need to configure a proxy with user/password and a certificate. My code bellow leads to this error:

ProxyError: ('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',))

proxyUrl="http://host:80"
scopes=["https://www.googleapis.com/auth/devstorage.read_write"]
credentials = service_account.Credentials.from_service_account_file(json, scopes=scopes)
default_headers = make_headers(proxy_basic_auth='user:password')
http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat, headers=default_headers)
authed_http = AuthorizedHttp(credentials, http)
http = httplib2shim.Http(pool=authed_http)
storage_client = storage.Client(_http=http, credentials=credentials)
bucket = storage_client.get_bucket('mybucket')

I will certainly wait, many thanks.

[theacodes]

Can you make any requests using the proxymanager alone? Like just manager.get('http://httpbin.org')?

[kipi78]

Yes it looks like the ProxyManager alone is working:

http = ProxyManager(proxy_url=proxyUrl, ca_certs=pathCertificat, headers=default_headers)
a = http.request('GET','http://httpbin.org')
print(a.data)

The expected HTML is printed.

[theacodes]

I wonder if the headers are being overridden. Why happens if you make a request and specify some random headers? http.request('GET','http://httpbin.org', headers={'foo': 'bar'})?

[kipi78]

As soon as I insert an error in the login/password in the headers, this small request does not work. Same thing if I try with random headers. So, I guess the headers are not overridden.