PubSub topic.create method should raise a Conflict error instead of RetryError/GaxError when topic exists

[lucmult]

Following the documentation from PubSub site if you run the example 2x it will generate the error:

GaxError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.ALREADY_EXISTS, Resource already exists in the project (resource=my-new-topic).)>) <class 'google.gax.errors.RetryError'>

However it should raise Conflict Error (google.cloud.exceptions.Conflict), because the API returns a 409 response.
You can see the status code using API Explorer:
https://developers.google.com/apis-explorer/#search/pubsub/pubsub/v1/pubsub.projects.topics.create

409  
{
 "error": {
  "code": 409,
  "message": "Resource already exists in the project (resource=my-new-topic).",
  "status": "ALREADY_EXISTS",
...
  }
}

Full traceback on Python:

>>> topic.create()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/google/home/lucmult/.virtualenvs/pubsub/local/lib/python2.7/site-packages/google/cloud/pubsub/topic.py", line 155, in create
    api.topic_create(topic_path=self.full_name)
  File "/usr/local/google/home/lucmult/.virtualenvs/pubsub/local/lib/python2.7/site-packages/google/cloud/pubsub/_gax.py", line 104, in topic_create
    topic_pb = self._gax_api.create_topic(topic_path)
  File "/usr/local/google/home/lucmult/.virtualenvs/pubsub/local/lib/python2.7/site-packages/google/cloud/gapic/pubsub/v1/publisher_client.py", line 278, in create_topic
    return self._create_topic(request, options)
  File "/usr/local/google/home/lucmult/.virtualenvs/pubsub/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 419, in inner
    return api_caller(api_call, this_settings, request)
  File "/usr/local/google/home/lucmult/.virtualenvs/pubsub/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 407, in base_caller
    return api_call(*args)
  File "/usr/local/google/home/lucmult/.virtualenvs/pubsub/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 368, in inner
    return a_func(*args, **kwargs)
  File "/usr/local/google/home/lucmult/.virtualenvs/pubsub/local/lib/python2.7/site-packages/google/gax/retry.py", line 126, in inner
    ' classified as transient', exception)
google.gax.errors.RetryError: GaxError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.ALREADY_EXISTS, Resource already exists in the project (resource=my-new-topic).)>)

[dhermes]

@lukesneeringer This is part of a more general question: how / where should we re-wrap GaxError?

[lucmult]

For anybody affected by this, to handle ALREADY_EXISTS exception you can use the following scheme:

try:
  # create topic
except Conflict as error:
   # handle the current conflict exception
except RetryException as error:
   status_code = error.cause.code()
   if status_code == status_code.ALREADY_EXISTS:
      # handle the already exists
   else:
      # nothing that we want just re-raise the exception
      raise

[lukesneeringer]

@lukesneeringer This is part of a more general question: how / where should we re-wrap GaxError?

The correct answer is to have GAX raise these different kinds of errors, rather than the catch-all GaxError. It could import them from google.cloud.exceptions.

[dhermes]

@lukesneeringer That would be much preferred to the current ad-hoc approach (only in use in one subpackage, though some other remappings also exist).

[marcboissonneaultjda]

@dhermes While your team is fixing the issue, can you to fix the subscription_create issue at the same time?
It's exactly the same issue but for subscription and I completely forgot to mention it in my support case with @lucmult

[tseaver]

@lukesneeringer Generalizing the solution from datastore requires moving datastore._gax._catch_remap_gax_error into core, most obviously core.exceptions. Doing that requires adding a from google.cloud.core to google.gax. We have unconditional dependencies to google.gax in other packages: would that be a problem for core?

[tseaver]

The issue within pubsub seems to be that the back-end API shifted from returning FAILED_PRECONDITION to ALREADY_EXISTS: we have handling which raises Conflict for the older code.

We should add a system test (or extend and existing one) to ensure that we don't regress whatever fix we apply now.