Provide a method to convert ndb Key's urlsafe format to/from datastore.Key

[ndenny]

In GAE standard, ndb has the concept of the urlsafe string version of a Key.
e.g. url_string = sandy_key.urlsafe() produces a result like agVoZWxsb3IPCxIHQWNjb3VudBiZiwIM.

However, I cannot find a way to use this format with the google.cloud.datastore library for code running outside GAE standard. Please add a method to the Key class to create urlsafe strings and a method to generate Keys from urlsafe strings. Alternatively, provide some documentation on how to decode/encode them with the existing library.

NB, the console support page says "The encoding method is available in any Cloud Datastore client library."

[lukesneeringer]

Hi @ndenny,
Thanks for reporting. We will look into this.

[dhermes]

For reference, urlsafe just takes the key, turns it into raw bytes as a serialized protobuf, then urlsafe-base64 encodes those raw bytes. However, the protobuf used by App Engine is different than the one used by the Cloud Datastore API (I think).

@jonparrott Can you weigh in?

---

If you'd like a "band-aid" for now:

>>> import base64
>>> from google.cloud import datastore
>>>
>>> client = datastore.Client()
>>> key = client.key('Foo', 'bar')
>>> key_pb_bytes = key.to_protobuf().SerializeToString()
>>> urlsafe = base64.urlsafe_b64encode(key_pb_bytes).rstrip(b'=')

[theacodes]

There's two separate requests here:

1. We need to add Key.urlsafe() and Key.from_urlsafe() that operates with our current proto format.
2. We should make sure the first method accepts App Engine standard's protobuf format as well.

[michaelenglo]

Go to the source and you will find this:

def _DecodeUrlSafe(urlsafe):
  """Decode a url-safe base64-encoded string.
  This returns the decoded string.
  """
  if not isinstance(urlsafe, basestring):
    raise TypeError('urlsafe must be a string; received %r' % urlsafe)
  if isinstance(urlsafe, unicode):
    urlsafe = urlsafe.encode('utf8')
  mod = len(urlsafe) % 4
  if mod:
    urlsafe += '=' * (4 - mod)
  #This is 3-4x faster than urlsafe_b64decode()
  return base64.b64decode(urlsafe.replace('-', '+').replace('_', '/'))

I just copied and pasted this snippet in my code and it works now. Hope this helps.

[dhermes]

@michaelenglo Thanks for providing the snippet, however the issue is that the raw bytes are different for an entity in App Engine and one using the Google Cloud Datastore API.

[michaelenglo]

@dhermes That is true, in fact, When I try to print it to the console, it gives me non-textual characters. I prematurely said it worked because I didn't get an exception for the first time when running it lol (been stuck on this for 2 hours now).

By any chance, have we found any solution to this already?

[dhermes]

@michaelenglo Feel free to provide some code. The issue though is that the two different APIs (the native RPC API in App Engine and the external Cloud Datastore API) use different protobuf definitions for entities, keys, etc.

I'm pretty sure the GAE protos are in google.appengine.datastore.entity_pb (I'm verifying right now) and the Cloud Datastore protos are here.

[dhermes]

Here is a confirmation of the difference

>>> import base64
>>> from google.appengine.datastore import entity_pb
>>> from google.appengine.ext import ndb
>>>
>>> key = ndb.Key('MyModel', 5910974510923776)
>>> urlsafe = key.urlsafe()
>>> urlsafe
'ahNkZXZ-c3R1ZHlidWRkeXgtaHJkchQLEgdNeU1vZGVsGICAgICAgMAKDA'
>>> urlsafe += '=='  # Needs padding
>>> raw_bytes = base64.urlsafe_b64decode(urlsafe)
>>> raw_bytes
'j\x13dev~studybuddyx-hrdr\x14\x0b\x12\x07MyModel\x18\x80\x80\x80\x80\x80\x80\xc0\n\x0c'
>>>
>>> ref = entity_pb.Reference(raw_bytes)
>>> print(ref)
app: "dev~studybuddyx-hrd"
path <
  Element {
    type: "MyModel"
    id: 0x15000000000000
  }
>

vs.

>>> from google.cloud import datastore
>>> client = datastore.Client(project='studybuddyx-hrd')
>>>
>>> key = client.key('MyModel', 5910974510923776)
>>>
>>> key_pb = key.to_protobuf()
>>> key_pb
partition_id {
  project_id: "studybuddyx-hrd"
}
path {
  kind: "MyModel"
  id: 5910974510923776
}
>>>
>>> key_pb_bytes = key_pb.SerializeToString()
>>> key_pb_bytes
'\n\x11\x12\x0fstudybuddyx-hrd\x12\x12\n\x07MyModel\x10\x80\x80\x80\x80\x80\x80\xc0\n'

[michaelenglo]

>>> print(ref)
app: "dev~studybuddyx-hrd"
path <
  Element {
    type: "MyModel"
    id: 0x15000000000000
  }
>

What kind of data format is this? Is this something that has a publicly available parser?

My only goal here is just to extract the ID from the key. And by the way, I don't understand why the data formatting must be different between AE Standard Env. API and the external GC client library?

[dhermes]

What kind of data format is this?

This is just a pretty-printed representation of an entity_pb.Reference object, which is a Python wrapper for a protobuf.

Is this something that has a publicly available parser?

The ref instance is already parsed. I recommend firing up the dev_appserver and then using the Interactive Console to play around.

My only goal here is just to extract the ID from the key.

The rules for deserializing a protobuf are public (i.e. not a Google secret), though I'm not sure what they are.

And by the way, I don't understand why the data formatting must be different between AE Standard Env. API and the external GC client library?

I can't help with that. Our library is just a wrapper for the existing cloud APIs. As a historical note, the Datastore within App Engine has existed since 2008, but the Cloud Datastore API came along in 2012 (and the current revision came along in 2015 or 2016).

[michaelenglo]

Okay, so I get that. Now the question will be how to actually convert the raw binary data after it has been decoded?

When I called the _DecodeUrlSafe(urlsafe) when urlsafe="agxzfm9lLXNlcnZpY2VyGwsSDkRyYXdpbmdWZXJzaW9uGICAgID4woQKDA", and print the returned value, I get this:

js~oe-servicer����DrawingVersion�ÇÇÇÇ°┬ä

How to convert this value into a protobuf? Is there an API call for this in the Client Library?

If this is possible, then I can call google.cloud.datastore.helpers.entity_from_protobuf(pb) to get the entity.

[dhermes]

You'll want to print the repr of the value. That already is a protobuf (as raw bytes). See my snippet above:

from google.appengine.datastore import entity_pb
ref = entity_pb.Reference(raw_bytes)