Trouble with TextProperties

[yjkogan]

My use case for the datastore api involves creating entities with at least one ndb.TextProperty. These sometimes are more than 500 characters, so if I just stick it on the Entity as a property, the save fails. I've read a bunch of the docs (including for the datastore JSON api) and can't find any indication for how to specify that something should be a text property rather than a string property (as indicated by the datastore). Is this something that's supported? If so, any help for how to do it would be appreciated!

[dhermes]

@yjkogan It's great to have you using the library! It looks like you're using bytes instead of string and that you're having them indexed=True. Could you provide some example code that is failing?

If you are in fact using bytes, use unicode instead of str in Python 2. See the note in the code which explains the difference and see the page in the docs where this not is rendered (starting with "When saving an entity to the backend...").

[yjkogan]

Thanks for the fast response @dhermes! I wrote up a response below, and then solved the issue, but I'm including it in case it's helpful for improving the documentation. The following code worked for me:

some_text_property = "THIS STRING IS LONGER THAN 500 CHARACTERS"
gae_key = key.Key('MyClass')
gae_entity = entity.Entity(key=gae_key, exclude_from_indexes=['some_text_property'])
gae_entity['some_text_property'] = some_text_property.encode('utf-8')
gae_entity.save()

This is what I wrote up before solving the issue:

So here's the property's declaration on the model:

class MyClass(ndb.Model):
  # some properties
  some_text_property = ndb.TextProperty()
  # more properties and methods

And here's the code I've tried, with accompanying tracebacks:

some_text_property = "THIS STRING IS LONGER THAN 500 CHARACTERS"
gae_key = key.Key('MyClass')
gae_entity = entity.Entity(key=gae_key)
gae_entity['some_text_property'] = some_text_property.encode('utf-8')
gae_entity.save()

which gives

Traceback (most recent call last):
  File "~/gcloud_populate_datastore.py", line 154, in <module>
    gae_entity.save()
  File "~/gcloud/datastore/entity.py", line 144, in save
    exclude_from_indexes=self.exclude_from_indexes)
  File "~/gcloud/datastore/connection.py", line 495, in save_entity
    result = self.commit(dataset_id, mutation)
  File "~/gcloud/datastore/connection.py", line 376, in commit
    datastore_pb.CommitResponse)
  File "~/gcloud/datastore/connection.py", line 100, in _rpc
    data=request_pb.SerializeToString())
  File "~/gcloud/datastore/connection.py", line 77, in _request
    raise six.moves.http_client.HTTPException(message)
httplib.HTTPException: Request failed with status code 400. Error was: Indexed blob value has more than permitted 500 bytes.

I get the same stack trace when I instead do the following:

some_text_property = "THIS STRING IS LONGER THAN 500 CHARACTERS"
gae_key = key.Key('MyClass')
gae_entity = entity.Entity(key=gae_key)
gae_entity['some_text_property'] = str.encode(some_text_property)
gae_entity.save()

[dhermes]

Great!

As I mentioned before you would have been better off doing

some_text_property = u"THIS STRING IS LONGER THAN 500 CHARACTERS"

then the entity would have stored it as a string and not bytes. (Did you read the notes I referred to?)

[yjkogan]

I did read the note, and saw it last night when I was fighting with this as well. I'm actually parsing the string from JSON and passing it as unicode to the Entity. When I do that, I get the following error instead:

bash
httplib.HTTPException: Request failed with status code 400. Error was: The string property some_text_property has a value that is too long. It cannot exceed 500 characters.

[yjkogan]

Thanks for the help resolving this issue!

[dhermes]

Yes, I forgot that neither string nor bytes could be indexed beyond 500 characters. I added a note about this in #532 since the library could decide to make the property non-indexed.

The relevant proto [documentation] says:

When indexed is true, stringValue is limited to 500 characters and the blob value
is limited to 500 bytes.

I'm not sure where it is in the actual datastore docs, but there might be a link in #532.

[yjkogan]

Awesome, thank you again for all the help!

[dhermes]

👍