GeoPt not honoured

[thinrhino]

We have a record in the datastore which has the object type ndb.GeoPtProperty. This record was set via ndb from the app engine. Now we are running a script on GCE to modify one field in this record, without touching the original location point.

When the record is saved from gCloud API, the record looses it's GeoPtProperty.

Before:

> record.location
datastore_types.GeoPt(19.179167, 72.9640237)

After:

> record.location
>

[dhermes]

/cc @pcostell

[dhermes]

@thinrhino Here is what I've just done:

from google.appengine.ext import ndb

class Foo(ndb.Model):
    bar = ndb.GeoPtProperty()

g = ndb.GeoPt(10.0, 10.0)
f = Foo(bar=g)
f.put()
print(f)

which created key=Key('Foo', 5634472569470976).

Then in gcloud-python:

from gcloud import datastore

client = datastore.Client()
key = client.key('Foo', 5634472569470976)
entity = client.get(key)
print(entity)

which is

<Entity[{'kind': u'Foo', 'id': 5634472569470976L}] {u'bar': <Entity {u'y': 10.0, u'x': 10.0}>}>

So the data is still there, but I'm not sure the geopt part can be set from the Cloud Datastore API (I am fairly certain it can't).

[jgeewax]

I believe this is doing what it should -- as GeoPt isn't something supported by the JSON API, but instead is an abstraction in DB / NDB

• GeoPt data type: https://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/api/datastore_types.py?r=503#877
• GeoPtProperty in NDB: https://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/ndb/model.py?r=503#1748

So under the hood, it looks like NDB is taking a GeoPt and turning into a pair of coordinates before saving via the API -- and then loading it into that property on the way out of the API as well.

If we get NDB running on top of the Cloud Datastore API this problem should go away as you'd just use NDB the whole time (see to #557).

[dhermes]

Actually stepping into Connection.lookup and checking out the protobuf response, it seems the "meaning" is set on the GeoPt property:

found {
  entity {
    key {
      partition_id {
        dataset_id: "s~FOO"
      }
      path_element {
        kind: "Foo"
        id: 5634472569470976
      }
    }
    property {
      name: "bar"
      value {
        entity_value {
          property {
            name: "x"
            value {
              double_value: 10.0
              indexed: false
            }
          }
          property {
            name: "y"
            value {
              double_value: 10.0
              indexed: false
            }
          }
        }
        meaning: 9
      }
    }
  }
}

The only real mention of "meaning" in the .proto file is

An entity value with meaning 9, 20 or 21 may be indexed.

so maybe we need to add support for setting "meaning" in these cases.

[dhermes]

@jgeewax Thanks for hunting down the source, ndb actually sets the meaning:

p.set_meaning(entity_pb.Property.GEORSS_POINT)

and the list of meanings is defined in google.appengine.datastore.entity_pb. It looks like 20 and 21 are no longer meanings, but maybe they were at some point (I checked r525, r500, r450, r400 and r300 and didn't see any mention of them)?

[jgeewax]

Yea we definitely need @pcostell to chime in on this -- I have no idea what the meaning value might mean in a broader sense. Hopefully we can use that..

Hypothetically, if that did exist, do you think gcloud.datastore should have a GeoPt data type?

[dhermes]

@jgeewax It wouldn't be that hard to do. We discussed / dismissed implementing meaning in #265 and that was the end of it (AFAICT).

[pcostell]

v1beta2 doesn't have a specific geopt field, which is why it is represented using this "legacy-supported" type (an entity_value with a meaning). However, v1beta3 does have a specific geopt which will be populated instead.

In general, fields with the meaning should always be preserved, however we should never be creating new fields with meanings set (which is why in the API they are not actually explained). You should still be able to edit this field in the same way you can edit an entity_value.

[dhermes]

@pcostell When can we start using v1beta3? What should we do about this in the meantime?

[pcostell]

"Soon". This should still work in v1beta2, as the property should still be editable as a normal entity_value. You should never be setting the meaning, but you should be persisting an existing meaning. So if users can edit an entity value, this should work.

[dhermes]

Is it worth making a change to preserve to meaning if we are just going to ditch meanings all together "soon"? (It really depends on your judgment of how soon.) We just did a release 8/12 so even if we made a fix it may not be in a release for another 14-21 days .

[pcostell]

Yes, we should always be preserving the meaning. There are other legacy properties that will still require meaning even after the v1beta3 launch.

[dhermes]

OK great.

Secondary question, what are the mysterious enums at 20 and 21 for meaning?

An entity value with meaning 9, 20 or 21 may be indexed.

[pcostell]

Secret :-). We don't want clients of the gcloud libraries actually depending on meaning value (they should only be persisting them).

At the moment (but we reserve the right to change them): 9 is a GeoPt which is exists in the geo-range (lat/lng), 21 is an "invalid" geopoint (any range, so really just a point), and 20 represents an App Engine UserProperty.