Use / recommend use of data-dir for datastore emulator

[dhermes]

@pcostell moving discussion over here. I wanted to discuss two things:

1. How can I incorporate the indexes into runs of the emulator?
2. Do you think we should add the emulator to our automated testing?

[dhermes]

When I tried to run it as-is with the current location of index.yaml I got an error:

$ gcloud beta emulators datastore start --data-dir=gcloud-python/system_tests/data
...
[datastore] Invalid project directory: .../gcloud-python/system_tests/data/.
    Did you create it using 'create'?
...

when doing the Java specific thing you suggested

$ mkdir gcloud-python/system_tests/data/WEB-INF
$ cp gcloud-python/system_tests/data/index.yaml \
>   gcloud-python/system_tests/data/WEB-INF

I ended up with the same error

[pcostell]

1. For testing, I would just have your static yaml of indexes that you use in production and in your emulator tests and keep it checked in to github. Within your <data-dir>/WEB-INF there is an index.yaml file where you can put the indexes.
2. Yes I would include it assuming it isn't too much work.

Hmm, it should do that for you automatically:

~ % gcloud beta emulators datastore start --data-dir=~/tmp/emdir

Results in:

Executing: /bin/bash /Users/pcostello/.local/google-cloud-sdk/platform/gcd/gcd.sh create -project_id=pcostello-cloud /tmp/emdir
[datastore] Created new Cloud Datastore project in '/Users/pcostello//tmp/emdir' with project ID 'pcostello-cloud'.
Executing: /bin/bash /Users/pcostello/.local/google-cloud-sdk/platform/gcd/gcd.sh start --host=localhost --port=8863 --store_on_disk=True --consistency=0.9 --allow_remote_shutdown ~/tmp/emdir
...

[dhermes]

So you're saying I should run the emulator with an empty temp directory and then manually copy the index file into <data-dir>/WEB-INF?

[pcostell]

ya probably. We should probably make it easier to get this setup specifically for indexes -- maybe a different flag?

[dhermes]

Yes I agree. It's especially painful since start just launches in the foreground.

[pcostell]

@eddavisson thoughts here? One of the reasons to require the directory is so that we can then upload the index.yaml file. But if we include index.yaml as a specific flag, we could change update/vacuum indexes to also just accept the index.yaml file.

Since we've moved away from xml, there is now only one file to deal with.

[eddavisson]

The local_db.bin file also ends up in the data directory.

When we were developing the Cloud SDK CLI, we tried to target two use cases:

1. Unit testing. Neither the data nor the required indexes are persisted on disk.
2. Interactive testing. Data and required indexes are persisted on disk. Resulting index.yaml file is suitable for update/vacuum commands.

Does 1) capture your use case (or are you tracking index usage across tests)?

[pcostell]

This isn't quite handled for unit testing in the case you want to do emulator + prod integration tests. In this case not persisting the indexes caused issues because prod used one index and the emulator generated a different index.

[dhermes]

@pcostell The above is mostly about how to run with our own indexes, which seems like a work in progress?

What about my second question? Do you think we should run the emulator tests on a regular basis or is hitting the live API on every merge to master sufficient?

[pcostell]

I don't think you need to run them as part of the merge -- the live API is probably better. However, integrating them with your normal testing infrastructure can make normal development iteration a lot easier.

[dhermes]

Good call. I'm doing to add an optional tox environment (e.g. we run tox -e docs-rtd if we want a preview of http://gcloud-python.readthedocs.org/ which is different than our "main" docs http://googlecloudplatform.github.io/gcloud-python/stable/ which runs as tox -e docs)