Batch dataset.put and dataset.delete.

[silvolu]

Like dataset.get, there should be a dataset.delete and dataset.put that accepts a list of keys / entities.

[dhermes]

I need to get on #336 too and hopefully these can come together. We should actually hash out the discussion in #188 as well, since many of our core patterns are very JavaScript-like and may not be the right way forward.

[tseaver]

Given a list of entities in hand, what is the advantage of a "convenience" method on the dataset to save or delete them, against just iterating the list and calling save or delete on each one? The get_entity and
get_entities methods on a dataset are there for when you don't already have the entities.

I guess there might be some utility to a delete method taking only keys, so that the user doesn't need to either fetch them via get_entities or convert them to protobufs to hand to the connection's delete_entities method.

[dhermes]

I think dataset.delete and dataset.put are valid, somewhat like ndb.put_multi and ndb.delete_multi.

[tseaver]

The Zen of Python says:

There should be one– and preferably only one –obvious way to do it.

What use is Dataset.put()? To make it work you have to have the entity objects in hand already, so why not just:

for entity in entities_to_put:
    entity.save()

If Dataset.delete() takes entities, the same applies (entities already have a delete() method). If it takes keys (only), then it might be useful, if you can reasonably expect that there is a way to have a bunch of keys in hand without also having the corresponding entities.

[silvolu]

It's mostly for performances reason. Doing:

for entity in entities_to_put:
    entity.save()

Means one RPC for each save. Batching means one RPC for all saves. (and same for deletes).

[tseaver]

Isn't that what you get from using a transaction? All the adds / saves / deletes get sent as a single RPC.

[silvolu]

That's meant for a transactional context, where if one fails, they all fail.

[elibixby]

I think it makes sense to have this be a mode/flag/subclass of Dataset rather than extra methods. 1. it keeps the interface of the class cleaner, 2. it adds support for batch mutations which contain a mix of mutation types. Ideally syntax could be similar to transaction syntax.

with dataset.batch_mutation() as b:
    for entity in entities_to_put:
        b.put(entity)
    for key in keys_to_delete:
        b.delete(key)

[dhermes]

IMO we need to focus on feature parity with the API before we can add nice sugar like that.

As @silvolu says, we need methods for batch puts and deletes.

[tseaver]

What is the expectation for error propagation after a non-transactional batch update / delete, where some of the operations failed?

[elibixby]

@tseaver There is none. If guarantees need to be made about mutation success transactions must be used.

@dhermes It's not necessarily sugar. If instead of immediately calling connection.commit() dataset writes to a mutation_pb buffer and immediately flushes that buffer (by calling connection.commit()) then a batch_mutator class could override that behavior and only flush the buffer on context exit. I think there is a lot of user experience value in having the same method signatures in a different context for batch mutations.

This implementation also works quite nicely for transactions. I have a class hierarchy I made for the Entityless implementation of the library which could easily be adapted, if you would be interested.

[tseaver]

@elibixby are you saying there is no way to detect the failures in batch mode? Who would use an API like that, and why would we go out of our way to support them?