with Batch() as batch: clause throws AttributeError regarding Connection object

[charlesbclifford]

Just installed gcloud-python this past Tuesday, and this is my 1st time using it. Shown below is the code that is throwing an AttributeError regarding the Connection object. The code is about a method which builds a batch of new entities which are intended to loaded into Datastore.

After the last batch.put(_ENTITY) statement executes successfully, and the with datastore.batch.Batch(connection=connection) as batch: clause completes, this error is thrown

Load_Datastore() Unexpected error: <type 'exceptions.AttributeError'>
AttributeError: 'Connection' object has no attribute 'commit'

I'm not sure why the reference to the commit attribute is made relative to the Connection object, and not to the Batch object.

Here's the abbreviated method call that throws the AttributeError regarding the Connection object:

from gcloud import storage
storage.set_default_bucket('')
storage.set_default_project('aaaaaaa-bbbbbb-33333')
from gcloud import datastore
datastore.set_default_dataset_id('xxx_xxx')
from gcloud.datastore.batch import Batch

def Load_Datastore():
    try:
        Get_Bucket_Metadata()
        #set configuration variables       
        connection = storage.get_connection()
        storage.set_default_connection(connection)
        datastore.set_default_connection(connection)

                    if not _RECORDS:
                        print "_RECORDS is empty"
                    else:
                        print "there are %s records to write into Datastore"%(str(len(_RECORDS)))
                        #start transaction with Datastore 
                        with datastore.batch.Batch(connection=connection) as batch:
                            _ENTITIES=[]                                           
                            for _RECORD in _RECORDS:
                                _PROPERTIES=[]

                                if _RECORD:
                                    for _KEY_VALUES in _RECORD:
                                        _KEY_VALUE_PAIR=_KEY_VALUES.split('|')
                                        if str(_KEY_VALUE_PAIR[0])=='ROW_KEY':
                                            _ENTITY_KEY=str(_KEY_VALUE_PAIR[1])
                                        else:
                                            _PROPERTIES.append("%s|%s"%(str(_KEY_VALUE_PAIR[0]),str(_KEY_VALUE_PAIR[1])))

                                #an empty list is False
                                if not _PROPERTIES:   
                                    print "_PROPERTIES is empty"
                                else:   
                                    try:
                                        key = datastore.Key('CI_ADJ_TYPE',_ENTITY_KEY,namespace='xxx',dataset_id='xxx_yyy')  
                                        try:
                                           DS_ENTITY = datastore.Entity(key)
                                            for _PROPERTY in _PROPERTIES:
                                                _NAME_VALUE=_PROPERTY.split('|')
                                                _COLUMN_NAME=str(_NAME_VALUE[0])
                                                _COLUMN_VALUE=str(_NAME_VALUE[1])
                                                DS_ENTITY.update({_COLUMN_NAME:_COLUMN_VALUE})
                                            _ENTITIES.append(DS_ENTITY)                                 
                                        except Exception, inst:
                                            print "datastore.Entity(%s) Unexpected error: %s"%(str(_ENTITY_KEY),str(sys.exc_info()[0]))
                                            if isinstance(inst,ValueError):
                                                print "ValueError: %s"%(inst.message)
                                            if isinstance(inst,AttributeError):
                                                print "AttributeError: %s"%(inst.message)
                                    except Exception, inst:
                                        print 'datastore.Key(CI_ADJ_TYPE,key_index,namespace=xxx,dataset_id=xxx_yyy) Unexpected error: ' + str(sys.exc_info()[0])
                                        if isinstance(inst,ValueError):
                                            print "ValueError: %s"%(inst.message)
                                        if isinstance(inst,AttributeError):
                                            print "AttributeError: %s"%(inst.message)
                                        #print 'datastore.Key() Exception instance type: ' + str(type(inst))                      
                            for _ENTITY in _ENTITIES:
                                batch.put(_ENTITY)
                                #print 'batch.put(_ENTITY)'

                        #query = datastore.Query(kind='CI_ADJ_TYPE')
                        #for result in query.fetch():
                         #   print result   

    except Exception, inst:
        print '\nLoad_Datastore() Unexpected error: ' + str(sys.exc_info()[0])
        if isinstance(inst,ValueError):
            print "ValueError: %s\n"%(inst.message)
        if isinstance(inst,AttributeError):
            print "AttributeError: %s\n"%(inst.message)
        if isinstance(inst,TypeError):
            print "TypeError: %s\n"%(inst.message)                            
        raise

Any suggestions to get past this bug are appreciated.

[dhermes]

You are calling connection = storage.get_connection() and using that connection with the datastore package.

You need a datastore connection for interacting with the datastore

from gcloud import datastore
datastore_cnxn = datastore.get_connection()

You can instead just let the implicit behavior take over and just execute

with datastore.Batch() as batch:
    ...

Sorry I didn't try to dig deeper into the code snippet, it is a bit of a mouthful.

[dhermes]

@charlesbclifford I'm going to close out. The issue is expected since Connection.commit will in fact be an attribute error for the Connection defined in storage, but won't be for the one defined in datastore.

Please re-open if you are still having issues or have remaining questions.

[charlesbclifford]

Thanks for the fix, which I implemented.

Now, using the correct datastore connection, this error is thrown:
gcloud.exceptions.ServiceUnavailable

There's got to be some other object property I've corrupted.

On Friday, May 8, 2015, Danny Hermes notifications@github.com wrote:

@charlesbclifford https://github.com/charlesbclifford I'm going to
close out. The issue is expected since Connection.commit will in fact be
an attribute error for the Connection defined in storage, but won't be
for the one defined in datastore.

Please re-open if you are still having issues or have remaining questions.

—
Reply to this email directly or view it on GitHub
#865 (comment)
.

Charles B. Clifford
(720) 842-7251

[dhermes]

Can you try making a datastore request with a short snippet and report back the stack trace?

For example

from gcloud import datastore

# Make sure proper env. vars. are set for config
q = datastore.Query('Foo')
print list(q.fetch())

---

I'm going to guess you are getting the 503 error because you haven't turned on the Cloud Datastore service yet in the APIs console. Can you make sure it is turned on?

[charlesbclifford]

In the Google Developers Console, of a project for which I am an
owner, under APIs and auth, I can see that the GoogleCloud Datastore API is
enabled.

When I import datastore, set the Datastore default data set id, and the
Datastore default connection, and submit the suggested commands, the
ServiceUnavailable error is returned.

Are there more than the above 2 defaults which must be set? My code must be
missing some configuration setting.

On Friday, May 8, 2015, Danny Hermes notifications@github.com wrote:

Can you try making a datastore request with a short snippet and report
back the stack trace?

For example

from gcloud import datastore

Make sure proper env. vars. are set for config

q = datastore.Query('Foo')print list(q.fetch())

---

I'm going to guess you are getting the 503 error because you haven't
turned on the Cloud Datastore service yet in the APIs console. Can you make
sure it is turned on?

—
Reply to this email directly or view it on GitHub
#865 (comment)
.

Charles B. Clifford
(720) 842-7251

[dhermes]

Can you post a stack trace? What OS are you using? How are you setting your environment variables (or your defaults in code?) What form of auth are you using?

[charlesbclifford]

I am gathering together all of the requested info and will post the info
shortly.

Thanks.

On Friday, May 8, 2015, Danny Hermes notifications@github.com wrote:

Can you post a stack trace? What OS are you using? How are you setting
your environment variables (or your defaults in code?) What form of auth
are you using?

—
Reply to this email directly or view it on GitHub
#865 (comment)
.

Charles B. Clifford
(720) 842-7251

[dhermes]

Did you intend to open a new issue?

[jgeewax]

Also, what happens when you look at the Datastore UI for your project (https://console.developers.google.com/project/_/datastore/stats , then choose your project from the list)