3

I want to build an app which has easy interactions with google storage, i.e., list files in bucket, download a file, and upload a file.

Following this tutorial, I decided to use a service account (not a user one) for authentification and followed the procedure. I created a public/private key on my console and download the key on my machine. Then I created the .boto file which points to this private key, and finally launched this program and it worked:

import boto
import gcs_oauth2_boto_plugin


uri = boto.storage_uri('txxxxxxxxxxxxxx9.appspot.com', 'gs')

for obj in uri.get_bucket():
  print '%s://%s/%s' % (uri.scheme, uri.bucket_name, obj.name)

As you can see, the package gcs_oauth2_boto_plugin is not used in the code, so I decided to get rid of it. But magically, when I comment the import gcs_oauth2_boto_plugin line and run the program again, I get this error:

C:\Users\...\Anaconda3\envs\snakes\python.exe C:/Users/.../Dropbox/Prog/s3_manifest_builder/test.py
Traceback (most recent call last):
  File "C:/Users/.../Dropbox/Prog/s3_manifest_builder/test.py", line 10, in <module>
    for obj in uri.get_bucket():
  File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\storage_uri.py", line 181, in get_bucket
    conn = self.connect()
  File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\storage_uri.py", line 140, in connect
    **connection_args)
  File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\gs\connection.py", line 47, in __init__
    suppress_consec_slashes=suppress_consec_slashes)
  File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\s3\connection.py", line 190, in __init__
    validate_certs=validate_certs, profile_name=profile_name)
  File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\connection.py", line 569, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "C:\Users\...\Anaconda3\envs\snakes\lib\site-packages\boto\auth.py", line 987, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials

So my questions are:

1- how can you explain that deleting an import which IS NOT USED in the code makes it fail?

2- more generally, to be sure to understand the authentification process, if I want to run my app on a machine, I must be sure to have the .boto file (which points to my service account private key) generated previously? Or is there a cleaner/easier way to give access to my application to Google Storage for in/out interactions?

For instance, I only have to provide public and private key as strings to my program when I want to connect to a S3 bucket with boto. I don't needto generate a .boto file, importing packages etc..., which makes it so much easier to use, isn't it?

1 Answer 1

1

1- how can you explain that deleting an import which IS NOT USED in the code makes it fail?

The first hint is that the module is named a "plugin", although exactly how that's implemented isn't clear on the surface. It intuitively makes some sense that not importing a module would lead to an exception of this kind, though. Initially, I thought it was a bad practice of doing stateful activity on a global during the init of importing that module. In some ways, that is what it was, but only because class hierarchies are "state" in the meta-programmable python.

It turns out (as in many cases) that inspecting the location that stacktrace was thrown from (boto.auth.get_auth_handler()) provides the key to understanding the issue.

(see the linked source for commented version)

def get_auth_handler(host, config, provider, requested_capability=None):
    ready_handlers = []
    auth_handlers = boto.plugin.get_plugin(AuthHandler, requested_capability)
    for handler in auth_handlers:
        try:
            ready_handlers.append(handler(host, config, provider))
        except boto.auth_handler.NotReadyToAuthenticate:
            pass

    if not ready_handlers:
        checked_handlers = auth_handlers
        names = [handler.__name__ for handler in checked_handlers]
        raise boto.exception.NoAuthHandlerFound(
            'No handler was ready to authenticate. %d handlers were checked.'
            ' %s '
            'Check your credentials' % (len(names), str(names)))

Note the reference to the class AuthHandler, which is defined in boto.auth_handler.

So, you can see that we need to look at the contents of boto.plugin.get_plugin(AuthHandler, requested_capability):

def get_plugin(cls, requested_capability=None):
    if not requested_capability:
        requested_capability = []
    result = []
    for handler in cls.__subclasses__():
        if handler.is_capable(requested_capability):
            result.append(handler)
    return result

So, it becomes clear, at last finally when we see that the class definition of the class OAuth2Auth in gcs_oauth2_boto_plugin.oauth2_plugin, in which it is declared as a subclass of boto.auth_handler.AuthHandler, signaling its auth capabilities to the boto framework via the following member variable:

capability = ['google-oauth2', 's3']

2- more generally, to be sure to understand the authentification process, if I want to run my app on a machine, I must be sure to have the .boto file (which points to my service account private key) generated previously? Or is there a cleaner/easier way to give access to my application to Google Storage for in/out interactions?

This has a more generalized answer: You can use a .boto file, although you can also use service account credentials, and you could even use the REST API and go through an oauth2 flow to get the tokens needed to send in the Authorization header. The various methods of auth to cloud storage are in the documentation. The tutorial/doc you linked shows some methods, you've used .boto for another method. You can read about the Cloud Storage REST API (JSON) here and you can read about python oauth2 flows of various kinds here.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.