How to log into Google Cloud Storage from a python function?

Question

I am new to google cloud storage and I try to set up a function that downloads a blob once a day. At the moment I am working in my Jupyter Notebook but finally, the code will run in an Azure Function. I am struggling with setting up the client that connects me to the bucket. I have a service account credential JSON which enables me to connect to google.

Locally I have found a solution:

from google.cloud import storage

client = storage.Client.from_service_account_json('<PATH_TO_SERVICE_ACCOUNT_JSON>')

The problem is that I do not have a path where I store my JSON in the cloud but I store it in the key vault. I came up with the following solution:

from google.cloud import storage
import json
from google.oauth2 import service_account

string_key = get_key_from_key_vault()
service_account_info = json.loads(string_key)
google_credentials = service_account.Credentials.from_service_account_info(
    service_account_info
)
scoped_credentials = google_credentials.with_scopes(
    ['https://www.googleapis.com/auth/cloud-platform.read-only'])
print(type(scoped_credentials))
client = storage.Client(credentials = scoped_credentials)

I am not totally sure if I need the scoped_credentials = ...part but I only have read permissions on the bucket. (if I skip the part the error stays the same)

When I go for this solution I get the following error:

DefaultCredentialsError: Could not automatically determine credentials. Please set 
GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For
 more information, please see https://cloud.google.com/docs/authentication/getting-started

I do not have a clue what I am doing wrong because I think that I already set the credentials explicitly.

Best P

It's the correct way to do,I don't know why it doesn't work. I also don't know what is this path_to_key. Of you have a path, you can directly use it in the storage client. — guillaume blaquiere
– guillaume blaquiere, Commented Aug 24, 2020 at 14:56
@guillaumeblaquiere, thanks for your answer. Thanks for the comment. There was a copy paste error in the code. I edited it seconds ago. — Pet
– Pet, Commented Aug 24, 2020 at 15:04

Jan Hernandez · Accepted Answer · 2020-08-24 20:44:09Z

1

you can set the environment variable GOOGLE_APPLICATION_CREDENTIALS with the path of the json file and authenticate your function by starting the storage client without parameters.

client = storage.Client()

*by default the storage client uses the file path on the environment variable GOOGLE_APPLICATION_CREDENTIALS

It is the easiest way to use JSON credentials and it is compatible with most of Google Cloud python libraries.

answered Aug 24, 2020 at 20:44

Jan Hernandez

4,6602 gold badges16 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Pet Over a year ago

Hello thanks for your answer, I totally understand your answer but I am looking for a solution where I do not have to store my JSON file in the storage. I want to read it from a key vault and this is where my problem starts :-( Best, P

Pet · Accepted Answer · 2020-08-25 08:41:14Z

1

after some more tests i found out that I missed to add project = None. If you add it an use the following command to create the client it works:

client = storage.Client(project = None, credentials = scoped_credentials)

Thanks for your help and food for thought :-)

answered Aug 25, 2020 at 8:41

Pet

2611 gold badge4 silver badges16 bronze badges

Comments

guillaume blaquiere · Accepted Answer · 2020-08-24 19:19:49Z

0

(I use the answer part because code formatting in comment is awful)

Can you try this and tell me if you see the 2 access token printed?

from google.cloud import storage
import json
from google.oauth2 import service_account
from google.auth.transport import requests as grequests

string_key = get_key_from_key_vault()
service_account_info = json.loads(string_key)

google_credentials = service_account.Credentials.from_service_account_info(
    service_account_info
)
google_credentials.refresh(grequests.Request())
print(google_credentials.token)


scoped_credentials = google_credentials.with_scopes(
    ['https://www.googleapis.com/auth/cloud-platform.read-only'])
scoped_credentials.refresh(grequests.Request())
print(scoped_credentials.token)

answered Aug 24, 2020 at 19:19

guillaume blaquiere

76.5k3 gold badges65 silver badges102 bronze badges

1 Comment

Pet Over a year ago

thanks for your answer. I tested your code and I can only see the second token (with scope). The first rises an error:

RefreshError: ('invalid_scope: Invalid OAuth scope or ID token audience provided.', '{"error":"invalid_scope","error_description":"Invalid OAuth scope or ID token audience provided."}')

Collectives™ on Stack Overflow

How to log into Google Cloud Storage from a python function?

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related