872 questions
1
vote
2
answers
110
views
Date Comparison in UNIX Shell Script failing
I am trying to extract the file date from the output of a gsutil ls -l command and compare it to other dates. However, the comparison appears to be not working.
lstr=$(gsutil ls -l gs://input-bucket/...
- 895
0
votes
0
answers
47
views
How to extract first n rows of a .tsv.bgz file with gsutil?
How to extract the first n rows of a block compressed tab-separated value text file (.tsv.bgz), which could only be accessed with gsutil, into a text file? The original file is very large, so I wonder ...
0
votes
1
answer
77
views
How to preserve only timestamp and mode of the file, an not other posix metadata when using "gcloud storage" command line interface?
When I use -P options with
gcloud storage cp -r -P gs://some/bucket/object ./
It tries to convert the user id also and fails with the following message
ERROR: Root permissions required to set UID ...
- 126
0
votes
1
answer
76
views
in MacOS and it pre-compiled version of crcmod not being detected
➜ datasets gsutil --version
gsutil version: 5.27
checksum: 5cf9fcad0f47bc86542d009bbe69f297 (OK)
boto version: 2.49.0
python version: 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:35:25) ...
- 1,493
2
votes
1
answer
1k
views
Firebase-Storage Access to fetch at '..' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource
I am tying to load an image using the following code:
const blob = await fetch(
"https://firebasestorage.googleapis.com/...?alt=media"
)
.then((r) => r.blob())
const reader = ...
- 5,510
0
votes
1
answer
113
views
AccessDeniedException when trying to delete a file from Google Cloud Storage despite having Storage Admin role
I am facing an AccessDeniedException (403 Access Denied) when trying to delete a file from a Google Cloud Storage bucket. I have assigned the necessary permissions and roles (Storage Admin), but I ...
- 23
1
vote
2
answers
239
views
How can I check that .gcloudignore in GC bucket works properly?
I wanted to exclude some of the files while pushing files to the GC storage. I created .gcloudignore in a root of my bucket. I enabled the .gcloudignore through the command: gcloud config set ...
- 53
1
vote
0
answers
226
views
How to download multiple files from Google Cloud Storage bucket and save each into its separate folder
I'm trying to download a batch of about 50 files from a Google Cloud Storage bucket. I have a bucket that contains multiple files. I can search and filter using the user interface until I have the ...
1
vote
0
answers
49
views
How to transfer file from someone else's gscloud bucket to google collab/python?
I have access rights to only a part of a google cloud bucket of another person. I can download the files from there to my desktop using terminal or using the bang(!) operator to get the files in my ...
0
votes
1
answer
156
views
Google Cloud Storage Signed URL not Accessible in HTML but Works in Browser
`I'm using Python with the Google Cloud Storage library to generate signed URLs for my objects in a bucket. These URLs are meant to be embedded in HTML to display images on a webpage. However, while ...
0
votes
1
answer
99
views
Error in executing gcloud commands in bash script in crontab
I have written a bash script that uses
gcloud command to activate service account and
gsutil command to move files to GCS bucket.
When I run the script using bash command, it executes well.But when I ...
- 105
3
votes
2
answers
3k
views
gcloud storage cp is not working for larger files
I am trying to use gcloud storage cp command for copying large files to gcp bucket. The command is working fine for smaller files. But it gives permission issue for larger files.
gcloud storage cp ...
- 105
1
vote
2
answers
292
views
IPython installation on Mac
I have uninstalled Anaconda on Mac and am now trying to install IPython using pip3.
When I try to install IPython, I get a prompt, "User for us-central1-python.pkg.dev: ". This asks for ...
- 377
0
votes
1
answer
265
views
Unable to download data from Google Cloud Storage to local directory using Google Cloud SDK Shell
Up until a year ago I was able to download folders and directories from my bucket on Google Cloud Storage, now I keep getting errors when I try to download using the same commands that previously ...
0
votes
1
answer
82
views
how to prepend value in content-disposition metadata for all files of GCP bucket wth gs util
I am having files in GCP bucket already uploaded and want to prepend "attachment" value for all files content-disposition property using gs utils.
Below command we have for updating property
...
- 27
0
votes
2
answers
735
views
How can a billing project be stated in Google's command line tool `gsutil cp` on Linux?
I have to use Google's gsutil to download data from a project with "Requester pays" enabled. This does not seem to be possible.
Two different options are suggested on the internet for this ...
- 2,176
0
votes
0
answers
27
views
Gsutil does not work on gcloud, but localy works
Il'show you part of code. When i use in my python script, which get files from Firebase to download. Works everything fine
command = '''gsutil -m cp -r gs://mk-dev.appspot.com/"Images/Project/...
2
votes
2
answers
768
views
GCS - Secure Way to Authenticate to GCS From On-Premises Server
I am looking for the secure/recommended way to connect and perform actions on GCS.
Need:
We have files on an on-premises Linux environment that needs to be uploaded to GCS on a hourly basis. Newer ...
- 669
0
votes
0
answers
143
views
Resumable copying of a large file to GCS
I am copying a large (19Gb) file from the internet to Google Cloud storage using the method described here and here (I use GC shell https://cloud.google.com/shell). So far so good, but it is expected ...
- 3,567
0
votes
0
answers
68
views
Sync Images from google server disk to Bucket
We are copying the images and documents from our local Ubuntu server to the Google bucket using the below command.
gsutil -m rsync -a public-read -R -C -f /media/data_disk/ITEM/ gs://dev-cdn-assets-...
- 71
0
votes
1
answer
173
views
GCP: can we use output of one GSUTIL command embedded in other GSUTIL command
I have a requirement where have a file in GCS and I need to remove the header and trailer of a file and load the data into other GCS bucket.
I have tried using something like this, but its not working....
0
votes
1
answer
2k
views
Setting cors on my GCP bucket not making a difference when accessing data from my bucket
I set all my GCP bucket items to be accessible to the public. I also set cors on my bucket using gsutil cors set cors-config.json gs://myBucketName. My cors config file looks like this: [{"origin&...
- 1,387
0
votes
1
answer
473
views
Cannot Connect to Google Cloud Storage from Google Compute Engine Virtual Machine
I have looked at posts with similar issues, telling me to update API permissions, which I have:
I have also granted permission to the service agent in my Google Cloud storage bucket:
I have tried ...
- 185
2
votes
1
answer
369
views
Google Cloud Storage - How do I copy a specific blob version (not the latest) via the CLI gsutil prompt?
I'm trying to understand if there is a way to copy a specific version of a file to a new location e.g. normally I would do the following:
gsutil -m cp gs://my-bucket/foo-bar.py ./
But if there were ...
- 335
0
votes
1
answer
115
views
Google Colab gsutil - 'ResumableUploadException' object has no attribute 'message'
I got tasked to move my company video archive from Google drive to a Google Cloud archive storage bucket. I have limited experience with gsutil and Google Colaboratory, but I ended up using Colab Pro ...
- 83
0
votes
2
answers
890
views
Error trying to add cors.json file in firebase project
I came accross an error when trying to add cors.json file to my Firebase project.
I have a budget app created with React and powered by authentification and cloud firestore and provided by Firebase.
...
- 49
0
votes
1
answer
90
views
gsutil setmeta: quotes and spaces within "Content-Disposition" metadata
I'm using gsutil setmeta to set a custom filename for downloading, by means of Content-Disposition property (Content-Disposition: attachment; filename="filename.jpg"; filename*="...
- 103
1
vote
0
answers
184
views
gsutil command error - [No space left on device. This can happen if gsutil is configured to save tracker files to an unwritable directory]
When trying to run the gsutil command, as follows:
mkdir -p train_data
gsutil -m cp -r $train_data_gcs_path train_data/
The above commands copies upto 80% data out of 105GB of data and then throws ...
0
votes
1
answer
2k
views
How to copy objects across gcp buckets preserving metadata. gsutil cp drops custom metadata keys wilth null values
I need to copy data across Google Cloud Platform - Cloud Storage(GCS) buckets (source is a GCS bucket and destination is a GCS bucket)
Since I perform copy along some more operations in small batches ...
- 729
0
votes
0
answers
107
views
How to delete just the file contents of the file from cloud storage
I want to delete the file contents from the file in gs bucket, just content, not the file, basically want a 0kb file.
Inside the shell script I tried cp /dev/null (basic Unix cmd) but got error ...
- 1
-1
votes
1
answer
446
views
Issue with Work Identity Federation (WIF) and gsutil ACL set Command
I've an existing setup in which I'm using workload identity federation (WIF) to authenticate circleCI with GCP and everything has been working perfectly fine. Its a simple workflow which uses gsutil -...
- 1,915
1
vote
2
answers
13k
views
Gcloud - How to use a service-account and a user account from the same machine?
I have created a service account for a coworker to upload files into my GCP bucket. I shared the service account credentials and the coworker activated them with the gcloud auth command.
$ gcloud auth ...
- 25
-1
votes
2
answers
2k
views
Move files within same GCP storage bucket and keep version history
How do you move files within the same storage bucket to different folders while keeping their version history? I couldn't find any related option for gsutil mv/cp.
- 730
2
votes
2
answers
385
views
Convert text from txt file into json using jq tool
I have a txt file with values obtained by calling the following command recursively: gsutil ls -r gs://bucket-test/** | while IFS= read -r key; do gsutil stat $key; done, it looks like this:
gs://...
- 147
0
votes
1
answer
96
views
gsutil rsync provide key at runtime
I am currently extending the functionality of a file transfer application to include Google Cloud synchronizations, however I'm running into an issue. In the backend, we are merely running gsutil ...
- 990
1
vote
0
answers
674
views
Alternative for skip operation's cost overhead : --no-clobber in CP operation of GCP
I am using gcloud storage cp for transferring large amount of data from source bucket to destination bucket.
I am using --no-clobber option to skip the existing file if copied already.
gcloud storage ...
- 2,956
2
votes
2
answers
5k
views
Check when GCS storage bucket directory contents were last modified using gsutil
I am transferring a large number of files to a google cloud storage bucket. I would like to make sure the transfer is continuing uninterrupted by checking the last time the directory in the bucket I ...
- 462
2
votes
2
answers
1k
views
Check Google Cloud Storage bucket exists via gsutil
I am writing a function to copy data between buckets through gsutil rsync command.
Just want to make sure if source and destination cloud storage bucket exists before starting rsync operation through ...
- 2,956
1
vote
1
answer
686
views
Access Denied when copying Amazon S3 files using gsutil - fine using aws s3 cp
Similar to this issue here I'm getting Access Denied when trying to copy files from an Amazon S3 bucket to a Google Storage bucket.
I've also broken it down into two parts, copying from S3 > local ...
- 585
0
votes
1
answer
291
views
GetObject from AWS s3 using GCP SA auth and store it localy
I'm trying to access the object in the s3 bucket with the GCP SA and store the s3 object localy.
I tried using the below commands , But got gslib.exception.HashMismatchException md5 signature ...
0
votes
2
answers
646
views
gsutil not working with short lived access tokens
We have been using gsutil to copy files onto GCS and it has been working fine.
We have shifted to use the short lived tokens for this purpose now. However, we notice that after authenticating the ...
- 313
1
vote
1
answer
347
views
How to delete specific sub directory with wildcard gsutil?
Inspired by this question i am trying to delete specific folders from my bucket using wildcard in a gsutil command such as :
gsutil rm -r gs://bucket-name/path/to/**/content
or
gsutil rm -r gs://...
- 2,975
0
votes
1
answer
181
views
How to execute a bash file containing curl instructions within a Google storage bucket and directly copy the contents to the bucket?
I have a bash script file where each line is a curl command to download a file. This bash file is in a Google bucket.
I would like to execute the file either directly from the storage and copy its ...
- 284
0
votes
0
answers
152
views
Uploading data to google cloud storage bucket using gsutil
I am uploading locally stored .mat file with 1.8KB to google cloud from MATLAB using fillowing gsutil command
gsutil cp -f data.mat gs://mytestbucket/
Uploading data takes more than 2 seconds.
...
- 1
0
votes
1
answer
252
views
Gsutil multiple commands in a shell script is working on gitbash for windows but won’t run properly in the linux cli
I have thousands of gsutil commands in a shell script with the syntax
“gsutil cp gs://name of bucket/from-path to-path”
When I execute this script in my windows local using a gitbash it works fine. ...
1
vote
1
answer
145
views
gsutil - what are the storage class options for cp?
I'm using gsutil CLI to copy files to buckets of Google Cloud. But I didn't find what are the options for specifying a storage class? I read the documentation and the options are not written there. It'...
- 3,361
0
votes
1
answer
855
views
gsutil does not work throwing openssl errors
Running any command with gsutil or issuing gsutil command self causes below error inside the cloud shell terminal.
I have tested it with latest Chrome and Firefox. I have also forcibly restarted the ...
- 3,299
0
votes
1
answer
887
views
How to download all bucket files. (The issue with the -m flag gsutil)
I am trying to copy all files from cloud storage bucket recursively and I am having problem with the -m flag as I have investigated.
The command that I am running
gsutil -m cp -r gs://{{ src_bucket }} ...
- 357
0
votes
1
answer
2k
views
How use cat and grep with gsutil and filter with a subdirectory name?
Currently, for getting a string (here: 123456789) into some files in all my buckets I do the following:
gsutil cat -h gs://AAAA/** | grep '123456789' > 20221109.txt
And I get the name of my path ...
- 144
1
vote
1
answer
359
views
How to deal with GSutil URI not working all the time
I am facing a little issue here that I can't explain.
On some occasions, I am able to open files from my cloud storage buckets using a GSutil URI. For instance this one works fine
df = pd.read_csv('gs:...
- 149