How to delete objects with gsutils following a wildcard pattern?

Question

We can run a gsutils command to delete objects as follows:

gsutil rm -a gs://bucket/**

Now I would like to only remove certain folders following a wildcard pattern:

gs://bucket/folder/{WILDCARD-A}/folderA/{WILDCARD-B}/folderB

The idea is to delete all files with target folder B.

How can this be achieved with gsutils?

Have you tried gsutil rm -r gs://bucket/folder/{WILDCARD-A}/folderA/{WILDCARD-B}/folderB applying your wildcat patterns? If you want to bulk delete a hundred thousand or more objects, avoid using gsutil, as the process takes a long time to complete. — Osvaldo
– Osvaldo, Commented May 24, 2022 at 21:28
@JohnAndrews Based on the context of your original question, that specific inquiry will require a new posted question as per Stack Overflow rules. — Hector Martinez Rodriguez
– Hector Martinez Rodriguez, Commented May 24, 2022 at 22:55
I’m voting to close this question because it's not a programming question - please read the tag wiki of google-cloud-storage — Perry
– Perry, Commented Jun 10, 2022 at 3:27

Osvaldo · Accepted Answer · 2022-06-13 14:23:28Z

2

Try gsutil rm -r gs://bucket/folder/{WILDCARD-A}/folderA/{WILDCARD-B}/folderB applying your wildcard patterns. If you have a large number of objects to remove, use the gsutil -m option, which enables multi-threading/multi-processing. In case you need to bulk delete a hundred thousand or more objects, avoid using gsutil, as the process takes a long time to complete. Instead use the Google Cloud console, which can delete up to several million objects, or Object Lifecycle Management, which can delete any number of objects.

edited Jun 13, 2022 at 14:23

answered May 27, 2022 at 19:27

Osvaldo

5092 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Osvaldo Over a year ago

Was this answer useful?

WJA Over a year ago

This solved it. Basically I put * for every place in the pattern where a wildcard was needed. There are some notes on the performance, as also outlined in the link you provided.

Collectives™ on Stack Overflow

How to delete objects with gsutils following a wildcard pattern?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related