How can I run cloud shell commands from Python?

Question

I've been trying to automate my GCP dataflow system. Uncompressed txt files are loaded into the pipeline much faster as compared to compressed gzip files because of some parallelization issue. So, I have to first convert my gzip files into txt files using gsutil commands in google interactive shell:

gsutil cat gs://nse-fao-data-test/FAO* | zcat |  gsutil cp - gs://nse-fao-data-test/uncomp/hello9.txt

Now to automate the system, I try to run this gcloud shell in my local by giving OS command in python and call it every time before my pipeline begins:

import os
import subprocess

def uncompress(in_file = 'gs://nse-fao-data-test/FAO*',out_file="gs://nse-fao-data-test/uncomp/uncompressed.txt"):
    subprocess.call("gsutil cat {0} | zcat |  gsutil cp - {1}".format(in_file,out_file))
    
def openShell():
    os.system("gcloud cloud-shell ssh --authorize-session")

While the openShell command works and starts gcloud shell in my local, but the uncompress does not execute. Is there any way I can automate command present in uncompress() function without writing it manually?

Gorkem · Accepted Answer · 2022-01-24 14:23:29Z

1

os.system runs in its own environment, when that command you call ends, it will exit. so your authorize-session is gone after the call.

You may checkout using subprocess.Popen instead and run multiple commands maybe.

There is a good answer on how to use it

and link to subprocess explanation states: "This module intends to replace several older modules and functions" with explanation on how to replace os.system commands

Edit: For specific use case on gcloud cloud-shell ssh, it opens up a new "interactive" session which you cannot pass commands programmatically, instead you may use --command=XXX argument while running it

To run a remote command in your Cloud Shell, run:
$ gcloud cloud-shell ssh --command=ls

For windows putty you need to allow batch for putty as it is asking entering Return key.

edited Jan 24, 2022 at 14:23

answered Jan 24, 2022 at 13:32

Gorkem

99111 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Stiefel Over a year ago

I get a "Access granted. Press Return to begin session." when running gcloud cloud-shell ssh ... how can I avoid this?

Collectives™ on Stack Overflow

How can I run cloud shell commands from Python?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related