Make Python wait for commands to end

Question

I have a python program which dynamically move and rename files into a hadoop cluster. The files usually range from 10mb(parsed) up to 1.5gb (raw data). For the move commands to finish it can take a while and from what I can tell python races through them and none of the move commands get to finish. What is the proper way to have python wait for previous commands. I store the commands in a variable and pass it to os.system. The relevant code is

os.system(moverawfile)
os.system(renamerawfile)
os.system(moveparsedfile)
os.system(renameparsedfile)

I know rename commands are done basically instantaneously. Am I not supposed to use os.system? How do i ensure that python will wait for each command to finish before moving onto the next one.

What is your exact code? os.system does not return until the command it spawns exits. — chepner
– chepner, Commented Mar 2, 2016 at 3:41
You should be using subprocess anyways. You can have an exception thrown on a command error, for example. — Alyssa Haroldsen
– Alyssa Haroldsen, Commented Mar 2, 2016 at 3:44
os.system just calls system(3), and that waits for the command to complete. — Kevin
– Kevin, Commented Mar 2, 2016 at 3:48
hadoop fs -put rawjsondata.txt /home/hadoop/project/March/raw/rawjsondata.txt thats the entirety of "moverawfile" — Sam
– Sam, Commented Mar 2, 2016 at 3:51

Bob Ezuba · Accepted Answer · 2016-03-02 03:45:44Z

1

I would suggest that you use run from subprocess as per Python documentation. It waits for your command to complete before returning.

answered Mar 2, 2016 at 3:45

Bob Ezuba

5101 gold badge5 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

mgilson Over a year ago

Some methods in subprocess wait -- others don't. . .

Sam Over a year ago

whats the proper syntax of subprocess?

ShadowRanger Over a year ago

os.system already waits; if it's behaving asynchronously with os.system, subprocess won't fix that (even though it's a good idea in general).

ShadowRanger Over a year ago

@Sam: For your example command, you'd do something like subprocess.check_call(['hadoop', 'fs', '-put', localfilepath, remotefilepath]) (I'm assuming the file paths are in variables somewhere). Use plain subprocess.call if you don't want it to raise an exception.

Collectives™ on Stack Overflow

Make Python wait for commands to end

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related