convert from bash shell to python3 using subprocess.Popen()

Question

I'm struggling with converting from bash shell to python3.

Here's shell command that I want to convert to python:

cat $outDir/aDir/* | cut -f2 | sort -u > $outDir/outFile.txt

I already use subprocess.call()and it worked but I want to know how make it with Popen().

Here's my code which didn't work :

import subprocess
import glob

filePath = outDir + 'aDir/*'
outFilePath = outDir + '/outFile.txt'

fileList = []
for files in glob.glob(filePath):
    fileList.append(files)
with open(files, 'r') as inFile, open(outFilePath, 'w') as outFile : 
  p = subprocess.Popen(['cat'], stdin=inFile, stdout=subprocess.PIPE)   
  p2 = subprocess.Popen(['cut', '-f2'], stdin = p1.stdout, stdout=subprocess.PIPE)
  p3 = subprocess.Popen(['sort', '-u'], stdin = p2.stdout, stdout = outFile)

and could you explain why shell=True is harmful? I saw it in many answers but don't know why...

Thank you.

stackoverflow.com/questions/3172470/… explains why you want to avoid shell=True. — tripleee
– tripleee, Commented Jun 3, 2016 at 6:06
@tripleee: and this shows that shell=True can be useful (if there is no untrusted input then it is more likely that one makes a mistake while reimplementing the shell pipeline using subprocess.Popen directly than one gets an error due to incompatibilities for a simple shell command). — jfs
– jfs, Commented Jun 4, 2016 at 19:57
The problem in't that it is useless, but that using it requires understanding. The shell has the unfortunate honor of having proportionally more users who don't know even the absolute basics than even PHP and VBscript combined. (This question is markedly above average in that respect.) — tripleee
– tripleee, Commented Jun 5, 2016 at 6:16

iruvar · Accepted Answer · 2016-06-03 02:36:32Z

2

You need to pass a list of files to cat So

subprocess.Popen(['cat'], stdin=inFile, stdout=subprocess.PIPE)

should become

subprocess.Popen(['cat'] + [fileList], stdout=subprocess.PIPE)

And consequently inFile should no longer be needed

So, all in all

import subprocess
import glob

filePath = outDir + '/aDir/*'
outFilePath = outDir + '/outFile.txt'

fileList = glob.glob(filePath)
with open(outFilePath, 'w') as outFile: 
  subprocess.Popen(['cat'] + [fileList], stdout=subprocess.PIPE)
  p2 = subprocess.Popen(['cut', '-f2'], stdin = p1.stdout, stdout=subprocess.PIPE)
  p3 = subprocess.Popen(['sort', '-u'], stdin = p2.stdout, stdout = outFile)

answered Jun 3, 2016 at 2:36

iruvar

23.5k7 gold badges58 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Will · Accepted Answer · 2016-06-03 07:32:20Z

0

What about just using shell=True and keeping the pipes?

with open(files, 'r') as inFile, open(outFilePath, 'w') as outFile : 
  p = subprocess.Popen('cut -f2 | sort -u', shell=True, stdin=filePath, stdout=subprocess.PIPE)
  p.communicate()

Or even, more simply:

p = subprocess.Popen("cat {} | cut -f2 | sort -u > '{}'".format(filePath, outFilePath), shell=True)
p.communicate()

Or, even more simply (thanks @tripleee!):

subprocess.call("cat {} | cut -f2 | sort -u > '{}'".format(filePath, outFilePath), shell=True)

As for shell=True, the only danger is really if your input is not safe. I'd recommend quoting all inputs with single quotes, and escaping and sanitizing all inputs.

edited Jun 3, 2016 at 7:32

answered Jun 3, 2016 at 3:29

Will

24.8k14 gold badges100 silver badges111 bronze badges

4 Comments

tripleee Over a year ago

If in contains funky stuff, you need to perform additional steps to escape it from shell interpretation.

tripleee Over a year ago

open(files, 'r') simply throws an error if files is a list.

tripleee Over a year ago

And once you've fixed those things, subprocess.Popen is just inconvenient; use subprocess.call instead.

Will Over a year ago

Sorry, in was meant to be filePath per OP's example. Also added call(), great point, thanks!

Collectives™ on Stack Overflow

convert from bash shell to python3 using subprocess.Popen()

2 Answers 2

Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related