TypeError while looping through a variable list of files with subprocess commands

Question

As background, I have created a list with elements of different filenames with their full path (/.../filea.dat) called fileList, it is of variable length. It has the form fileList = ['/../filea.dat', '/../fileb.dat'].

I would like to execute a subprocess command on every file in that list of files, then later analyze the components of each of the files (and generated new files) separately.

for i, elem in enumerate(fileList):
   hexed = fileList[i]
   subprocess.Popen("hexdump " + hexed + " > hexed.dat", shell=True)

   with open("hexed.dat", "r") as f:
      for line in f:
         if "statement" in line:
            value = "examplevalue"
   if value == "examplevalue"
      other subprocess statements that create a file that will again be used later

Right now I have a TypeError: cannot concatenate 'str' and 'list' objects. Let me know if I'm on the right track with this method approach as well.

Please let me know if I need to provide additional clarification; I tried to simplify to the basics as the other details aren't important to the issue.

1- if you want to fix the TypeError then ask about the TypeError (though the error message seems pretty clear. edit your question and add the complete traceback). Thus your question might be useful to somebody else too. If you want to know how to read a subprocess output line by line then ask about that (though there are many existing questions already). If you want to know how to hexlify a file in Python; you can ask about it too as a separate question (you don't need to run hexdump) Try to limit your questions to a single issue per question. — jfs
– jfs, Commented May 11, 2016 at 21:57
I can see how me having multipart question isn't ideal, but for me many of the issues tie together. While, as you say, there are other options for hexdump I am using the subprocess to perform other things beside hexdump, that was just an easy portion to convey here. I assure you I have been looking at other questions, I just simply didn't find those answers helpful for my situation so I posted my question. — stackofpancakes
– stackofpancakes, Commented May 11, 2016 at 22:27

tdelaney · Accepted Answer · 2016-05-11 22:02:52Z

2

You are close. You got the type error because Popen requires that you also set shell=True when passing in a string instead of a list. But there is another problem: Popen doesn't wait for the process to complete so when you read the file, there isn't anything useful in it yet. A different strategy is to skip file redirection and read the output stream directly. Note also that you don't need to use enumerate... the for loop gets the values in the list already. And we can skip the shell and pass the command as a list.

for hexed in fileList:
    proc = subprocess.Popen(["hexdump", hexed], stdout=subprocess.PIPE,
        stderr=open(os.devnull, 'w'))
    for line in proc.stdout:
        if "statement" in line:
            value = "examplevalue"
    proc.wait()
    if proc.returncode != 0:
        print('error') # need less lame error handling!   
    if value == "examplevalue"
        other subprocess statements that create a file that will again be

answered May 11, 2016 at 22:02

tdelaney

77.9k6 gold badges91 silver badges129 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

stackofpancakes Over a year ago

I actually did have shell=True I don't know how I missed adding that into my question. I have made your adjustments and it does seem to be properly executing now. But originally I wanted that output to go into a file as I'll have other parts of my code look at those generated files and generate additional files. I don't think i can pull the stdout shortcut later on I'll have to generate those files.

Collectives™ on Stack Overflow

TypeError while looping through a variable list of files with subprocess commands

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related