Python how to direct subprocess to output file

Question

I have created a script that receives a bunch of log files as input in order to do some pattern matching. However, my "processFiles" method is not working properly. It should write all data to "fileDest". But the created file remains empty. If I put a "print processFiles" statement under the function itself I can see lines as throughput on my terminal.

To make matters more interesting, bunzip2 reports this error while I run the script:

Proccessing /opt/syslog/app/applog-20140314.bz2. bunzip2: Can't open input file > : No such file or directory. bunzip2: Compressed file ends unexpectedly; perhaps it is corrupted? *Possible reason follows. bunzip2: No such file or directory Input file = /var/tmp/parsed_applog-20140314.decompressed, output file = (stdout)

It seems something in my code sends the output to stdout. And not to the file.

def parse_log_files(self):
    sub_dir = os.listdir(self.base_path)
    for directory in sub_dir:
        if re.search('app\d+', directory):
            fileInput = self.base_path + '/' + directory + '/applog-' + str(self.date.strftime('%Y%m%d')) + '.bz2'
            fileDest = '/var/tmp/parsed_log_files-'  + str(self.date.strftime('%Y%m%d')) + '.decompressed'
            if not os.path.isfile(fileDest):
                subprocess.Popen(['touch',fileDest]).communicate()[0]
            proccessFiles = subprocess.Popen(['/bin/bunzip2','-cd',fileInput,' > ',fileDest],stdout=subprocess.PIPE).communicate()[0]
            accessFileHandle =  open(self.file_out, 'r')
            readFileHandle = accessFileHandle.readlines()
            print "Proccessing %s." % fileInput

jfs · Accepted Answer · 2014-03-17 17:44:05Z

1

' > ' is a shell redirection syntax. Popen doesn't spawn the shell unless you ask (you shouldn't). If you want to redirect the output of a subprocess to a file then use stdout=file_object parameter e.g.:

from subprocess import check_call

with open('/path/to/output', 'wb', 0) as output_file:
    check_call(['command', 'arg1', 'arg2'], stdout=output_file)

answered Mar 17, 2014 at 17:44

jfs

417k210 gold badges1k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user.py Over a year ago

Misread the error message, this is indeed the problem. Cheers!

Collectives™ on Stack Overflow

Python how to direct subprocess to output file

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related