Python log parser

Question

I have a large log with several commands (ending with ;) and their outputs (till END) like the following:

<blabla;

foo
...
...

END

<xyz;

...
...

END

--and so on

The requirement is to have separate files with command names like

blabla
xyz

and in each file should be their respective outputs.

So far I have:

def generateDicts(log_fh):
currentDict = {}
for line in log_fh:
    if line.endswith(";"):
       if line.endswith("END"):
          yield currentDict
       currentDict = {""}
   else:
      currentDict["text"] += line
yield currentDict

with open("logfile.txt") as f:
print list(generateDicts(f))

Please help.

1) What is your question? 2) What is insufficient with your solution? Does it print an error? Does it fail to perform correctly? — Robᵩ
– Robᵩ, Commented Apr 27, 2016 at 5:16

Robᵩ · Accepted Answer · 2016-04-27 05:21:28Z

1

Your post says you need to write to files, but your example doesn't do any file I/O. Here is a program that opens, closes, and writes to files.

import fileinput

output = None
for line in fileinput.input():
    line2 = line.strip()
    if line2.startswith('<'):
        output = open(line2[1:].split(';')[0], 'w')
    elif line2 == 'END':
        output.close()
        output = None
    elif output:
        output.write(line)

answered Apr 27, 2016 at 5:21

Robᵩ

170k20 gold badges251 silver badges323 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

slyclam Over a year ago

where do I put the filename?

Robᵩ Over a year ago

You put the filename on the command line. Save my program as "parse_log.py" and then run this command: python parse_log.py logfilename.log

slyclam Over a year ago

I ran it. It generated a blank file with the frst command name. Here is the error I got: >python parse_log.py test.log Traceback (most recent call last): File "parse_log.py", line 9, in <module> output.close() AttributeError: 'NoneType' object has no attribute 'close'

Robᵩ Over a year ago

Did you run it against the example in your post? For me it runs perfectly against that input. It seems as if your must multiple ENDs without an intervening "<". Does every command have a < at the beginning of the line?

Ahasanul Haque · Accepted Answer · 2016-04-27 06:09:40Z

0

You can use re module

import re
with open('test','r') as f,open('output','w') as f1:
    f1.write("\n".join(re.findall(r'\<(\w+)\;',f.read())))

Output:

blabla
xyz

However, if file size is too large, you can consider reading line by line from the file rather than read it as a whole.

answered Apr 27, 2016 at 6:09

Ahasanul Haque

11.2k6 gold badges45 silver badges61 bronze badges

Collectives™ on Stack Overflow

Python log parser

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related