Most efficient way to parse a file in Python

Question

I want to know what the most efficient way is to parse a text file. For example, lets say I have the following text file:

Number of connections server is: 1

Server status is: ACTIVE

Number of connections to server is: 4

Server status is: ACTIVE

Server is not responding: 13:25:03

Server connection is established: 13:27:05

What I want to do is to go through the file and gather information. For example, number of connections to the server, or the times the server went down. I want to save these values in maybe lists, so that I can view or plot them later.

So what is the best way to perform this, assuming I have my keywords in a list as follows:

referenceLines = ['connections server', 'Server status', 'not responding']

Note that I do not have the complete sentence in the list, but only a part of it. I want to go through the file, line-by-line, and check if the read line corresponds to any entry in the referenceLines list, if so, get the index of the list entry and call the corresponding function.

What would be the most efficient (time, memory) way to do this, as a typical text file will be about 50MB in size.

Thank you.

Any

more efficient than for line in open('filename.txt', 'r'): --do whatever?? — Chris Pfohl
– Chris Pfohl, Commented May 22, 2012 at 13:04
Just as a side note, it's a bit more pythonic to do things as simply as possible and as obviously as possible, only worrying about efficiency after you've proven (by experience or measurement) that it's inefficient — Chris Pfohl
– Chris Pfohl, Commented May 22, 2012 at 13:09
Is it possible to have the entire phrase before colon in referenceLines? Comparing strings for exact match should be faster than substring search. — Janne Karila
– Janne Karila, Commented May 22, 2012 at 13:25
@Christopher. Thank you. At this point I'm basically trying to figure what is the best way to do this. Especially since I have only a substring in referenceList,I want to know how I can compare a complete line with a list that has only a substring, not the other way. — madu
– madu, Commented May 22, 2012 at 13:52
@JanneKarila The lines are usually very long and they are not in the oder of message:value. So I have to compare a substring. Thank you. — madu
– madu, Commented May 22, 2012 at 13:53

mikerobi · Accepted Answer · 2012-05-22 13:09:35Z

4

If every line is seperated by ": ", you can split the string.

message, value = line.split(': ', 1)

edited May 22, 2012 at 13:09

answered May 22, 2012 at 13:04

mikerobi

21k5 gold badges49 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

eumiro Over a year ago

line.split(': ', 1) would be more appropriate.

madu Over a year ago

Thank you mikerobi and eumiro. I have given only an example, but in the real file, the lines are very long and the message:value is usually not seperated by ':'.

rupello · Accepted Answer · 2012-05-22 13:13:33Z

1

As a practical approach, I suggest that you implement this in a series of steps while measuring the performance at each step to gauge the cost of the approach you are using with your test data.

For example:

How long does it take to simply read the file line by line?
How long if you split() each line?
How long if you run re.match() on each line?

The optimal solution will depend on your data, for example, how many reference lines your are using, but it should only take a few seconds on a modern machine

answered May 22, 2012 at 13:13

rupello

8,5312 gold badges39 silver badges35 bronze badges

2 Comments

madu Over a year ago

Thanks rupello. Right now I'm concerned with the best way to detect whether the line read from file (complete line) to the substrings in the referenceList.

rupello Over a year ago

Take a look at this question: stackoverflow.com/questions/3260962/…

marbdq · Accepted Answer · 2012-05-22 13:19:03Z

1

If the text file you want to parse always contains the same fields in the same order, then mikerobi's solution is good. Otherwise, you need to iterate through the lines and try detecting referenceLines...

answered May 22, 2012 at 13:19

marbdq

1,2458 silver badges5 bronze badges

Comments

Janne Karila · Accepted Answer · 2012-05-23 12:00:23Z

1

Here's one possible approach. It uses a regular expression pattern of the form 'keyword1|keyword2' to search for multiple keywords at once.

def func1(line):
    #do something

def func2(line):
    #do something

actions = {'connections server': func1,
           'Server status': func2}

regex = re.compile('|'.join(re.escape(key) for key in actions))

for line in file:
    for matchobj in regex.finditer(line):
        actions[matchobj.group()](line)

edited May 23, 2012 at 12:00

answered May 23, 2012 at 7:11

Janne Karila

25.3k6 gold badges59 silver badges97 bronze badges

1 Comment

madu Over a year ago

Thank you Janne. Regex is what I wanted to try out too but didn't know how to implement erxactly. I will try to implement this way.

Collectives™ on Stack Overflow

Most efficient way to parse a file in Python

4 Answers 4

2 Comments

2 Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

2 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related