PHP : Optimizing File Parsing

Question

I have a log file of about 200,000 rows . Each row's format is :

AAA||BBB|C|DDD||

Now I parse the values using the following parsing loop :

$fh = fopen($filename, 'r');
if($fh === FALSE) {
  return null;
}
$result = array();
while(!feof($fh)) {
  $line = fgets($fh);
  $tokens = explode('||', $line);
  $a = $tokens[0];
  list($b, $c, $d) = explode('|', $tokens[1]);
  // then I can get the values of AAA , BBB , C and DDD and put it into an array
  $result[$a] = array('a' => $a, 'b' => $b, 'c' => $c, 'd' => $d);
}

$result[$a] contains all I need, however the parsing time is ~2.1 second. What can I do to reduce parsing speed ?

Benchmark and profile using tools designed for the task, like xdebug. This will tell you where the bottleneck is. — Charles
– Charles, Commented May 2, 2013 at 6:23
Disadvantage of fgetcsv() is, I can't check for errors row by row, and ignore / report the problematic line. Once a malformed line is detected, fgetcsv() will fail. — Raptor
– Raptor, Commented May 2, 2013 at 6:33

Raptor · Accepted Answer · 2013-05-03 03:04:07Z

Thanks for all the answers & comments . I did a benchmark of the following functions :

fgets
fgetcsv
stream_get_line
stream_get_line + str_getcsv
fgets + str_getcsv

with the following codes ( omitted the while-loop ) :

// fgets ( same code in the question )
$tokens = explode('||', $line);
$a = $tokens[0];
list($b, $c, $d) = explode('|', $tokens[1]);
$result[$a] = array('a' => $a, 'b' => $b, 'c' => $c, 'd' => $d);

// fgetcsv
ini_set('auto_detect_line_endings',TRUE);
list($a, $nouse1, $b, $c, $d, $nouse2, $nouse3) = fgetcsv($fh, 200, '|');
$result[$a] = array('a' => $a, 'b' => $b, 'c' => $c, 'd' => $d);

// stream_get_line
$line = stream_get_line($fh, 200, PHP_EOL);
$tokens = explode('||', $line);
if(count($tokens) != 3) {
  continue;
}
$a = $tokens[0];
list($b, $c, $d) = explode('|', $tokens[1]);
$result[$a] = array('a' => $a, 'b' => $b, 'c' => $c, 'd' => $d);

// stream_get_line + str_getcsv
$line = stream_get_line($fh, 200, PHP_EOL);
list($a, $nouse1, $b, $c, $d, $nouse2, $nouse3) = str_getcsv($line, '|');
$result[$a] = array('a' => $a, 'b' => $b, 'c' => $c, 'd' => $d);

// fgets + str_getcsv
$line = fgets($fh);
list($a, $nouse1, $b, $c, $d, $nouse2, $nouse3) = str_getcsv($line, '|');
$result[$a] = array('a' => $a, 'b' => $b, 'c' => $c, 'd' => $d);

They parse the same text file in same path in same testing machine. The line format is :

AAA||BBB|C|DDD||

Here is the result (tested 3 times and take average timings) :

fgets : 3.1452s
fgetcsv : 4.3895s
stream_get_line : 3.1956s
stream_get_line + str_getcsv : 3.7148s
fgets + str_getcsv : 3.4751s

Unexpectedly, fgetcsv() is the slowest. But why ?

sidenote: stream_get_line() only available in PHP 5.

Luke Mills · Accepted Answer · 2013-05-02 06:21:25Z

1

File parsing in PHP is slow. I did some benchmarking between fgetcsv and a custom csv function a while ago, and fgetcsv was a clear winner (By a factor of around 10 I think). You should be able to rearrange your code to use fgetcsv, using '|' as your delimiter.

answered May 2, 2013 at 6:21

Luke Mills

1,6261 gold badge11 silver badges18 bronze badges

2 Comments

Raptor Over a year ago

Disadvantage of fgetcsv() is, I can't check for errors row by row, and ignore / report the problematic line. Once a malformed line is detected, fgetcsv() will fail.

Luke Mills Over a year ago

Not sure if this will help, but you could use fgets() to read the line, str_getcsv() to parse it, and use your own function to parse only error rows. str_getcsv() should still parse the line much faster thatn a custom php function.

Aaron Bruce · Accepted Answer · 2013-05-02 06:19:12Z

0

Hm, I'm not sure how much this will really help, but what about exploding on | and assigning $a = $tokens[0], $b = $tokens[2], etc. You'll reduce your explode calls per iteration by one.

You also could achieve a similar thing by using fgetcsv with '|' as your delimiter. Again, not sure how much that's really going to improve things.

answered May 2, 2013 at 6:19

Aaron Bruce

1,1901 gold badge12 silver badges24 bronze badges

1 Comment

Aaron Bruce Over a year ago

Interesting. I was skeptical that you'd get much improvement but I definitely wouldn't have expected it to be worse. It must have to do with parsing for the other 'rules' of a csv file, ie, may or may not be wrapped in ' or ". Things like that.

Collectives™ on Stack Overflow

PHP : Optimizing File Parsing

3 Answers 3

Comments

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related