Reading file PHP using too much memory

Question

So, I'm trying to parse a text file that has 24MB and 314134 lines. The problem is, I feel like my script is using way too much memory.

This is the code:

if(file_exists($filePath)) {
            $data = file_get_contents($filePath);

            $lines = explode("\n", $data);
            foreach ($lines as $line) {
                //Split the line.
                $spllitedLine = explode(';', utf8_encode($line));

                //Get the fields by their index.
                $localidade = !empty($spllitedLine[3]) ? $spllitedLine[3] : '';
                $codigo_postal = $spllitedLine[14] . '-' . $spllitedLine[15];

                $morada = (!empty($spllitedLine[5]) ? $spllitedLine[5] : ' ') . ' ' .
                    (!empty($spllitedLine[6]) ? $spllitedLine[6] : ' ') . ' ' .
                    (!empty($spllitedLine[7]) ? $spllitedLine[7] : ' ') . ' ' .
                    (!empty($spllitedLine[8]) ? $spllitedLine[8] : ' ') . ' ' .
                    (!empty($spllitedLine[9]) ? $spllitedLine[9] : '');

                //Create a new CTT location and save it to the Database.
                $location = new CttLocations();
                $location->address = preg_replace('/\s\s+/', ' ', $morada);
                $location->location = $localidade;
                $location->zipcode = $codigo_postal;
                $location->save(false);

                //Unset the variables to free space.
                unset($location);
                unset($line);
                unset($morada);
            }
        }

This is currently using 153MB of memory and it's not even in the half of the file. I've read that using fopen() fgets() and fclose() it's a better solution but I was using roughly the same amount of memory with those methods. What am I doing wrong? I thought by unsetting the variables I would free some much needed space. I think 150MB is WAY too much for an operation like this. Any thouhgts?

My guess would be additional overhead by PHP and the explode statement. You store every line of the contents in an array. So thats 2x 24MB + overhead. — Syntasu
– Syntasu, Commented May 12, 2015 at 11:51

Loïc · Accepted Answer · 2015-05-12 11:48:48Z

1

This :

 $data = file_get_contents($filePath);

Is way to heavy for big files.

This is how you read a file line by line :

$handle = fopen("inputfile.txt", "r");
if ($handle) {
    while (($line = fgets($handle)) !== false) {
        // process the line read.
    }

    fclose($handle);
} else {
    // error opening the file.
}

answered May 12, 2015 at 11:48

Loïc

12k2 gold badges35 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Reading file PHP using too much memory

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related