How do I optimize memory usage in PHP for this function?

Question

This function is passed about 70k objects to process. It has no problem loading the array, and it gets through about half the iterations before it fails. Memory is limited to ini_set('memory_limit','465M'); (cloud service). It always fails in the $googleFunction: /app/vendor/googleads/googleads-php-lib/src/Google/Api/Ads/Common/Lib/AdsSoapClient.php.

I've already tried passing the array as a reference, switching from array_chunk to using an array_slice at a time, passing the $chunk by reference to the $googleFunction, unsetting the $chunk at the end, and calling gc_collect_cycles() after each iteration.

How can it still fail? There must be a memory leak somewhere, but there are no large assignments aside from $chunk and $result, and each function called goes out of scope during each iteration, so anything it may have allocated is supposed to be garbage collected. I feel it may have something to do with the function references. After about a quarter of the iterations, it uses 240M. It grows by about 10M per iteration.

  function googleJob($job = null, &$list, $googleFunction, $before = null) {
    // initialize $total, $gaw, $processed
    for ($chunkIndex = 0; $chunkIndex < count($list); $chunkIndex += 2000) { 
      echo '3:'.memory_get_usage().' ';
      $chunk = array_slice($list, $chunkIndex, 2000); # limit of 2000/request
      echo '4:'.memory_get_usage().' ';
      if ($before) {
        foreach ($chunk as $item) {
          $before($item); # function reference 
        }
      }
      echo '5:'.memory_get_usage().' ';

      $i = 0; // try harder to make Google work
      $result = null;
      do {
        try {
          $result = $gaw->$googleFunction($chunk);
          echo '6:'.memory_get_usage().' ';
        } catch (\SoapFault $e) { # try to remove the bad items and try again
          // no errors generated
        }
      } while ($result == null && $chunk); // Retry the request if not empty

      array_walk($chunk, function($item) { $item->save(); });
      echo '7:'.memory_get_usage().' ';
      $processed += count($chunk);
      if ($job) {
        $job->progress = round($processed / $total * 100);
        $job->save() or Yii::error($job->getErrors());
      }
      echo '8:'.memory_get_usage().' ';      
      unset($chunk);
      unset($result);
      echo '9:'.memory_get_usage().'... ';
      gc_collect_cycles();
      echo memory_get_usage().PHP_EOL;
    }
  }

Memory 'profiling' output:

3:110267832 4:110372112 5:111920328 6:123908368 7:129432080 8:129432080 9:121662520... 121662520
3:121662520 4:121766800 5:123281704 6:138001000 7:143493888 8:143493888 9:135745264... 135745264

May or may not be relevant here, but this blog describes in detail what garbage collection does. I've never had luck trying to tell PHP how to manage memory. It's not really designed that way. — Machavity
– Machavity ♦, Commented Sep 14, 2015 at 0:49
you might want to consider using a queue to process these jobs — FuzzyTree
– FuzzyTree, Commented Sep 14, 2015 at 0:53
There's no point in passing $list as a reference, because you never modify that array. — Barmar
– Barmar, Commented Sep 14, 2015 at 1:06
What does $item->save() do? That looks like the only part of the function that will use lots of memory, because it saves every input item somewhere. — Barmar
– Barmar, Commented Sep 14, 2015 at 1:09

volkinc · Accepted Answer · 2015-09-14 12:44:53Z

1

It seems to me you are abusing soap service. if you telling us that your code fails at $googleFunction I can offer you set the $chunk with 100 or 200 objects.

The second thing is a $item->save() function. If you have an access to the class you should check if there are HAS-IS classes. The only place PHP leak the memory is like this construction:

class Foo {
    function __construct()
    {
        $this->bar = new Bar($this);
    }
}

class Bar {
    function __construct($foo = null)
    {
        $this->foo = $foo;
    }
}



for($i = 0; $i < 10; $i++){

     $foo = new Foo();
     unset($foo);
     echo number_format(memory_get_usage()) . "<br>";

}

so If you have objects, I suspect Active Record objects, created in a save() function don't forget to unset them. the easy way is to add destruct like this:

function __destruct()
    {
        unset($this->bar);
    }

This should help

edited Sep 14, 2015 at 12:44

answered Sep 14, 2015 at 2:36

volkinc

2,1381 gold badge16 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Chloe Over a year ago

Google recommends 2000, and supports up to 5000. But I will try 100 at a time, and test for leaks with circular references. Ok even with smaller chunks of 200, it still leaks.

Chloe Over a year ago

Interesting, $save() does in fact leak memory! foreach($agk as $v) { $v->save(); echo memory_get_usage().PHP_EOL; }

Collectives™ on Stack Overflow

How do I optimize memory usage in PHP for this function?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related