Decrease memory usage of recursive functions in PHP

Question

I have a recursive function in PHP where it loops in an API where it just allows you to recover 200 records of it at a time.

But since this API has a very high response latency we decided to use a local intermediate database where I add these records and the same will be shown on the site.

However, as this API has more than 30000 records the recursive function it consumes a lot of memory due to it in the case of 30000 records it would have to be recursively invoked more than 1500 times and it ends up giving the famous StackOverflow.

I wonder if there is a manual way to clear the memory of this function by calling it again without losing the value of it.

code example:

public function recursive ($index = 0, $offset = 200) {
   $api = GetConectApi ($index, offset)-> Getrecords ();
   foreach ($api $value) {
      \\Here is my necessary loop
   }
   if (count ($API) > 0) {
      $this Recursive ($index + 200, $offset + 200);
   }
}

I would like to find a way that when it called the recursive function again eliminated the previous allocation without losing the reference value that was passed.

Hi, thanks for you answer, cache works for me after the first interaction loop is done, and this check took place once a day, so regardless of using caching the script will run 1 time a day to make possible'm going updates in the logs — Fabio William Conceição
– Fabio William Conceição, Commented Sep 26, 2017 at 14:41

IMSoP · Accepted Answer · 2017-09-26 15:35:10Z

To expand on user3720435's answer, you are using a lot of memory by creating a new $api variable each time you run the function. To understand why, let's "unroll" the code - imagine it was all written out in sequence with no function calls:

$api1 = GetConectApi ($index1, offset1)-> Getrecords ();
foreach ($api1 => $value1) {
    // Here is my necessary loop
}
if (count ($api1) > 0) {
    // RECURSION HAPPENS HERE
    $index2 = $index1 + 200, $offset2 = $offset1 + 200
    $api2 = GetConectApi ($index, offset)-> Getrecords ();
    foreach ($api2 => $value2) {
        // Here is my necessary loop
    }
    if (count ($api2) > 0) {
        // RECURSE AGAIN, AND AGAIN, AND AGAIN
    }
}

Note that I've renamed all the variables as $api1, $api2, etc. That's because each time you run the function, $api is actually a different variable. It has the same name in your source code, but it doesn't represent the same piece of memory.

Now, PHP doesn't know you're not going to use $api1 again when you create $api2, so it has to keep both in memory; as you end up with more and more sets of data, it needs more and more memory.

user3720435's suggestion is to add unset($api) before the recursion:

$api = GetConectApi ($index, offset)-> Getrecords ();
foreach ($api => $value) {
      // Here is my necessary loop
}
if (count ($api) > 0) {
      unset($api);
      // code as before
}

This tells PHP that you don't need that memory any more, so as it recurses, it won't build up. You'll still build up multiple copies of $index and $offset, but these are likely to be very small in comparison.

All that being said, it's not clear why you need recursion here at all. The whole thing could actually be changed to a simple loop:

do {
    $api = GetConectApi ($index, offset)-> Getrecords ();
    foreach ($api => $value1) {
       // Here is my necessary loop
    }
    $index = $index + $offset;
} while (count ($api) > 0)

A do..while loop always executes once, and then keeps repeating until the condition becomes false. Unrolled it looks like this:

// do...
    $api = GetConectApi ($index, offset)-> Getrecords ();
    foreach ($api => $value1) {
       // Here is my necessary loop
    }
    $index = $index + $offset;
if (count ($api) > 0) { // while...
$api = GetConectApi ($index, offset)-> Getrecords ();
    foreach ($api => $value1) {
       // Here is my necessary loop
    }
    $index = $index + $offset;
}
if (count ($api) > 0) { // while...
// etc

Note that we don't need to allocate any extra memory as we go round the loop, because we haven't entered a new function - we're just using the same variable over and over again.

Thanks so much, you explaination helped me a lot and solved my problem, thanks for real

user3720435 · Accepted Answer · 2017-09-26 15:13:47Z

4

You can try doing a cleanup on your $api variable when done with it.

$cnt = count($api);
$api = null;
unset($api);
if ( $cnt > 0) {

edited Sep 26, 2017 at 15:13

answered Sep 26, 2017 at 14:40

user3720435

1,4762 gold badges20 silver badges29 bronze badges

2 Comments

Fabio William Conceição Over a year ago

But if I give unset in my API I will not have to create the variable again? Isn't that going to generate the same memory consumption?

user3720435 Over a year ago

You are creating a new $api everytime you call recursive. That isn't changing. Try it: $cnt = count($api); $api = null; unset($api); if ($cnt > 0) {

Klamius · Accepted Answer · 2017-09-26 15:02:45Z

1

you can use queue system to fetch all data and save it to your database like RMQ

or you can have [index] in your DB let's say set to 0

then you add cron job to fetch the data from API without recursion and it will run every minute for example

it will go to db get index and you have the offset and fetch the data and increase the index

1 min later the job will run again go to db get index and you have the offset and fetch the data and increase the index and so on

answered Sep 26, 2017 at 15:02

Klamius

1771 gold badge2 silver badges10 bronze badges

1 Comment

Fabio William Conceição Over a year ago

Thanks so much for you answer, the answer above helped me

Collectives™ on Stack Overflow

Decrease memory usage of recursive functions in PHP

3 Answers 3

1 Comment

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related