Break up memory hogging PHP script with nested for loops

Question

I am trying to build a table of part numbers with this script. I am quickly running out of memory. I have commented out the loops I couldn't even get to:

 foreach ($mat_codes as $mat_code) {
     for ($n = 0; $n < count($shape_codes); $n++){
        for ($d1 = 0; $d1 <= 12; $d1++){
          for ($d1_dec = 1; $d1_dec <= 16; $d1_dec++){
            for ($d2 = 0; $d2 <= 12; $d2++){
              //for ($d2_dec = 1; $d2_dec <= 16; $d2_dec++){
                $build_part = (object) NULL;
                $build_part = $mat_code->part_code;
                $build_part .= $shape_codes[$n]->part_code;
                $build_part .= '.';
                if ($d1 != 0) {
                  $build_part .=  $d1 . '-';
                }
                $build_part .= $d1_dec;
                $build_part .= '.';
                //  if ($d2 != 0) {
                //    $build_part .=  $d2 . '-';
                //  }
                // $build_part .= $d2_dec;
                // $build_part .= '.';

                // * save $build_part to MYSQL * //
             //}
           }
        }
      }
    }
  }

Unsurprisingly this is returning

PHP Fatal error: Allowed memory size of 134217728 byets exhauseted (tried to allocate 71 bytes)

For the record there are about 16 values in $mat_codes and 16 in $shape_codes. By my calculations, I'm trying to create 692,224 part numbers in this example alone. I think I'm going to add more and have up to ~34 million. IMHO, it would be silly to try and get this done by just allocating more memory to the job. My question -- of course -- how do I break this script up into smaller chunks?

Edit: I have a workaround. I apologize it is out of the scope I defined in the original question. I didn't mention that this is simply a table of potential part numbers. Particularly, all the potential part numbers allowed by the logic of the part numbering system. Rather than have a master list of part numbers at the ready just waiting to be used I figure I might be better off validating new part input against the part numbering logic, a function like could_this_be_a_part_number($input). It could just test the inputted string against the part numbering logic as opposed to having some master table I would need to check against.

Edit x2: I'm sorry this is the worst. I moved unset($build_part) into the innermost for loop. The problem was I was trying to a development function in Drupal to print out each $build_part before it was unset - dpm($build_part). The script runs fine, its trying to build a webpage with that many part numbers printed to it is what breaks the script.

What's the context for this? With that much data, it feels like you may get more mileage out of putting this in a database, or in-memory structure like Redis. — Chris Henry
– Chris Henry, Commented Jun 7, 2013 at 2:16
Oh well this is all going to be saved to my MYSQL database. I'm just trying to write a script to populate that database. — kevinaskevin
– kevinaskevin, Commented Jun 7, 2013 at 2:17
Is there any reason you're not storing the value created in the inner loop? You're running out of memory because you're (almost) endlessly concatenating. If you put each result into the DB, you would have way less of a memory problem. — Chris Henry
– Chris Henry, Commented Jun 7, 2013 at 2:19
This might take long, but I don't see why it should use so much memory. To start, I would take out the line where you cast null to an object (why do you do that?) and simply unset the variable $build_part at the end of the inner loop. By the way, running 34 million separate sql queries in one script is definitely going to time out so you might want to do a batch insert a few for's back. — jeroen
– jeroen, Commented Jun 7, 2013 at 2:23

Smuuf · Accepted Answer · 2013-06-07 16:34:42Z

First of all: $build_part seems like it's not supposed to be an object, because you treat it like a string, so don't initialize it like that.

$build_part = NULL; will suffice.

Second of all: We don't know how you'd like your data to be saved in your database, but you could (as my original answer somehow proposed) build up a query, send it to the database and then continue.

Now let's have a look at your code:

$i = 0;
foreach ($mat_codes as $mat_code) {
     for ($n = 0; $n < count($shape_codes); $n++){
        for ($d1 = 0; $d1 <= 12; $d1++){
          for ($d1_dec = 1; $d1_dec <= 16; $d1_dec++){
            for ($d2 = 0; $d2 <= 12; $d2++){
              for ($d2_dec = 1; $d2_dec <= 16; $d2_dec++){
                $i++;
                $build_part = (object) NULL;
                $build_part = $mat_code->part_code.".";
                $build_part .= $shape_codes[$n]->part_code;
                $build_part .= '.';
                if ($d1 != 0) {
                  $build_part .=  $d1 . '-';
                }
                $build_part .= $d1_dec;
                $build_part .= '.';
                 if ($d2 != 0) {
                   $build_part .=  $d2 . '-';
                 }
                $build_part .= $d2_dec;
                // echo "$i $build_part <br>";
                // * save $build_part to MYSQL * //
             }
             // This is first triggered when the count reaches 16 items
             echo "--- lvl 5<br>";
           }
           // 208 items
           echo "--- lvl 4<br>";
        }
        // 3328 items
        echo "--- lvl 3<br>";
      }
      echo "--- lvl 2<br>";
    }
    echo "--- lvl 1<br>";
}

I have determined (and you can do it somewhere else) the level 3 sign as the point where you could be sending some pre-constructed MySQL query to the database. Something like:

$mysqli->query("INSERT INTO parts (part_id) VALUES $very_long_string;");

Also look at this MySQL command: INSERT DELAYED, I think maybe it might (sic!) be quite useful in your situation.

$very_long_string would be built in the most inner loop like:

$very_long_string .= "('$build_part'),";

and in the end (before we create and send the final query) we would cut the last comma:

$very_long_string = rtrim($very_long_string, ",");

Then send the 3000+ items long query into the database. Then unset($very_long_string) and start again.

So I guess the final code would look something like this:

foreach ($mat_codes as $mat_code) {
for ($n = 0; $n < count($shape_codes); $n++){
    for ($d1 = 0; $d1 <= 12; $d1++){
        $very_long_string = null;
        for ($d1_dec = 1; $d1_dec <= 16; $d1_dec++){
            for ($d2 = 0; $d2 <= 12; $d2++){
                for ($d2_dec = 1; $d2_dec <= 16; $d2_dec++){
                $build_part = NULL;
                $build_part = $mat_code->part_code.".";
                $build_part .= $shape_codes[$n]->part_code;
                $build_part .= '.';
                if ($d1 != 0) {
                    $build_part .=  $d1 . '-';
                }
                $build_part .= $d1_dec;
                $build_part .= '.';
                if ($d2 != 0) {
                    $build_part .=  $d2 . '-';
                }
                $build_part .= $d2_dec;
                $very_long_string .= "('$build_part'),";
                }
            }
        }
        $very_long_string = rtrim($very_long_string, ",");
        $mysqli->query("INSERT INTO parts (part_id) VALUES $very_long_string;");
        unset($very_long_string);
    }
}
}

EDIT after reading your EDIT: I was of course dealing with the fact that you wanted to populate the database :D .. If you need a validating function, then that's a whole another story.

To respond to you and @chris-henry and @jeroen. This is a long part number broken up like such: AAA.BBB.#-#.#-#.###.EEE the number of potential items between the .s and -s are: 16.16.13-16.13-16.24.10. Thus I need ~34 million unique strings. I dont understand where in the existing for loop I should write this option to the database. I cant input the part number before its complete.
@fortmac After you save to the database, you need to unset. In other words, you need to make sure that all 692,224 of the completed part numbers aren't still in memory once you are done with them.
Thanks for the thoughtful response. Don't worry about the edit section above. Thats a practical workaround, the original question stands on its own.

Collectives™ on Stack Overflow

Break up memory hogging PHP script with nested for loops

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related