0

I have this code that combines 3 arrays using array_map() to store them to a csv file to the format I like:

$list = array_map(null, $array1, $array2, $array3);
$filename = ‘file.csv’;
$fp = fopen($filename, ‘w’);
foreach ($list as $fields){
 fputcsv($fp,$fields);
}
fclose($fp);

The problem is the array contains millions of rows so whenever I run my script, It always throws a Fatal Error: Allowed memory exhausted.

Is there a way to batch the storing of array_map() to the variable $list?

I tried finding a solution for this online for a few days now and all the solutions I found is not compatible with my code. I’m basically on my last straw here and I’m open on trying any idea you guys could come up with!

Improve this question
2
  • Why not separately save the data of each array? why combine them? any specific reason? Commented Dec 9, 2022 at 7:02
  • So that its on a single csv file. I need them to be on a separate column on the csv file Commented Dec 9, 2022 at 7:09

2 Answers 2

Reset to default
1

Assuming they all have the same number of elements, loop over the first and add the rest into the write without having to build a new array with all of the values...

$filename = ‘file.csv’;
$fp = fopen($filename, ‘w’);
foreach ($array1 as $key => $value1){
    fputcsv($fp, [$value1, $array2[$key], $array3[$key]]);
}
fclose($fp);

If your arrays have different keys, you can use next and current to iterate over the second and third arrays irrespective of the keys themselves...

reset($array2);
reset($array3);

foreach ($array1 as $key => $value1) {
    print_r([$value1, current($array2), current($array3)]);
    next($array2);
    next($array3);
}
Improve this answer
Sign up to request clarification or add additional context in comments.

18 Comments

This wont work since the array is not of the same value. Is there a way to make them work when they’re not the same value? Not creating s new array sounds great
as per my understanding, this will work if we do array_values of each array. @Nigel Ren please comment
Not sure what you mean about the same value, can you give a short example.
@NigelRen what is OP trying to say is that [$key] will not work as his keys for all three array is different. This is what I understood.
Yes, @Anant-Alivetodie is right
|
1
  1. Get a more memory for PHP process.
  2. It's possible to process the arrays in chunks to reduce the amount of memory used at any given time. Here is one way you could do this:

Create a function that will process a chunk of the arrays and write the resulting rows to the CSV file. The function should take three arguments: the arrays to process, the start index, and the number of items to process.

    function processChunk($array1, $array2, $array3, $start, $numItems) {
      $filename = ‘file.csv’;
      $fp = fopen($filename, ‘w’);
    
      // Process the chunk of arrays, starting at the specified index
      // and processing up to $numItems items.
      for ($i = $start; $i < $start + $numItems && $i < count($array1); $i++) {
        $fields = array_map(null, $array1[$i], $array2[$i], $array3[$i]);
        fputcsv($fp, $fields);
      }
    
      fclose($fp);
    }

Use a loop to process the arrays in chunks. For example, you could process 1000 items at a time. Inside the loop, call the processChunk() function to process each chunk of the arrays.

    $chunkSize = 1000;
    for ($i = 0; $i < count($array1); $i += $chunkSize) {
      processChunk($array1, $array2, $array3, $i, $chunkSize);
    }

This should allow you to process the arrays in smaller chunks, which should reduce the amount of memory used. You can use for this cron.

Improve this answer

3 Comments

Thank you for this suggestion, would try this
this will not work : $fields = array_map(null, $array1[$i], $array2[$i], $array3[$i]);
Yup did not work, nice idea though! Gave me something to learn

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.