PHP | Updating array items through function while iterating - Big performance issue

Question

i am iterating an array with more than 3000 items in it array looks something like that:

[
  [
    'id' => 1,
    'type' => 'income'
    'amount' => 10
  ],

  [
    'id' => 2,
    'type' => 'expense',
    'amount' => 20
  ],
  .......  
]

while iterating i call functions that manipulate another array in the same class something like that:

$this->data->revenue->each(function($row) use($user)
{
    if ($row->isIncome())
    {
        $this->addRevenueRowToColumn($row, 'income'); 
        $this->addRevenueRowToColumn($row, 'total'); 
    }

    if ($row->isExpense())
    {
        $this->addRevenueRowToColumn($row, 'expense'); 
        $this->subtractRevenueRowToColumn($row, 'total'); 
    }
}

this is what the functions do:

protected function addRevenueRowToColumn(&$row, $columnName)
{
    $this->report['byMonth'][$row->getMonthTableKey()]['byDepartment'][$row->department_id][$columnName] += $row->amount;
    $this->report['byMonth'][$row->getMonthTableKey()]['total'][$columnName] += $row->amount;
    $this->report['totals']['byDepartment'][$row->department_id][$columnName] += $row->amount;
    $this->report['totals']['total'][$columnName] += $row->amount;
}

protected function subtractRevenueRowToColumn(&$row, $columnName)
{
    $this->report['byMonth'][$row->getMonthTableKey()]['byDepartment'][$row->department_id][$columnName] -= $row->amount;
    $this->report['byMonth'][$row->getMonthTableKey()]['total'][$columnName] -= $row->amount;
    $this->report['totals']['byDepartment'][$row->department_id][$columnName] -= $row->amount;
    $this->report['totals']['total'][$columnName] -= $row->amount;
}

it takes about 11 seconds to process the data and display it

what should i do? thanks in advance!

Can you iterate through some hardcoded or lately-defined number of results. E.g.: Iterate first time through first 200 results, when they are processed, continue with another 200? — Ivan Yonkov
– Ivan Yonkov, Commented Aug 8, 2013 at 7:52
One word: database. Sorry for the programming which looks skilled enough for a real programming language. — Joop Eggen
– Joop Eggen, Commented Aug 8, 2013 at 7:52
Hey Royal - I thought about doing that before asking here, I'll try and update. Joop - you suggest I'll do these calculations in the query? Or just cache the data? — sUP
– sUP, Commented Aug 8, 2013 at 8:06
I would agree with @Joop, if the data exists in the database anyway then use the database to manipulate the data you want. — Jake Aitchison
– Jake Aitchison, Commented Aug 8, 2013 at 8:33
@milkshake, do you have a suggestion on how i should do it? i make about 20 calculations per row in a different data set what is the best approach of doing that in sql? — sUP
– sUP, Commented Aug 8, 2013 at 9:12

Gadoma · Accepted Answer · 2013-08-08 11:51:44Z

1

speed up php only solution
Apart from the suggested use of a database to organize your stuff, if you still want it the hard way ;) you can avoid iterating (let php do it internaly) over the whole array by using one of the php functions below:

array_map — Applies the callback to the elements of the given arrays

and

array_walk — Apply a user function to every member of an array

I see in your code, you have the 'use' clause, so I presume you are PHP > 5.3. In that case, you can do the following:

$yourdata = array_map(
                    function($row) use ($user)
                    { 
                    /*$user->doStuff();*/ 
                    return $row; 
                    }, $yourdata
                    );

Furthermore, a lot of overhead is the display rendering part. If you have a lot of things to display, for example using a simple echo, it's faster to do:

$result = "";
$result .= $something;
$result .= $somethingelse;
echo $result;

than

echo $something;
echo $somethingelse;

enhance solution to use database
But besides that, it will surely be beneficial if you use the database. The most obvious thing is to store your data in some db tables and use some sql-ish solution to query it. It will speed your script up for sure.

speed up the db+php solution
Next, you can get a major performance boost by doing most of your calculations (business logic) inside the database engine in form of stored procedures.

For example if you'd go with mysql, you could create a cursor and iterate over your table rows in a loop. On every row you do some stuff, while ofcourse having direct access to all the tables/columns of your scheme (and possibly other stored procedures/functions). If you are doing a lot of math-ish calculations it's a great solution, but ofcourse IMHO it's usually less convenient (more sophisticated) to write your stuff in SQL rather than PHP ;)

edited Aug 8, 2013 at 11:51

answered Aug 8, 2013 at 11:31

Gadoma

6,5751 gold badge33 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Gadoma Over a year ago

sure thing! :) The performance gain is usually significant when you have your business login written in SQL instead of PHP - it's a great motivator to learn detailed PL/SQL :)

sUP Over a year ago

@Gadoma i can't use native array functions becuase the "array" i'm processing is actually a collection class and each item is an instance of the model. about the dabase - i don't have the knowledge of doing so and i don't really have much time to do it now. but just to learn, what should i google? :) is there any other way do accomplish what i want in PHP? or its a lost cause? btw: when i remove all the logic that manipulates $report (the array where i sum all data by several properties), the loading time is ~550ms. with the processing (add/subtract items in $report) it takes up to 11s

Gadoma Over a year ago

@sUP you're welcome! If you feel my post is the solution to your question you can accept it as the correct anwer or upvote it, so in future others can find it and also benefit from your example. Regards, EDIT: Are you using any PHP framework ? It'd be a lot easier if you could explain the context of all the processing you try to enhance.

sUP Over a year ago

yeah sorry i wanted to make another line in comment so i hit enter by mistake. i posted my full comment (and still thank you!)

Gadoma Over a year ago

I remember when I was doing some similar stuff back in the day (manipulating large datasets with php) I came across that problem as well. Writing OOPhp to process a lot of stuff is not blazing fast, I agree.What can be done ? You can put your processing stuff in a separate procedural php file and run it from you OO code by fetching it with curl for example. Another way is ofcourse using some caching or storing your subcalculations or in-calculation-objects in some way. Thinking about an sql-oriented solution, try to keep stuff neede for subcalculations in the db as well (just like you wrote)

|

Collectives™ on Stack Overflow

PHP | Updating array items through function while iterating - Big performance issue

1 Answer 1

11 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Related