php script speed damping

Question

i have a script running threads i.e. it uses a db of about 50,000 entries and processes them in batches of 400 each .. as such the script runs fine for first.. but gradually dies out.. at an estimate the script should run for about 6 hours but it dies out after an hour or so.

there's no such problem with the code as it works fine for first time, i unset all the not required variables and other stuff to increase the performance but still after an hour or so it dies out.

i use 4 parallel connections to mysql db to achieve this target making up limited 4 connections to db.

why does the script speed damp after time..??

------------------------- DONE!! exhausted memory and was creating some std class obj and now using curl_multi_exec.. works like a charm..!! :)

Ryan Kempt · Accepted Answer · 2012-03-12 05:40:41Z

1

It could be running out of memory? It's hard to tell without seeing the script, maybe offloading some of the work onto the DB depending on your queries could speed some things up - need more information obviously.

answered Mar 12, 2012 at 5:40

Ryan Kempt

4,2096 gold badges32 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sarthak Sawhney Over a year ago

i increased the memmory limit to 256mb. made the php time out to 0. it just keeps getting slower and slower gradually.. starts with a pump n then starts slowing down. the script is about 8000 lines long.. so cant post it here..

DanRedux · Accepted Answer · 2012-03-12 05:41:34Z

0

It can only really be a leak.

Your steps should be:

Create 4 handlers and give each a connection to the DB
While you haven't processed the whole DB
For every handler, execute:
    Process 400 records

Now, as long as you're re-using the same connections, and your "process" step doesn't create anything without unlinking it afterwards, it should be able to go on forever.

answered Mar 12, 2012 at 5:41

DanRedux

9,3996 gold badges25 silver badges41 bronze badges

2 Comments

Sarthak Sawhney Over a year ago

Create 4 handlers and give each a connection to the DB - DONE While you haven't processed the whole DB For every handler, execute: Process 400 records - DONE this is what i'm doing right now.. i use persistent connection to my mysql db and made sure that the persistent time out is more than 50 seconds which is more than enough. i have four parallel connections. after analysis of the database i figured that at one point of time only one of the four thread is processing and then the other.

Sarthak Sawhney Over a year ago

i'm using curl multiple times.. actually 50,000 times.. can that be an issue..??

Collectives™ on Stack Overflow

php script speed damping

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related