Optimize apache/php for CURL crawler with multi curls

Question

I'm running multi curl crawler with 500 open connections all time. Do you have any advices for apache/php settings in order to get best performance?

This is my settings:

<IfModule mpm_prefork_module>
    StartServers          50
    MinSpareServers       50
    MaxSpareServers      100
    MaxClients          1500
    MaxRequestsPerChild   0
</IfModule>

BrenoZan · Accepted Answer · 2014-02-06 18:12:19Z

1

Start your crawler with a small limit of process and then run this script

ps -C httpd,apache2,apache -o rss --no-header | awk '{a=a+$1; b=b+1} END {print a" "a/b" "b}'

it will show you an average use of memory per apache process.. then you can calculate how many processes do you have on limit for your structure based on your ram.

answered Feb 6, 2014 at 18:12

BrenoZan

3041 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

BrenoZan Over a year ago

I can anticipate that your MaxRequestsPerChild will affect too much the memory.. set it to 10 or 100... and set ServerLimit to 1500. ps, sorry the poor english

Nikola Over a year ago

That command gave me this result "640600 12319.2 52" but I don't understand what that means ???

BrenoZan Over a year ago

640Mb total, 12,3 Mb per process, 52 processes ;)

Nikola Over a year ago

So, what should I change in Settings? In my case CPU is overloading to 98%, but memory only 50%

BrenoZan Over a year ago

are you using php 5.5 or APC module on earlier phps? it will help

|

Collectives™ on Stack Overflow

Optimize apache/php for CURL crawler with multi curls

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related