PHP Multithread CURL Capability Question

Question

I have a php script that comes down to the following:

while ($arr = mysql_fetch_array($result))
{
   $url = $arr['url'];
   exec("curl $url > /dev/null &");
}

$url will represent a remote script.

My question is, what can I expect if I try to cycle through 2,000 URLs.

Will opening that many CURL connections cripple my server? Could I fire all 2000 in less than one minute?

What I am trying to do is prevent my users from having to setup cronjobs by opening connections and running their remote scripts for them.

Can you guys advise? I'm out of my league today.

Hudson

Artefacto · Accepted Answer · 2010-06-23 14:52:27Z

3

Take a look at curl_multi_init. It will not fire multiple processes so it should be softer on your server.

I would advise you to fire only 3-15 at a time, depending on the load the server can handle and the complexity of the scripts. 2000 at a time will probably make you run out of file descriptors or other limit.

answered Jun 23, 2010 at 14:52

Artefacto

98.1k17 gold badges207 silver badges230 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

atwellpub Over a year ago

It may actually be cleaner and just as quick to us exec("curl $url > /dev/null &"); Because > /dev/null & disregards the need to have output completed before moving on to next curl because it disregards any output. I am not certain.

Artefacto Over a year ago

@atw You can configure curl to send a HEAD request instead and then the remote servers will send only headers, which you can disregard anyway (sending to /dev/null doesn't make it magically faster)

atwellpub Over a year ago

Say the remote script takes three minutes to complete (data compiler) would the returned HEAD act as a read receipt, and still guarantee that the connection will remain open for the entire 3 minutes?

Artefacto Over a year ago

@atw With another plus: with PHP, if you only make requests in batches of say 5, you can easily tell when those 5 have ended and then fire another 5. That's a bit harder to do with jobs. Of course, the best option (without going to async I/O) would be to use Java's thread pool or something to that effect so that you could always have 5 requests taking place (as soon one finished, another would be started).

Artefacto Over a year ago

@atw Yes, it will remain open. The only difference is the http response will have no body (the echoes/etc in the PHP script will have no effect)

|

Martin Wickman · Accepted Answer · 2010-06-23 15:02:24Z

1

You need to keep an eye on the number of opened files (connections) so you don't hit the file-max limit on your server.

answered Jun 23, 2010 at 15:02

Martin Wickman

20k13 gold badges85 silver badges111 bronze badges

Collectives™ on Stack Overflow

PHP Multithread CURL Capability Question

2 Answers 2

8 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

8 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related