PHP faster than cURL?

Question

what is the fastest way to get the http status code. I have a list within about 10k URL's to check. And in best case it checks them every 15 minutes. So i've a php script what uses simple curl functions and loop through them all. But it takes way too much time. Any suggestions what i can do to improve that? What about parallel checks on multiple urls? how many could php manage? I'm very new to this whole performance thing.

This is what i have:

public function getHttpStatus(array $list) {
    $list = array(…); // Array contains 10k+ urls from database.
    for($i = 0; $i < count($list); $i++) {
            $ch = $list[$i];
        curl_setopt($ch, CURLOPT_NOBODY, 1); 
        curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE); 
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
        curl_setopt($ch, CURLOPT_MAXREDIRS, 5);

        $c = curl_exec($ch); 
        $info = curl_getinfo($ch);
        echo $info['http_code'] . '<br />';
    }
}

Thanks in advance!

Or maybe is it the best way to execute that on command line? — YeppThat'sMe
– YeppThat'sMe, Commented Oct 15, 2013 at 18:14
Maybe consider using multithreading for this (github.com/krakjoe/pthreads/blob/master/examples/…) — bwoebi
– bwoebi, Commented Oct 15, 2013 at 18:17

Mike Brant · Accepted Answer · 2013-10-15 18:26:57Z

3

You might consider using curl_multi_exec() - http://php.net/manual/en/function.curl-multi-exec.php, which allows you to process multiple curl handles in parallel. If you like, you can take a look at using a very lightweight REST client I wrote which supports curl_multi_exec(). The link is here:

https://github.com/mikecbrant/php-rest-client

Now, I didn't set up this library to work with HEAD requests, which would actually be much more efficient than GET requests if you are only looking for response codes. But this should be relatively easy to modify to support such a use case.

At the very least this REST client library can give you good sample code with regards to how to work with curl_multi_exec()

Obviously, you would need to play around with the number of concurrent requests that you should use based on what your available hardware and the services you are making requests against can handle.

edited Oct 15, 2013 at 18:26

answered Oct 15, 2013 at 18:16

Mike Brant

71.4k10 gold badges102 silver badges104 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Mike Brant Over a year ago

@YeppThat'sMe The number of handles that can be executed in parallel is going to be determined by your system hardware (memory, CPU) and what it can handle, as well as the response from the server being called (how long it takes to respond, how much data is sent, etc). You would really need to test within your environment to see what works. I haven't really ever run into a case where I have had to use more than say 20 handles at a time. If you have to execute say thousands of requests in a few seconds, you should probably be looking at another solution anyway.

Collectives™ on Stack Overflow

PHP faster than cURL?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related