any way to speed up php function? many curl requests

Question

Curl function to check if link is available is repeated about 600 times, and it take about 20 minutes, i remember year ago makes something and it start to make the same in 30 sec

//return the availability of link
function check($url){
$agent = 'Mozilla/4.0 (compatible;)';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url );
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_NOBODY, TRUE);
curl_setopt($ch, CURLOPT_TIMEOUT_MS, 60000);
curl_setopt($ch, CURLOPT_PROXYTYPE, 'HTTP');
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_FAILONERROR, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT_MS, 60000);
curl_setopt($ch, CURLOPT_PROXY, '127.0.0.1:11107');
    // set start time
    $mtime = microtime(); 
    $mtime = explode(' ', $mtime); 
    $mtime = $mtime[1] + $mtime[0]; 
    $starttime = $mtime; 
$page = curl_exec($ch);
$httpcode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
echo curl_error($ch); echo '<br />';
    // set end time
    $mtime = microtime(); 
    $mtime = explode(" ", $mtime); 
    $mtime = $mtime[1] + $mtime[0]; 
    $endtime = $mtime; 
        $totaltime = ($endtime - $starttime);
curl_close($ch);

echo "This script executed in " .$totaltime. " seconds. With ".$httpcode." Status.";
if($httpcode>=100 && $httpcode<600) return $httpcode; else return 0;
}

and i call this function with

$new_pr   = check($url);

Perhaps multi-curl

Mark Baker
– Mark Baker

2014-04-18 23:12:47 +00:00
Commented Apr 18, 2014 at 23:12 — Mark Baker
– Mark Baker, Commented Apr 18, 2014 at 23:12

David Harris · Accepted Answer · 2014-04-18 23:19:44Z

4

Use a multi cURL handler.

I also noticed that you have an extraordinarily high timeout value. This will contribute to decreased speed if the site actually takes that long to respond. Beware.

$opts = array(CURLOPT_HEADER => true, CURLOPT_TIMEOUT => 30, CURLOPT_NOBODY => true, CURLOPT_USERAGENT => $agent, CURLOPT_FAILONERROR => true, CURLOPT_RETURNTRANSFER => true, CURLOPT_CONNECTTIMEOUT => 60, CURLOPT_PROXY => '127.0.0.1:11107');
$mh = curl_multi_init();
$urls = array('http://url.com', 'etc');

foreach($urls as $url) {
    $ch = curl_init($url);
    curl_setopt_array($ch, $opts);
    curl_multi_add_handle($mh, $ch);
}
$results = array();
do {
    while(($exec = curl_multi_exec($mh, $running)) == CURLM_CALL_MULTI_PERFORM);
    if($exec != CURLM_OK) {
        break;
    }
    while($ch = curl_multi_info_read($mh)) {
        $j++;
        $ch = $ch['handle'];
        $error = curl_error($ch);
        if(!$error) {
            $results = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        } else {
            $results[] = $error;
        }
        curl_multi_remove_handle($mh, $ch);
        curl_close($ch);                
    }
} while($running);
curl_multi_close($mh);

answered Apr 18, 2014 at 23:19

David Harris

2,71719 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ferrysyahrinal Over a year ago

awesome.. thanks.. but, how if the case i want post to same url but different post data???

CoverUp · Accepted Answer · 2014-10-26 01:17:14Z

3

Multi curl is good but the main thing here is speed up curl it self

curl_setopt($curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );

Regards

answered Oct 26, 2014 at 1:17

CoverUp

2592 silver badges2 bronze badges

Collectives™ on Stack Overflow

any way to speed up php function? many curl requests

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related