php multiple curl urls with while loop

Question

I am developing a small script and I am bit messed up with using a couple of curl and while loop.

I want to stop processing curl at a point when one of the URL is giving me a information. Note: I have multiple curl requests.

So my concept is,

I have a couple of URLS, which I have to process and get information from. If information is found on a particular URL, it will be giving me a string. If no information is found, it will give me no value. So I have nearly 10 URLs to process approximately. In all cases, any one of the URL will be giving me information, so the remaining urls will be producing no value. Since processing there much URLS, latency is a issue. So suppose in the sample code below, if the url ends with value2.php gives me a result, then I immediately wanted to stop processing the other URLs. Because I already got the result and no point in running other curl. Then finally I have to print the result.

Also I have a condition where none of the URL produce any result and it will be great if someone shows me how to handle that also.

My sample code.

<?php
///functions here
do {

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value1.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value2.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value3.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value4.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value5.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value6.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value7.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value8.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);


$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value9.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"example[dot]com/value10.php?process=$param");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$combined = curl_exec ($ch);
curl_close ($ch);

} while (strlen($combined) != 0);
echo $combied;

///functions here

?>

are your urls are constants? why don't you call just one call to curl with say array of addresses until your result will be sufficient for you? — user15
– user15, Commented Apr 4, 2016 at 15:20
yeah all URLs are constant, but each are different URLs. How can I stop processing curl say if 3rd url gives me result. I have to process a while loop and am struck at that. — Arun Code
– Arun Code, Commented Apr 4, 2016 at 15:25
Is it necessary to stop after the result is found? In other words, do they have to be executed in a specific order and altering the order would mess something up, or does it not matter? — Mike
– Mike, Commented Apr 4, 2016 at 15:35
@Mike yeah it is necessary to stop the process if the result is found. Because once I got the result there is no point of processing remaining URLs because all the remaining will be giving me no result. No there is no need of any specific order for processing. Altering the order is totally fine. — Arun Code
– Arun Code, Commented Apr 4, 2016 at 15:38
OK, so it's not necessary, just preferred. I'll post a different answer in a few minutes. — Mike
– Mike, Commented Apr 4, 2016 at 15:39

Sazzadur Rahman · Accepted Answer · 2016-04-04 15:53:35Z

5

Try the following:

<?php

function callCURL($url) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $combined = curl_exec ($ch);
    curl_close ($ch);
    return $combined;
}


function getResult($urls) {
    $return = array();

    foreach ($urls as $url) {
        $response = callCURL($url);
        if (strlen($response) !== 0) {
            $return[] = $response;
            break;
        }
    }
    return $return;
}

$urls = array("example.com/value1.php?process=$param", "example.com/value2.php?process=$param", "example.com/value3.php?process=$param")

$result = getResult($urls);

answered Apr 4, 2016 at 15:53

Sazzadur Rahman

2,9741 gold badge21 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Arun Code Over a year ago

this works great buddy. As other recommend, can you please update this answer with curl multi.?

Mike · Accepted Answer · 2016-04-04 16:27:28Z

Like you have it in your question and how the other answers have it, the problem is that for each request you have to send it, wait for a response, process the data and only then do you make your subsequent requests. This works fine, but it is highly time-inefficient. Say, for example, each request takes 100 ms to make (which probably not unrealistic). For 10 requests you're looking at 1 second of load time. Instead I would recommend forgetting about trying to stop making requests after you find your result and send all the requests... simultaneously. This can be accomplished with PHP's curl_multi_* functions.

// Put all of your URLs in here. I'm just using google for
// all as an example:
$urls[] = 'http://www.google.com';
$urls[] = 'http://www.google.com';
$urls[] = 'http://www.google.com';
$urls[] = 'http://www.google.com';
$urls[] = 'http://www.google.com';
$urls[] = 'http://www.google.com';

// Get cURL handles
foreach ($urls as $key => $url) {
    $chs[$key] = curl_init();

    // Set all your options for each connection here
    curl_setopt($chs[$key], CURLOPT_URL, $url);
    curl_setopt($chs[$key], CURLOPT_HEADER, 0);
}

//create the multiple cURL handle
$mh = curl_multi_init();

//add the handles
foreach ($chs as &$ch) {
    curl_multi_add_handle($mh,$ch);
}

$active = null;
//execute the handles
do {
    $mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);

while ($active && $mrc == CURLM_OK) {
    if (curl_multi_select($mh) != -1) {
        do {
            $mrc = curl_multi_exec($mh, $active);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
}

foreach ($chs as $url=>&$ch) {
    $html = curl_multi_getcontent($ch);

    // [do what you want with the HTML]

    curl_multi_remove_handle($mh, $ch); // remove the handle (assuming  you are done with it);
}

curl_multi_close($mh);

user15 · Accepted Answer · 2016-04-04 15:27:53Z

0

You can use something like this. I didn't check the code, after debug it should work.

function callcurl($url) 
{
  $ch = curl_init();
  curl_setopt($ch, CURLOPT_URL,"example[dot]com/value5.php?process=$param");
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  $ret = curl_exec ($ch);
  curl_close ($ch);
}

$urls = array('url1', 'url2'); /// etc
$combined = ''
$cnt = 0;
do {
  $combined = callcurl($urls[$cnt++]);
} while (strlen($combined) != 0 && $cnt < count($urls)); //

print $combined;

answered Apr 4, 2016 at 15:27

user15

1,05210 silver badges20 bronze badges

Comments

Jaquarh · Accepted Answer · 2016-04-04 15:47:32Z

0

You can achieve this using a for loop.

$fileNames = array(
   '0' => 'valueabc',
   '1' => 'valuedef',
   [...] => [...]
);
$combined = array();
for ($i = 1; $i < 10; $i++)
{
    $ch = curl_init();
    $url = "example[dot]com/" . $fileNames[$i] . ".php?process=" . $param;
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $combined[$i] = curl_exec ($ch);
    curl_close ($ch);
}

To see all the responses you can foreach $combined.

foreach ($combined as $value=>$response)
{
    // TODO: Work with the response
    echo "[" . $value . "]" . $response;
}

edited Apr 4, 2016 at 15:47

answered Apr 4, 2016 at 15:32

Jaquarh

6,7478 gold badges48 silver badges119 bronze badges

3 Comments

Arun Code Over a year ago

Hi Kyle, thanks for the sample code. BUT in my case, The posted code in my question is edited to hide the real page names publicly. Actually the url looks like example[dot]com/mypageabc.php?process=$param, example[dot]com/mypagedef.php?process=$param and so on, not any numeric value

Jaquarh Over a year ago

Oh, apologies then. How about sticking the names in an array and replacing the $i inside the /value$i with $arrayOfFileNames[$i] ? - ill update it.

Jaquarh Over a year ago

I'd consider a more OOP method of doing so and maybe make the use of constructors. Good luck though.

Collectives™ on Stack Overflow

php multiple curl urls with while loop

4 Answers 4

1 Comment

Comments

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related