PHP Batch Processing For Loop

Question

I have an end point that I can send a GET request to and it returns a different result depending on the limit and offset.

https://example-foo-bar.com?l=100&o=0 // returns the first 100 items.

I want to create a for loop (or a Nested for loop I assume) that returns a 100 items at a time, adding the result to an array each time, until the end of the response. I have the code for sending the curl request and storing the result, just struggling on the batch processing part.

Something like:

https://example-foo-bar.com?l=100&o=0

https://example-foo-bar.com?l=100&o=99

https://example-foo-bar.com?l=100&o=199

https://example-foo-bar.com?l=100&o=218 // end of response ?

I also know how many result there are in total, stored as $count;

I ended up with something like this but it doesn't feel like the best practice:

function testLoop(){

  $limit = 100;

  $count = getCount();

  $j = ceil($count/$limit);

  for ($i = 0; $i < $j; $i++){
    $offset = $i*100;
    echo 'https://example-foo-bar?l='.$limit.'&o='.$offset.'';
  }
 }

 testLoop();

“until the end of the file” - what file?

C3roe
– C3roe

2018-08-20 10:37:46 +00:00
Commented Aug 20, 2018 at 10:37 — C3roe
– C3roe, Commented Aug 20, 2018 at 10:37
End of response even

Shh
– Shh

2018-08-20 10:45:02 +00:00
Commented Aug 20, 2018 at 10:45 — Shh
– Shh, Commented Aug 20, 2018 at 10:45

Shh · Accepted Answer · 2018-08-20 13:09:45Z

1

I am not sure if I understand the question correctly. But are you looking for something like that?

$offset = 0;
$limit = 100;
$run = true;
$result_array = array();
while($run) {      
  $result_array = array_merge($result_array, json_decode(file_get_contents("https://example-foo-bar.com?l=".$limit."&o=".$offset),true));
  $offset = $offset + $limit;
  if($offset == {somenumber}) {
    $run = false;
  }
}

Then use a cron job to call the php file

edited Aug 20, 2018 at 13:09

Shh

3091 gold badge6 silver badges21 bronze badges

answered Aug 20, 2018 at 11:22

Dieter Kräutl

6774 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Shh Over a year ago

Result is a Json object

Dieter Kräutl Over a year ago

I updated the answer. Please check if that does what you want.

Dieter Kräutl Over a year ago

If you are happy, please markl it as the correct answer :-)

Gopal Rathod · Accepted Answer · 2018-08-20 11:34:50Z

0

Create table 'schedule' and store the data id, link_name,offset and status column
set cron to execute every 10 minutes and take First an entry (one) which status =0
pass param to testLoop($limit) to call function. It may entire link of only offset =0, offset =99, offset =199 like that
After completed to update status=1 in schedule table.
After 10 minute cron Call Step1.

answered Aug 20, 2018 at 11:34

Gopal Rathod

1977 bronze badges

Comments

Saurabh Chandra Patel · Accepted Answer · 2018-08-20 11:39:16Z

0

Best way to use Cron for such type of batch process you can also use php-resque

answered Aug 20, 2018 at 11:39

Saurabh Chandra Patel

13.7k7 gold badges94 silver badges80 bronze badges

Collectives™ on Stack Overflow

PHP Batch Processing For Loop

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related