Javascript fetch multiple pages

Question

there is url: "http:/myurl/mydata/pages/" the task is call url+"1",url+"2",...,until there is no correspondent page, number of pages is unknown I am not sure about design of the decision. I know I can you "async await" approach, can I avoid it? Is the next approach ok?

let url= "http:/myurl/mydata/pages/";
let finished=true;
let page=1;
let datas={};
while(!finished) {
   fetch(url+page)
  .then(function(data) {
      collectData(page,data);
      page+=1;
    })
  .catch(function(error) {
      finished=true;  
  });
}  
function collectData(page,data){
  datas[page]=data;
}

Can I use Promises for this task?

Does this answer your question? Resolve promises one after another (i.e. in sequence)? — jknotek
– jknotek, Commented Nov 14, 2019 at 21:02
Seems that you want to scrap some webs, if that is the case, you may use Cheerio.js (cheerio.js.org) — lerichard_v
– lerichard_v, Commented Nov 14, 2019 at 21:04
If it is for creating offline version of data, then you might want to use some sort of search algorithm to find how many pages there are first. Like try binary searching for the last page, if this is in the hundreds. I wouldn't want to wait for each request if I was running this on many urls. Then once you find out the last page, you can request all of them at once, if there are no limitations, using Promise.all. — Sam Ulloa
– Sam Ulloa, Commented Nov 14, 2019 at 22:44

LCPunch · Accepted Answer · 2019-11-14 22:14:58Z

1

If you want a async-await approach:

async function fetchPages(url, page) {
    try{
        const data = await fetch(url+page);
        collectData(page+url, data);
        await fetchPages(url, page + 1);
    } catch(e) {
        return 'Its over';
    }
  }
};

edited Nov 14, 2019 at 22:14

answered Nov 14, 2019 at 21:42

LCPunch

294 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

isick · Accepted Answer · 2019-11-14 21:38:13Z

0

This looks like a job for recursion!

let url= "http:/myurl/mydata/pages/";
let page=1;
let datas={};
recurse().then(/*...*/)


function recurse() {
    return new Promise(function(resolve, reject) {
        fetch(url+page)
        .then(function(data) {
            collectData(page,data);
            page+=1;
            return recurse()
        })
        .catch(function(error) {
            resolve()
        });
    })
}

function collectData(page,data){
    datas[page]=data;
}

NOTE: I didn't test this, so might not work exactly as written. But it should set you on the right track.

answered Nov 14, 2019 at 21:38

isick

1,4871 gold badge12 silver badges25 bronze badges

Collectives™ on Stack Overflow

Javascript fetch multiple pages

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related