0

I am crawling 5 different sites for data using node-fetch and cheerio. everything checks out but I need to collect the returned data from these 5 separate functions in an array.

First I store function name and url for each site in an object like so

    url: 'https://sampleurl.com',
    crawl: FirstLinkCrawl
}

const secondLink = {
    url: 'https://sampleurl.com',
    crawl: secondLinkCrawl
}
}```

Then I write the function to crawl each site like so, I ran this function with and without promise, both check out


```const secondLinkCrawl = (body) =>{
    return new Promise((res, rej)=>{
    "this crawl function is ran here, everything works fine, Data is an object"
        const error = false
        if(!error){
            res(Data)
        }else{
            rej()
        }
    })         
}```

This here is my fetch function that takes the url and a callback, which is the crawl function 

```async function Fetch(url, callback){
    const response = await fetch(url)
    const html = await response.text()
    callback(html)
}

Then I call the fetch and crawl using promise.all() in an attempt to have it return in an array, but in an array,

    const promises = [
     Fetch(firstLink.url, firstLink.crawl), 
     Fetch(secondLink.url, secondLink.crawl)]

    Promise.all(promises)
        .then(values => console.log(values))
        .catch(err => console.log(err))
}

When I run this, I get [ undefined, undefined ] But when I run it without promises and simply log the result, both of them run successfully. My aim is to get my results in a single array. what can I do?

I also tried declaring an array a the top of the page and then pushing each result into the array, after which I log the array at the bottom of the functions call. But the array returns empty

1 Answer 1

1

You're not returning anything from Fetch function that's why it's undefined. You can fix it by -

async function Fetch(url, callback){
    const response = await fetch(url)
    const html = await response.text()
    const result = await callback(html);
    return result;
}

As the callback, you are passing in Fetch function returns Promise so we can await it and return the result

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.