I am trying to scrape a site "https://shmoti.com" but it seems they have blocked my ip. So I am trying to use a proxy with node.js .
I am using node-fetch module to get the html body of the site .
How to use proxy with node-fetch? Also I don't know where to get such free proxies . Any help with this ?
I need to use fetch module itself since I am doing all my processing async way .
https://github.com/bitinn/node-fetch#options
it says that from their documentation
{
// These properties are part of the Fetch Standard
method: 'GET',
headers: {}, // request headers. format is the identical to that accepted by the Headers constructor (see below)
body: null, // request body. can be null, a string, a Buffer, a Blob, or a Node.js Readable stream
redirect: 'follow', // set to `manual` to extract redirect headers, `error` to reject redirect
// The following properties are node-fetch extensions
follow: 20, // maximum redirect count. 0 to not follow redirect
timeout: 0, // req/res timeout in ms, it resets on redirect. 0 to disable (OS limit applies)
compress: true, // support gzip/deflate content encoding. false to disable
size: 0, // maximum response body size in bytes. 0 to disable
agent: null // http(s).Agent instance, allows custom proxy, certificate, lookup, family etc.
}
so probably you could try
fetch('example.com',{
agent: new HttpsProxyAgent('http://127.0.0.1:8580')
}).then(function(res){
...
})
