Error 403 while scraping a website in python using requests and selenium

Question

i am trying to scrape a website "https://coinatmradar.com/" . I am using requests, beautifulsoup and selenium (wherever required) to scrape data. But after a while, my ip got blocked by the website as it was using cloudflare protection.

country_url = "https://coinatmradar.com/country/226/bitcoin-atm-united-states/"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
response=requests.get(country_url, headers=headers)
soup=BeautifulSoup(response.content,'lxml')

This is the part of code that i am using. I am getting response 403. Is there other way around to make it work with requests and selenium both?

Matteo Bianchi · Accepted Answer · 2021-07-05 20:05:51Z

1

Try to set your headers like that:

headers = {'Cookie':'_gcar_id=0696b46733edeac962b24561ce67970199ee8668', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}

answered Jul 5, 2021 at 20:05

Matteo Bianchi

4421 gold badge6 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Krawly Over a year ago

i will give it a try. I am also using selenium. What changes for chromedriver should i do to use it without getting blocked?

Collectives™ on Stack Overflow

Error 403 while scraping a website in python using requests and selenium

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related