Web scraping Twitter pages Using Python

Question

Using my twitter developer credentials I get twitter API from news channels. Now I want to access the source of the news using URL embedded with twitter API data. I try to get using BeautifulSoup and requests to get content of the twitter page. But continuously I got an error 'We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?' I cleaned the browser and try to use every browser. But got the same response. Please help to solve this problem.

from bs4 import BeautifulSoup
import requests

url = 'https://twitter.com/i/web/status/1283691878588784642'
# get contents from url

content = requests.get(url).content

# get soup
soup = BeautifulSoup(content,'lxml')

Paul595 · Accepted Answer · 2020-07-16 10:51:09Z

1

Do you get the error 'We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?' when running the script or when visiting it with your GUI web browser? Have you tried getting your data through legacy?

If you get this error when running the script, there is nothig you can do like clearing browser cache, etc. The only way to get around this problem is to find another way to access the Twitter page in your python program.

From my experience I would say that the easiest way around this problem is to use gecko driver with FireFox. So Twitter gets all the features it needs.

answered Jul 16, 2020 at 10:51

Paul595

211 silver badge5 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Cordelia Over a year ago

Thank you ,but still couldn't found solution for this matter

Paul595 Over a year ago

Did gecko driver throw the same error or a different?

Collectives™ on Stack Overflow

Web scraping Twitter pages Using Python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related