I am looking to get the contents of a text file hosted on my website using Python. The server requires JavaScript to be enabled on your browser. Therefore when I run:
import urllib2
target_url = "http://09hannd.me/ai/request.txt"
data = urllib2.urlopen(target_url)
I receive a html page saying to enable JavaScript. I was wondering if there was a way of faking having JS enabled or something.
Thanks
Selenium is the way to go here, but there is another "hacky" option.
Based on this answer: https://stackoverflow.com/a/26393257/2517622
import requests
url = 'http://09hannd.me/ai/request.txt'
response = requests.get(url, cookies={'__test': '2501c0bc9fd535a3dc831e57dc8b1eb0'})
print(response.content) # Output: find me a cafe nearby
There is an alternative for urllib that supports JavaScript HTML Session with JavaScript Support (render) .
from requests_html import HTMLSession
custom_session = HTMLSession()
r = custom_session.get('http://python-requests.org')
r.html.render() #Run JavaScript
print(r.html.search('Python 2 will retire in only {months} months!')['months'])
'<time>25</time>'
