I'm using selenium to get the html for this site:
http://timesofindia.indiatimes.com/world/us
I'm using selenium because this site only gives you all the html if you scroll down. However when I run this code:
# Open the Driver
driver = webdriver.Chrome(chromedriver)
#create a list to store the htmls
master_lst = []
#looping through the times of india urls
for url in urls[:3]:
#make the driver the the url
driver.get(url)
#this is to scroll down twelve time
for i in range(12):
# wait 5 seconds
time.sleep(5)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
html_source = driver.page_source
data = Beautifulsoup(html_source, 'lxml')
master_lst.append(data)
I get this error:
TimeoutException: Message: timeout: Timed out receiving message from renderer: -0.004
I've tried to change the sleep times and the times I scroll down to no avail.
I've seen similar question in here but none that address this kind of problem.
Let me know what you all think!
Thanks!
