5

I am using selenium to scrape an infinite scrolling page.

I am trying to use this code:

import time
import pandas as np
import numpy as np

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

browser = webdriver.Chrome()
url = 'https://twitter.com/search?f=tweets&q=csubwaystats%20since%3A2018-05-28%20until%3A2018-08-28'

browser.get(url)
time.sleep(1)

SCROLL_PAUSE_TIME = 0.5

# Get scroll height
last_height = webdriver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    webdriver.execute_script("window.scrollTo(0,document.body.scrollHeight);")

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = webdriver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

I obtained this code from multiple sources, the most recent being:

How can I scroll a web page using selenium webdriver in python?

I updated it to include "webdriver" instead of "driver" because I import selenium as webdriver. It doesn't work otherwise.

My issue is that when I run the code I get:

AttributeError: module 'selenium.webdriver' has no attribute 'execute_script'

I don't really understand what this means and how to fix it? I haven't been able to find information on this.

I am new to python and so am probably missing something obvious but any advice would be appreciated.

Improve this question
2
  • please update the question with the exact code you use Commented Aug 29, 2018 at 23:21
  • Just did so. Thank you! Commented Aug 29, 2018 at 23:38

4 Answers 4

Reset to default
8

webdriver is the name of the module, not your instance of it. In fact, you assigned the instance you created to the name browser with this line: browser = webdriver.Chrome()

so instead of calling webdriver.execute_script() (which will give you an AttributeError), you must call it using your instance, like this: browser.execute_script().

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

To make it work you have to create an instance of webdriver, e.g.:

from selenium import webdriver

driver = webdriver.Chrome() # webdriver.Ie(), webdriver.Firefox()...
last_height = driver.execute_script("return document.body.scrollHeight")

You can download Chromedriver from here

You also need to add path to Chromedriver to your environment variable PATH or just put downloaded file into the same folder as your Python executable...

Improve this answer

3 Comments

Ahh, that's my bad. I didn't include the code where I did just this. I have from selenium import webdriver driver = webdriver.Chrome('location of chromedriver') ... and then I continued with what was above
Sorry I am not sure how to format comments properly
@agra94 , you should not just define driver, but use it in following lines (note that I used driver.execute_script(), not webdriver.execute_script())
1 
AttributeError: module 'selenium.webdriver' has no attribute 'execute_script'

You are getting this error because 'execute_script' is not a class attribute, you just can not use it directly. Since it is an instance attribute you should create an instance of the class. Please check here to learn more about classes.

This will work fine now since 'execute_script' is running as an instance attribute.

last_height = browser.execute_script("return document.body.scrollHeight")

Your final code would have looked like this:

import time
import pandas as np
import numpy as np

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

browser = webdriver.Chrome()
url = 'https://twitter.com/search?f=tweets&q=csubwaystats%20since%3A2018-05-28%20until%3A2018-08-28'

browser.get(url)
time.sleep(1)

SCROLL_PAUSE_TIME = 0.5

# Get scroll height
last_height = browser.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    webdriver.execute_script("window.scrollTo(0,document.body.scrollHeight);")

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = webdriver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height
Improve this answer

5 Comments

"Do not import selenium as webdriver, import webdriver from selenium"... Considering exception log OP has already imported webdriver from selenium. "You need the webdriver class"... webdriver is not a class, but module where WebDriver class is defined. "selenium is a python program that has python classes in it"... No comments :)
Sorry, I should have clarified. I did use 'from selenium import webdriver' (without the quotations of course) I also did create an instance of webdriver but I use chrome. I have it in my path as well.
@Andersson Please forgive my ignorance about that, I'm a student who is trying to give answers which my efforts in them. I should not have bring up things I do not know though.
@agra94 Well, I am sorry that I did not read your question properly. It is abvious that you already can automate stuff with Chrome, it is my bad.
@CoreyGoldberg I have fixed indentation.
0

For others check your function name. For me I wrote Java function name not the Python one

driver.execute_script("script") # Python
driver.ExecuteScript("script"); # Java

Posting this here because its the top google result for the error

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.