Selenium web scraping in python cant read .text of elements

Question

I am trying to scrap reviews from verizon website and I found the xpath of reviews by doing inspect on webpage. I am executing below code but this review.text doesnt seems to be working perfectly all the time. I get correct text sometimes and sometimes it just prints Error in message -

Not sure , what am I doing wrong..

from selenium import webdriver

url = 'https://www.verizonwireless.com/smartphones/samsung-galaxy-s7/'
browser = webdriver.Chrome(executable_path='/Users/userName/PycharmProjects/Verizon/chromedriver')
browser.get(url)
reviews = []
xp = '//*[@id="BVRRContainer"]/div/div/div/div/div[3]/div/ul/li[2]/a/span[2]'

# read first ten pages of reviews ==> 
for j in range(10):
    reviews.extend(browser.find_elements_by_xpath('//*[@id="BVRRContainer"]/div/div/div/div/ol/li[*]/div/div[1]'
                                                  '/div/div[2]/div/div/div[1]/p'))
    try:
        next = browser.find_element_by_xpath(xp)
        next.click()
    except:
        print(j,"error clicking")

# Print reviews  ===>
for i, review in enumerate(reviews):
    try:
         print(review.text)
    except:
         print("Error in :" review)

Andersson · Accepted Answer · 2017-02-23 08:56:36Z

2

You should improve the logic of your code. Note, that you cannot get text of elements from the first page after redirection to next page- you need to get text before clicking "Next" button.

Try to use below code instead:

from selenium import webdriver
from selenium.common.exceptions import WebDriverException
import time

url = 'https://www.verizonwireless.com/smartphones/samsung-galaxy-s7/'
browser = webdriver.Chrome()
browser.get(url)
reviews = []
xp = '//a[span[@class="bv-content-btn-pages-next"]]'

# read first ten pages of reviews ==> 
for i in range(10):
    for review in browser.find_elements_by_xpath('//div[@class="bv-content-summary-body-text"]/p'):
        reviews.append(review.text)
    try:
        next = browser.find_element_by_xpath(xp)
        next.location_once_scrolled_into_view
        time.sleep(0.5) # To wait until scrolled down to "Next" button
        next.click()
        time.sleep(2) # To wait for page "autoscrolling" to first review + until modal window dissapeared
    except WebDriverException:
        print("error clicking")


for review in reviews:
    print(review)

answered Feb 23, 2017 at 8:56

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Pushkr Over a year ago

Thank you so much! I did try using WebDriverWait(browser, timeout=10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="BVRRContainer"]'))) but I guess my placement of that logic was wrong altogether..

Andersson Over a year ago

Usually, using ExplicitWait is good idea. You can try to implement it to replace those time.sleep() and save some time

Pushkr Over a year ago

Sure, I am very new to selenium. I will definitely look into ExplicitWait.

Collectives™ on Stack Overflow

Selenium web scraping in python cant read .text of elements

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related