Selenium script doesn't print the XPath value

Question

I made this Selenium script as a practice to scrape JS heavy pages.

The programs, start up a WebDriver enters a website, then press a button so they all show up then I want just pull some data, the names of the clubs, but there is a problem.

It just prints [], can someone tell me what I'm doing wrong here?

And my goal is to get the names of the clubs like Acadiana Kennel Club, Inc.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

option = webdriver.ChromeOptions()
option.add_argument(" - incognito")

browser = webdriver.Chrome('/home/djurovic/Desktop/Linux ChromeDriver/chromedriver', chrome_options=option)
browser.get('https://webapps.akc.org/club-search/?fbclid=IwAR1X9TkSI49bHgH3w4VmgrMS05sxLbbazaMO17Q1rEfVq7Pj4Ze66B4hdLM#/agility')

timeout = 20

buttonXpath = '//a[@class="button"]'
namesXpath = '//*[@class="ng-binding"]/text()'

try:
    buttonElement = WebDriverWait(browser, timeout).until(lambda browser: browser.find_element_by_xpath(buttonXpath))
    buttonElement.click()
    clubNames = WebDriverWait(browser, timeout).until(lambda browser: browser.find_elements_by_xpath(namesXpath))
    print(clubNames)
except TimeoutException:
    print('Timed out waiting for page to load')
    browser.quit()

Can you try using a for loop? Just to test it. clubNames = browser.find_elements_by_xpath("//*[@class='ng-binding']"), and then iterate through the elements in clubNames list printing text / innerHTML — Alichino
– Alichino, Commented Dec 20, 2018 at 15:56
Will give it a try, does Selenium have some function like Scrapy does , for extract_first() ? — Stefan
– Stefan, Commented Dec 20, 2018 at 15:59
It looks like you want buttonElement to be the result of the browser.find_element_by_xpath call, and likewise for clubNames. But I don't think your WebDriverWait calls return that. — Will
– Will, Commented Dec 20, 2018 at 15:59

Andersson · Accepted Answer · 2018-12-20 16:31:16Z

2

'//*[@class="ng-binding"]/text()' XPath syntax is not supported by Selenium as XPath can return WebElements only.

Try below

buttonXpath = '//a[@class="button" and @name="Search"]'
namesXpath = '//a/strong[@class="ng-binding"]'

try:
    buttonElement = WebDriverWait(browser, timeout).until(lambda browser: browser.find_element_by_xpath(buttonXpath))
    buttonElement.click()
    clubNames = [club.text for club in WebDriverWait(browser, timeout).until(lambda browser: browser.find_elements_by_xpath(namesXpath))]
    for clubName in clubNames: 
        print(clubName)
except TimeoutException:
    print('Timed out waiting for page to load')
    browser.quit()

edited Dec 20, 2018 at 16:31

answered Dec 20, 2018 at 16:06

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Stefan Over a year ago

Works as a charm ! Thanks @Andersson

Collectives™ on Stack Overflow

Selenium script doesn't print the XPath value

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related