How can i scrape the text from a located element using Selenium and Python

Question

I am trying to run the following code

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)
driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
time.sleep(20) #bypass cloudflare
price = driver.find_element_by_xpath('//*[@id="battlepet-page"]/div[1]/table/tr[3]/td/span')
print (price)

so i can scrape the "Current Price" from the page . But this xpath location won't return the text value( i also tried the "text" varriant in the end with no success.

thanks in advance for any reply

S A · Accepted Answer · 2019-05-05 18:28:28Z

2

First, use WebdriverWait to wait for the element instead of sleep.

Second, Your locator is not finding the element.

Try this,

driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
price = WebDriverWait(driver,30).until(EC.visibility_of_element_located((By.XPATH,"//div[@id='battlepet-page']/div/table/tr[@class='current-price']/td/span")))

print(price.text)

To use wait import the followings,

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

answered May 5, 2019 at 18:28

S A

1,9151 gold badge11 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Sers · Accepted Answer · 2019-05-05 18:24:53Z

You should wait for visibility of element before getting text. Check WebDriverWait in example below:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
rom selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)

wait = WebDriverWait(driver, 20)

driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
current_price = wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, ".current-price .price"))).text

print(current_price)

undetected Selenium · Accepted Answer · 2019-05-05 19:50:11Z

1

To scrape the value of Current Price from the webpage you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tr.current-price td>span"))).text)

Using XPATH:

print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//th[text()='Current Price']//following::td[1]/span"))).text)

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

answered May 5, 2019 at 19:50

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

Collectives™ on Stack Overflow

How can i scrape the text from a located element using Selenium and Python

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related