Retrieving text from Python Selenium

Question

I'm trying to pull some Diablo II trading prices of a trading page using Selenium.

So far I've managed to locate the object I'm interested in using classes, but I can't retrieve the actual text which is what I need.

I have the following code:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get('https://traderie.com/diablo2resurrected/product/3382986705/buying?prop_Platform=PC&prop_Mode=softcore&makeOffer=false')

# Close cookie popup
driver.find_element(By.XPATH, '//*[@id="tyche_cmp_modal"]/div/div/div/div[5]/div[2]/a').click()

# Locate entire page of offers
All = driver.find_element(By.CLASS_NAME, "row")

# Locate individual offer
Offer = All.find_element(By.CLASS_NAME, "col-xs-12")

# Locate price in each offer
Price = Offer.find_element(By.CLASS_NAME, "sc-eCImPb")

# Print price
print(str(Price.text))

# Close page
driver.close()

The print turns out empty, but it should return something like 3x Vex, or similar. What am I doing wrong?

Prophet · Accepted Answer · 2022-02-06 14:48:04Z

1

There are several issues here:

You should add waits. Preferably Expected Conditions explicit waits.
You are using a wrong locator for price element
Since there are multiple offers there you should iterate over the results in a loop.
Variable names should be lowercased according to accepted convention.

I think your code should be something like this:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
wait = WebDriverWait(driver, 20)

driver.get('https://traderie.com/diablo2resurrected/product/3382986705/buying?prop_Platform=PC&prop_Mode=softcore&makeOffer=false')

# Close cookie popup
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="tyche_cmp_modal"]/div/div/div/div[5]/div[2]/a'))).click()

wait.until(EC.visibility_of_element_located((By.XPATH, "//a[contains(@style,'capitalize')]")))
time.sleep(1)

prices = driver.find_elements(By.XPATH, "//a[contains(@style,'capitalize')]")

for price in prices:
    print(price.text)

# Close page
driver.close()

UPD
To iterate over offers and get each offer seller's name and the price you can do something like this:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
wait = WebDriverWait(driver, 20)

driver.get('https://traderie.com/diablo2resurrected/product/3382986705/buying?prop_Platform=PC&prop_Mode=softcore&makeOffer=false')

# Close cookie popup
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="tyche_cmp_modal"]/div/div/div/div[5]/div[2]/a'))).click()

wait.until(EC.visibility_of_element_located((By.XPATH, "//a[contains(@style,'capitalize')]")))
time.sleep(1)

offers = driver.find_elements(By.CLASS_NAME, "col-xs-12")

for offer in offers:
    name = offer.find_element(By.XPATH, ".//a[contains(@href,'profile')]")
    prices = offer.find_elements(By.XPATH, ".//a[contains(@style,'capitalize')]")
    #now you can extract texts from name with name.text
    #and iterate over prices with
    for price in prices:
        price_text = price.text
    #You can put all these in dictionary etc.


# Close page
driver.close()

edited Feb 6, 2022 at 14:48

answered Feb 6, 2022 at 12:44

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Martin Over a year ago

Works quite well, thanks! But how do I see which prices belong to which offer? That was why I divided into All, Offer, and Price which I would then store and loop through.

Prophet Over a year ago

Ah, OK. We can add this. I will edit my solution. But what information do you want to get from the offers?

Martin Over a year ago

I was thinking of storing it in a dictionary in the end with sellers name and the price. Something like this: mydict = {'unconquerable': {'Lo': 1},'Chispas':{'Vex': 3,'Ohm': 1}}

Martin Over a year ago

that is absolutely beautiful, and I can actually recognize some of my original references as well :) thank you very much! Some of the offers contains different price options with stated with "OR" with XPATH "//*[@id="92956358"]/div/div/div[2]/div[1]/div[3]/div/div/div[2]/div/div[2]". What would be the best way to add this "OR" between each price option?

Prophet Over a year ago

It's totally OK, you are welcome!

|

Collectives™ on Stack Overflow

Retrieving text from Python Selenium

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related