Python code works locally but not on server (Discord bot with selenium)

Question

My code takes user input and uses it to search for an item on a webpage. it scrapes the html code of the webpage in order to get the items that appeared as a result:

driver = webdriver.Chrome()
driver.get(urlItem)
#this last line is the one that's not working on heroku
item_list = driver.find_elements_by_class_name('price-history-link')

this is the html code:

<ul class="no-padding small-block-grid-3 large-block-grid-5 text-center word-wrap-break-word">
<li>
<a href="/item/7399/"><img src="/assets/imgs/items/7399.gif" alt="A Grey Faerie Doll" title="A Grey Faerie Doll" class="item-result-image"></a><br><a href="/item/7399/">A Grey Faerie Doll</a>
<br><span class="text-small"><a href="/item/7399/price-history/" class="price-history-link" title="October 31, 2019">2,400,000 NP</a></span>
</li>
</ul>

When I run the code locally, the bot works perfectly. When I run the exact same code from my hosting site (heroku), it doesn't return any items. I thought that maybe it was a loading issue, so I added a :

WebDriverWait(driver, 10).until(ec.visibility_of_element_located((By.CLASS_NAME, "price-history-link")))

when this code is in place, it returns an exception (only when run on the hosting site, it works perfectly when run on my pc)

raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message:

yes, the exception ends there, it doesnt actually show me the message.

I really don't understand why it won't work on heroku... any help is very appreciated!

Edit: Put in a catch for a NoSuchElementException, but it didn't catch anything

Have a look at the potential duplicate, it sounds scarily similar to your issue! — EGC
– EGC, Commented Nov 4, 2019 at 4:22
Alternatively this: What is the version of Firefox? Selenium 2.7.0. supported only up to Firefox 7 (and that without native events). Can you reproduce the same behaviour with Selenium 2.20.0? Empty Error Message and Empty Error Message — EGC
– EGC, Commented Nov 4, 2019 at 4:23

undetected Selenium · Accepted Answer · 2019-11-04 13:49:07Z

0

You saw it right. WebDriverWait() in conjunction with expected_conditions() will always return TimeoutException on failure.

You can find a detailed discussion in Random Selenium E2e Tests Fail because of timeouts on Azure DevOps but work locally and with remote Selenium (BrowserStack Automate)

This usecase

As you are using the Locator Strategy CLASS_NAME as price-history-link possibly this CLASS_NAME is associated with some hidden elements as well. Hence, when you use find_elements_by_class_name(), elements are identified where as visibility_of_element_located() results into TimeoutException.

Solution

To simulate find_elements_by_class_name() along with WebDriverWait() and in conjunction with expected_conditions() you need to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following solutions:

Using CSS_SELECTOR:

item_list = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "a.price-history-link[href$='/price-history/'][title*='20']")))

Using XPATH:

item_list = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//a[@class='price-history-link' and contains(@href, '/price-history/')][contains(., 'NP')]")))

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

answered Nov 4, 2019 at 13:49

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

SouthRescuer Over a year ago

Thanks so much for the answer! Sadly it didnt work. This bot searches for an item, and checks the results and returns them, apparently the searches are coming up empty for some reason. my theory is that my hosting site is blocking the page from showing the results

undetected Selenium Over a year ago

@SouthRescuer ...bot searches for an item... would be easy as search boxes are generally within the Top Window where as the search results can be within an iframe. So traverse up the DOM and check if the results are within an iframe.

Collectives™ on Stack Overflow

Python code works locally but not on server (Discord bot with selenium)

1 Answer 1

This usecase

Solution

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

This usecase

Solution

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related