Selenium timeout exception even after loop is executed

Question

I encountered a problem which I do not understand why my code is working this way. Essentially I am attempting to run a for loop for x amount of times but my code keeps saying TimeoutException

Traceback (most recent call last):
  File "/Users/Ryan/Desktop/selftest1.py", line 33, in <module>
    EC.presence_of_element_located((By.ID, "ctl00_lblStockname"))
  File "/Library/Python/2.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
TimeoutException: Message:

And the body of my code is :

for x in range(1,10):

    baseurl = 'http://www.hkexnews.hk'
    url = 'http://www.hkexnews.hk/listedco/listconews/advancedsearch/search_active_main.aspx'

    driver = webdriver.Firefox()
    driver.get(url)
    driver.find_element_by_id("ctl00_txt_stock_code").clear()
    driver.find_element_by_id("ctl00_txt_stock_code").send_keys(x)
    driver.find_element_by_id("ctl00_rbAfter2006").click()
    Select(driver.find_element_by_id("ctl00_sel_DateOfReleaseFrom_y")).select_by_visible_text("1999")
    Select(driver.find_element_by_id("ctl00_sel_tier_1")).select_by_visible_text("Financial Statements/ESG Information")
    Select(driver.find_element_by_id("ctl00_sel_tier_2")).select_by_visible_text("Annual Report")
    driver.find_element_by_css_selector("label > a > img").click()

    match = re.compile('\.(html|pdf)')
    try:
        element = WebDriverWait(driver, 1).until(
            EC.presence_of_element_located((By.ID, "ctl00_lblStockname"))
        )
    finally:
        f = driver.page_source
        soup = BeautifulSoup(f,'html.parser')
        for link in soup.findAll('a'):
            try:
                href = link['href']
                if re.search(match, href):
                    file = open("newfile.txt", "a")
                    file.write(baseurl+href+'\n')
                    file.close
                    print ('finished write')
                    print baseurl+href
            except KeyError:
                pass
    driver.quit()

To my understanding the timeout exception is being thrown because of the first try. But shouldn't the loop stop when it hits finally? Also I attempted to add an exception after 'try' and before 'finally' for timeout error and it gave me the error

error: [Errno 61] Connection refused

I am honestly lost at how to fix this issue or what is causing the issue in the first place.

EDIT:

I added back a exception block after resetting everything and it seems to be working fine now. As in:

try: 
   ...
except TimeoutException:
        driver.quit
finally:
   ...

Just for future references if anyone wants to know the solution.

thebadguy · Accepted Answer · 2016-11-14 16:41:33Z

I have updated the code,please try this

from selenium import webdriver
from selenium.webdriver.support.ui import Select
import re
from bs4 import BeautifulSoup
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By


for x in range(1,10):

    baseurl = 'http://www.hkexnews.hk'
    url = 'http://www.hkexnews.hk/listedco/listconews/advancedsearch/search_active_main.aspx'

    driver = webdriver.Chrome()
    driver.maximize_window()
    driver.get(url)
    driver.find_element_by_id("ctl00_txt_stock_code").clear()
    driver.find_element_by_id("ctl00_txt_stock_code").send_keys(x)
    driver.find_element_by_id("ctl00_rbAfter2006").click()
    Select(driver.find_element_by_id("ctl00_sel_DateOfReleaseFrom_y")).select_by_visible_text("1999")
    Select(driver.find_element_by_id("ctl00_sel_tier_1")).select_by_visible_text("Financial Statements/ESG Information")
    Select(driver.find_element_by_id("ctl00_sel_tier_2")).select_by_visible_text("Annual Report")
    driver.find_element_by_css_selector("label > a > img").click()

    match = re.compile('\.(html|pdf)')

    wait = WebDriverWait(driver, 10)
    wait.until(EC.presence_of_element_located(
            (By.XPATH, '//*[@id="ctl00_lblStockName"]')))

    f = driver.page_source
    soup = BeautifulSoup(f,'html.parser')
    for link in soup.findAll('a'):
        try:
            href = link['href']
            if re.search(match, href):
                file = open("newfile.txt", "a")
                file.write(baseurl+href+'\n')
                file.close()
                print ('finished write')
                print baseurl+href
        except KeyError:
            pass
    driver.quit()

It is giving this output:

C:\Python27\python.exe 
C:/XXXX/kimpster.py
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0412/LTN20160412398.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0429/LTN201504291354.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0407/LTN20140407336.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0408/LTN20130408921.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0410/LTN20120410623.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0411/LTN20110411707.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0422/LTN20100422489.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0423/LTN20080423279.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0319/LTN20090319103.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0319/LTN20090319097.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0421/LTN20160421233.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0422/LTN20150422417.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0423/LTN20140423340.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0422/LTN20130422293.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0425/LTN20120425287.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0421/LTN20110421583.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0423/LTN20100423265.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0420/LTN20090420355.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0423/LTN20080423322.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0407/LTN20160407581.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0413/LTN20150413273.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0428/LTN20140428711.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0429/LTN20130429395.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0426/LTN20120426622.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0426/LTN20110426450.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0423/LTN20100423393.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0423/LTN20090423238.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0425/LTN20080425250.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0319/LTN20150319329.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0324/LTN20140324959.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0402/LTN201304021122.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0326/LTN20120326263.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0326/LTN20120326253.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0330/LTN20090330188.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0320/LTN20090320083.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0428/LTN201604281016.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0429/LTN20150429233.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0429/LTN20140429945.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0429/LTN201304291031.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0426/LTN20120426229.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0421/LTN20110421266.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0429/LTN20100429830.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0428/LTN200904281430.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0429/LTN20080429728.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0323/LTN20160323343.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0313/LTN20150313356.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0327/LTN20140327637.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0326/LTN20130326368.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0326/LTN20120326620.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0426/LTN20110426261.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0408/LTN20100408709.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0429/LTN20090429932.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0416/LTN20080416269.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2016/0425/LTN20160425745.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2015/0423/LTN20150423635.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2014/0422/LTN20140422239.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2013/0417/LTN20130417330.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2012/0423/LTN20120423313.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2011/0406/LTN20110406041.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2010/0426/LTN20100426737.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2009/0428/LTN20090428495.pdf
finished write
http://www.hkexnews.hk/listedco/listconews/SEHK/2008/0429/LTN20080429825.pdf

Process finished with exit code 0

Is this not what you expect.As its writing this in file also.

Yes it is the right output i want, but I can't replicate it. I'm guessing its the difference in internet speeds, my page doesn't load as quickly as your network so i add the try block in order to make sure the page has loaded.
@kimpster, please try this code, hope this will work for you also

Wonka · Accepted Answer · 2016-11-14 15:51:16Z

1

You have to set an implicitrly_wait(time) to the browser http://selenium-python.readthedocs.io/waits.html

driver.implicitly_wait(10)

Is the MAX time the browser will wait to find a web element

pd: Your try-finally dont catch the exception, use try-except-finally

answered Nov 14, 2016 at 15:51

Wonka

1,9012 gold badges15 silver badges23 bronze badges

1 Comment

kimpster Over a year ago

Doesn't the function of WebDriverWait fulfil do that already? Cause when i run the code it writes to the file just fine with variable x=1. From my understanding the code waits for the new element to become available then arrives to the finally clause and breaks out of the try and finally clause into the next line

Collectives™ on Stack Overflow

Selenium timeout exception even after loop is executed

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related