The script I am running does not return consistent data from run to run. I believe as I iterate through pages, it is not waiting for all pages to load Javascript and AJAX in full. In an attempt to remedy this I added the explicit wait below, but it returns the following error:
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Chrome()
url = 'http://www.website.com'
browser.get(url)
try:
element = WebDriverWait(browser, 10).until(EC.presence_of_all_elements_located((By.ID, "results-main")))
finally:
browser.quit()
print (browser.page_source)
An implicit wait will run without errors, but it also does not return consistent source code.
browser.quit(). Note that block of code afterfinallyoperator executes always. So you might need to executebrowser.page_sourcebeforebrowser.quit()