0

Yes i know that this type of question has been answered many times before, but none of them helped me. Actually i didn't know much about it so need your help!

My problem: I am scraping through a website and it needs a CAPTCHA to search for every input. So i use FireFox as my browser as it asks for the captcha one time and doesn't change it. My code asks the user for CAPTCHA one time and then click on search button and tries to scrape the data, but when it clicks on the search button again (as it is in a loop) then it raises this error:

selenium.common.exceptions.StaleElementReferenceException: 
Message: The element reference of <input id="txt_ALPHA_NUMERIC" class="ui-inputfield ui-inputtext ui-widget ui-state-default ui-corner-all" name="txt_ALPHA_NUMERIC" type="text"> is stale; 
either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed

My old code:

from selenium import webdriver                    # Import module 
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys   # For keyboard keys 
import time    
import pandas as pd

URL = 'https://vahan.nic.in/nrservices/faces/user/searchstatus.xhtml'      # Define URL 
browser = webdriver.Firefox(executable_path=r'C:\Users\intel\Downloads\Setups\geckodriver.exe')  

browser.get(URL)

vehicle_no = browser.find_element_by_xpath("""//*[@id="regn_no1_exact"]""")
vehicle_no.send_keys('RJ14CX3238')

captcha_input = input("enter your captcha ")
captcha = browser.find_element_by_xpath("""//*[@id="txt_ALPHA_NUMERIC"]""")
captcha.send_keys(captcha_input)
button_click = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[5]/div/button/span").click()    

i = 111
attempt = 1
max_attempts = 2
while True: 
    i = i + 1
    time.sleep(4)
    reg_no = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[6]/div/div/div/table/tbody/tr[2]/td[2]/span").text
    date = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[6]/div/div/div/table/tbody/tr[2]/td[4]").text 

    vehicle_no = browser.find_element_by_xpath("""//*[@id="regn_no1_exact"]""")
    vehicle_no.send_keys('RJ14CX3' + str(i))

    captcha.send_keys(captcha_input)

    button_click = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[5]/div/button/span").click()    
    browser.execute_script("return arguments[0].scrollIntoView(true);", button_click)

Updated new code now:

from selenium import webdriver                    # Import module 
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys   # For keyboard keys 
import time    
import pandas as pd
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

URL = 'https://vahan.nic.in/nrservices/faces/user/searchstatus.xhtml'      # Define URL 
browser = webdriver.Firefox(executable_path=r'C:\Users\intel\Downloads\Setups\geckodriver.exe')  

browser.get(URL)

vehicle_no = browser.find_element_by_xpath("""//*[@id="regn_no1_exact"]""")
vehicle_no.send_keys('RJ14CX3238')

captcha_input = input("enter your captcha ")
captcha = browser.find_element_by_xpath("""//*[@id="txt_ALPHA_NUMERIC"]""")
captcha.send_keys(captcha_input)
button_click = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[5]/div/button/span").click()    

i = 111
while True: 
    button_click = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[5]/div/button/span")
    WebDriverWait(browser, 10).until_not(EC.visibility_of_element_located((By.ID, "overley")))
    browser.execute_script("return arguments[0].scrollIntoView(true);", button_click)

    i = i + 1
    #reg_no = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[6]/div/div/div/table/tbody/tr[2]/td[2]/span").text
    #date = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[6]/div/div/div/table/tbody/tr[2]/td[4]").text 
    time.sleep(5)
    vehicle_no.send_keys('RJ14CX3' + str(i))
    WebDriverWait(browser, 10).until_not(EC.visibility_of_element_located((By.ID, "overley")))
    captcha.send_keys(captcha_input)

Also fix any other problems if is in my code. Any help would be appreciated!! Thanks in advance.

3
  • 1
    Your xpath seems absolute. Did you try relative xpath? Commented Feb 6, 2020 at 17:17
  • 1
    Could you specify exactly which line the error is being thrown at? I am assuming captcha.send_keys(captcha_input). Looks like the reference to captcha is stale since you queried it by @id="txt_ALPHA_NUMERIC" Commented Feb 6, 2020 at 17:23
  • Yes it's captcha.send_keys(captcha_input) and I don't know how to solve it... @snnguyen Commented Feb 7, 2020 at 1:16

1 Answer 1

1

Simply re-find the button element in the loop, each time, rather than before the loop starts. Any time the DOM mutates, previous references are marked as stale, and will require a new instance. Interacting with Captcha's mutate the DOM, and mark the page as dirty (having changed/modified), which Selenium uses to flag "staleness".

Sign up to request clarification or add additional context in comments.

3 Comments

Can you please provide the code. Because I didn't get it @Asyranok. It will be huge help from your side...
This "button_click = browser.find_element_by_xpath("/html/body/form/div[1]/div[3]/div/div[2]/div/div/div[2]/div[5]/div/button/span").click() " put this line of code under "while True: " and before "i = i + 1"
I did that also, with some more useful changes but at the end it again produces the same error. Please help me with code fixes. I have also updated my code. @Asyranok

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.