Looping through a list of XPaths using Regex, python

Question

Here is my code

src = driver.page_source
XPATHLoop=['//*[@id="General"]/fieldset/dl/dd[15]', '//*[@id="General"]/fieldset/dl/dd[14]', '//*[@id="General"]/fieldset/dl/dd[13]','//*[@id="General"]/fieldset/dl/dd[12]']
for d in XPATHLoop:
    Checkpath = re.search(d,src)
    if Checkpath =='//*[@id="General"]/fieldset/dl/dd[15]':
        Status= driver.find_element_by_xpath('//*[@id="General"]/fieldset/dl/dd[8]').text
        break
    elif  Checkpath == '//*[@id="General"]/fieldset/dl/dd[14]':
        Status= driver.find_element_by_xpath('//*[@id="General"]/fieldset/dl/dd[7]').text
        break
    elif Checkpath == '//*[@id="General"]/fieldset/dl/dd[13]':
        Status = driver.find_element_by_xpath('//*[@id="General"]/fieldset/dl/dd[6]').text
        break
    elif Checkpath == '//*[@id="General"]/fieldset/dl/dd[12]':
        Status= driver.find_element_by_xpath('//*[@id="General"]/fieldset/dl/dd[5]').text
        break
    else:
        Status= "NULL"
print(Status)

Output is 'NULL' meaning that it isn't finding anything, while these paths do/can exist in the source. I am currently using selenium and regex. I'm currently thinking that there may be another method in regex to pull the xpaths.

Have you tried using lxml ? lxml.de/xpathxslt.html or xml from the standard lib docs.python.org/3/library/… — JoseKilo
– JoseKilo, Commented Dec 12, 2019 at 15:45

Sers · Accepted Answer · 2019-12-12 15:57:46Z

2

To parse HTML and XML documents (page source) and get elements with locators, you can use beautifulsoup, how to use it.

Regular expression do not parsing HTML documents. You get NULL because Checkpath = re.search(d,src) is None.

Here is example how you can get status without loop and parsing page source.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

p = wait.until(EC.visibility_of_all_elements_located((By.XPATH, '//*[@id="General"]/fieldset/dl/dd')))

status = "NULL"

r = range(0, len(p))
if 15 in r:
    status = p[8].text
elif 14 in r:
    status = p[7].text
elif 13 in r:
    status = p[6].text
elif 12 in r:
    status = p[5].text

edited Dec 12, 2019 at 15:57

answered Dec 12, 2019 at 15:51

Sers

12.3k2 gold badges14 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

JeffC · Accepted Answer · 2019-12-12 17:05:45Z

1

From the code you have posted, the strings you have hardcoded are all the same other than the index of the DD element. Since the Status index is always 7 less than the index of Checkpath, you can just loop through 15-12, do your search, and then Status is just 15-7. The code is below.

src = driver.page_source
loop = [15,14,13,12]

for d in loop:
    Checkpath = re.search('//*[@id="General"]/fieldset/dl/dd[' + str(d) + ']',src)
    Status = driver.find_element_by_xpath('//*[@id="General"]/fieldset/dl/dd[' + str(d - 7) + ']').text
print(Status)

answered Dec 12, 2019 at 17:05

JeffC

26.4k5 gold badges35 silver badges56 bronze badges

Comments

SublimizeD · Accepted Answer · 2019-12-12 19:22:49Z

1

My solution to this bug is listed below I was able to perform a nest while loop within a for loop. My list XpathLoop shows the numerical values of the dd items. This loops through until it finds my desired dt 'Status Date'. After it finds the desired text it enteres a while loop until the status date format appears. The dd items are completely independent of the dt items on the webpage.

XPATHLoop=[4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]

for d in XPATHLoop:
  try:
  findtool = driver.find_element_by_xpath('//[@id="General"]/fieldset/dl/dt['+str(d)+']')
        if findtool.text == 'Status Date':
            Status=driver.find_element_by_xpath('//[@id="General"]/fieldset/dl/dd['+str((d))+']').text

            i=1
            while re.search(r'(\d{4})',Status) == None:
                Status= driver.find_element_by_xpath('//*[@id="General"]/fieldset/dl/dd['+str((d+i))+']').text
                i=i+1
  except:
     pass
  print(Status)

answered Dec 12, 2019 at 19:22

SublimizeD

1341 gold badge1 silver badge10 bronze badges

1 Comment

SublimizeD Over a year ago

@JeffC the idea of treating the list num as a variable definitely helped. Thanks a ton. I utilized it in a way that it now can navigate the page until it finds what i need it to. The while loop is very important as the the two lists are independent of one another.

Collectives™ on Stack Overflow

Looping through a list of XPaths using Regex, python

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related