Selenium & Python: how to find a list of buttons and click them in a loop

Question

There is a table with some data I need to scrape, but first I need to click on a button (inside the table) for each line so I can go to another frame and then pick the data (more detailed one).

After that, I use driver.back() and click on the next button, but I'm not getting it.

Here is what I tried to obtain a list of buttons and then build a loop:

list_buttons_consult_cri = WebDriverWait(driver, 10).until(EC.presence_of_elements_located((By.XPATH, "//a[@class='btn btn-black btn--small btn-registro']")))

but it's not working, it gives me a list with [0] elements.

Also tried with By.CLASS to find the elements however

when I try this, I can get through the first button:

button_cri = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//a[@class='btn btn-black btn--small btn-registro']")))

driver.execute_script("arguments[0].click();", button_cri)

but my interest is to build a loop to go and click for each button, once at a time.

Here is some of the html (with the two first buttons in the a tags):

<table id="resultEmissaoCri" class="table table--black">
    <thead>
        <tr>
            <th scope="col" class="no-break">Companhia Emissora</th>
            <th scope="col" class="no-break">Agente Fiduciário</th>
            <th scope="col" class="break-column">Nº Emissão</th>
            <th scope="col" class="no-break">Tipo de Oferta </th>
            <th scope="col" class="no-break">Código CETIP</th>
            <th scope="col" class="break-column">Nº Série</th>
            <th scope="col" class="no-break">Tipo de Série</th>
            <th scope="col" class="no-break">Vl. Global Série<br>(R$ em milhões)</th>
            <th scope="col" class="no-break">Vl. Total Oferta<br>(R$ em milhões)</th>
            <th scope="col" class="break-column">Data de Emissão</th>
            <th scope="col" class="no-break">Identificação do CRI</th>
            <th scope="col" class="no-break"></th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td data-label="Companhia Emissora" data-type="fantasiaEmissor">RB CAPITAL COMPANHIA DE SECURITIZACAO</td>
            <td data-label="Agente Fiduciário" data-type="nomeAgenteFiduciario">OLIVEIRA TRUST DISTRIBUIDORA DE TÍTULOS E VALORES MOBILIÁRIOS S.A.</td>
            <td data-label="Nº Emissão" data-type="numEmissao">1</td>
            <td data-label="Tipo de Oferta " data-type="indTipoOferta">Pública de Esforços Restritos</td>
            <td data-label="Código CETIP" data-type="codCETIP"></td>
            <td data-label="Nº Série" data-type="numSerie">296</td>
            <td data-label="Tipo de Série" data-type="indTipoCRI">Série Única</td>
            <td data-label="Vl. Global Série (R$ em milhões)" data-type="valorGlobalSerie" class="align-right">21,54</td>
            <td data-label="Vl. Total Oferta (R$ em milhões)" data-type="valorTotalOferta" class="align-right">21,53</td>
            <td data-label="Data de Emissão" data-type="dataEmissao">06/11/2020</td>
            <td data-label="Identificação do CRI" data-type="descrIdentificacaoCRI" class="break-row">1407</td>
            <td data-label="" data-type="button">
                <div class="table__buttons"><a class="btn btn-black btn--small btn-registro" title="Dados Emissão" href="/emissao/1407/1/296"><i class="fa fa-file-text-o" aria-hidden="true"></i></a><a class="btn btn-black btn--small btn-dados" title="Dados Periódicos" href="/periodica/1407/1/296"><i class="fa fa-calendar" aria-hidden="true"></i></a></div>
          
        </td>
    </tr>
    <tr>
        <td data-label="Companhia Emissora" data-type="fantasiaEmissor">RB CAPITAL COMPANHIA DE SECURITIZACAO</td>
        <td data-label="Agente Fiduciário" data-type="nomeAgenteFiduciario">VORTX DTVM LTDA</td>
        <td data-label="Nº Emissão" data-type="numEmissao">1</td>
        <td data-label="Tipo de Oferta " data-type="indTipoOferta">Pública de Esforços Restritos</td>
        <td data-label="Código CETIP" data-type="codCETIP"></td>
        <td data-label="Nº Série" data-type="numSerie">303</td>
        <td data-label="Tipo de Série" data-type="indTipoCRI">Série Única</td>
        <td data-label="Vl. Global Série (R$ em milhões)" data-type="valorGlobalSerie" class="align-right">20,00</td>
        <td data-label="Vl. Total Oferta (R$ em milhões)" data-type="valorTotalOferta" class="align-right">20,00</td>
        <td data-label="Data de Emissão" data-type="dataEmissao">29/10/2020</td>
        <td data-label="Identificação do CRI" data-type="descrIdentificacaoCRI" class="break-row">1403</td>
        <td data-label="" data-type="button">
            <div class="table__buttons"><a class="btn btn-black btn--small btn-registro" title="Dados Emissão" href="/emissao/1403/1/303"><i class="fa fa-file-text-o" aria-hidden="true"></i></a><a class="btn btn-black btn--small btn-dados" title="Dados Periódicos" href="/periodica/1403/1/303"><i class="fa fa-calendar" aria-hidden="true"></i></a></div>
        </td>

and the link of html if someone wants: https://www.anbima.com.br/pt_br/informar/dados-de-emissao-de-cri.htm

marco · Accepted Answer · 2020-12-24 09:06:29Z

For each line you should get the button, click it, get your data and then come back. Here an example:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as ec  


table = WebDriverWait(driver, 10).until(ec.presence_of_element_located((By.XPATH, "//table")))
trs = WebDriverWait(table, 10).until(ec.presence_of_all_elements_located((By.XPATH, ".//tr")))
for i in range(len(trs)):
    table = WebDriverWait(driver, 10).until(ec.presence_of_element_located((By.XPATH, "//table")))
    trs = WebDriverWait(table, 10).until(ec.presence_of_all_elements_located((By.XPATH, ".//tr")))
    tds = WebDriverWait(trs[i], 10).until(ec.presence_of_all_elements_located((By.XPATH, ".//td")))
    last_td = tds[-1]
    button = WebDriverWait(last_td, 10).until(ec.element_to_be_clickable((By.XPATH, ".//a[@class='btn btn-black btn--small btn-registro']")))
    button.click()
    #get your data
    driver.back()

Ryan Burch · Accepted Answer · 2020-12-23 20:17:34Z

What I would recommend doing is creating a list of the buttons you wish to click, by first creating a variable which finds all the elements by xpath. Then, have a for loop which goes through that list to find the button class by get_attribute.

You will then need to grab the keys from that list and put them into a variable. Lastly, have another for loop which goes through each item in the list and does what you wish. See the example below.

buttons = []
button_cri = driver.find_elements(By.XPATH, '(//a[@class="btn btn-black btn--small btn-registro"])')

for i in range(len(button_cri)):
    buttons.append(button_cri[i].get_attribute("button"))

all_buttons = list(dict.fromkeys(buttons))

for b in all_buttons:
    driver.get(b)
    *Rest of your code here*

I have made some assumptions in the above code. What might help is seeing your full code of what you are currently trying.

Thanks

Arundeep Chohan · Accepted Answer · 2020-12-24 05:29:08Z

1

To get all 11 buttons to click and go back do the following.

driver.get("https://www.anbima.com.br/pt_br/informar/dados-de-emissao-de-cri.htm")  
wait=WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#LGPD_ANBIMA_global_sites__text__btn"))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CLASS_NAME, 'full')))
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.btn.btn-black.right"))).click()
i=1
while True:
    try:
        #print("/html/body/div[1]/div/div/div/main/table/tbody/tr[{}]/td[12]/div/a[1]".format(i))
        elem=wait.until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[1]/div/div/div/main/table/tbody/tr[{}]/td[12]/div/a[1]".format(i))))
        driver.execute_script("arguments[0].click();", elem)
        i+=1
        driver.back()
        wait.until(EC.frame_to_be_available_and_switch_to_it((By.CLASS_NAME, 'full')))
    except Exception as e:
        print(e)
        break

This will do the following:

Handle popup
Switch to iframe
Open up table
Loop for all buttons

Imports

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

answered Dec 24, 2020 at 5:29

Arundeep Chohan

9,9895 gold badges17 silver badges36 bronze badges

1 Comment

Ramon Over a year ago

Thank you very much! It worked fine. Not just for moving through buttons but also moving through pages as well

KunduK · Accepted Answer · 2020-12-23 20:55:03Z

0

To get list of elements you have to use presence_of_all_elements_located()

list_buttons_consult_cri = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, "//a[@class='btn btn-black btn--small btn-registro']")))

answered Dec 23, 2020 at 20:55

KunduK

33.4k5 gold badges19 silver badges42 bronze badges

Collectives™ on Stack Overflow

Selenium & Python: how to find a list of buttons and click them in a loop

4 Answers 4

Comments

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related