Custom For Loop in Python

Question

Is there any pythonic way to do the same as code bellow, but in a pythonic way?

I created this code for web scraping a website, but I think there should be a better way for adding the contents to lists other than repeating the same code for each element.

here are the lists i will add elements to:

Proporcao_de_Sobras = []
liq_dir =[]
liq_sobras=[]
liq_reservas=[]
Encerramento=[]
n_emissao =[]
tp_ofert =[]
inv_minimo =[]

And here is the code I am using to add the elements to lists.

try:
    for x in driver.find_elements_by_xpath('//*[@id="tablepress-6"]/tbody/tr[*]/td/span[2]'):
        Proporcao_de_Sobras.append(x.text)
except:
    pass

try:
    for x in driver.find_elements_by_xpath('//*[@id="tablepress-6"]/tbody/tr[*]/td/span[4]'):
        liq_dir.append(x.text)
except:
    pass

try:
    for x in driver.find_elements_by_xpath('//*[@id="tablepress-6"]/tbody/tr[*]/td/span[6]'):
        liq_sobras.append(x.text)
except:
    pass

try:
    for x in driver.find_elements_by_xpath('//*[@id="tablepress-6"]/tbody/tr[*]/td/span[8]'):
        liq_reservas.append(x.text)
except:
    pass

try:
    for x in driver.find_elements_by_xpath('//*[@id="tablepress-6"]/tbody/tr[*]/td/span[10]'):
        Encerramento.append(x.text)
except:
    pass

try:
    for x in driver.find_elements_by_xpath('//*[@id="tablepress-6"]/tbody/tr[*]/td/span[12]'):
        n_emissao.append(x.text)
except:
    pass

try:
    for x in driver.find_elements_by_xpath('//*[@id="tablepress-6"]/tbody/tr[*]/td/span[14]'):
        tp_ofert.append(x.text)
except:
    pass

try:
    for x in driver.find_elements_by_xpath('//*[@id="tablepress-6"]/tbody/tr[*]/td/span[16]'):
        inv_minimo.append(x.text)
except:
    pass

This goes on for more 5 or 6 times.

You can do a list of your lists [Proporcao_de_Sobras, liq_dir, ...] and iterate over it. And you can use their index to set your x value in span[x]. Using x = index*2 + 2. You can get index using enumerate() — Clément
– Clément, Commented Aug 12, 2020 at 12:48
except: pass will bite you at some point and waste much time. Happens to everyone. If you have reasons to believe some exception(s) may be thrown, specify ex: except KeyError, IndexError): pass — JL Peyret
– JL Peyret, Commented Aug 13, 2020 at 17:10

Mandel Warrior · Accepted Answer · 2020-08-14 01:07:36Z

3

Here's another pythonic way using dictionaries:

def get_data(your_lists):
    data = {}
    for list_index, list_name in enumerate(your_lists):
        try:
            data[list_name] = [x for x in find_elements_by_xpath(f'//*[@id="tablepress-6"]/tbody/tr[*]/td/span[{(list_index + 1) * 2}]')]
        except:
            pass
    return data

your_lists = ['Proporcao_de_Sobras', 'liq_dir', 'loq_reservas', 'Encerramento', 'n_emissao', 'tp_ofert', 'inv_minimo']

all_data = get_data(your_lists)

edited Aug 14, 2020 at 1:07

Mandel Warrior

3103 silver badges6 bronze badges

answered Aug 12, 2020 at 13:25

Ivan Litteri

581 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Raymond Reddington · Accepted Answer · 2020-08-12 12:47:14Z

2

Pythonic way N1, using mutability of lists:

def get_text(x_path, dest_list):
    for x in driver.find_elements_by_xpath(x_path):
        dest_list.append(x.text)

Proporcao_de_Sobras = []
get_text('//*[@id="tablepress-6"]/tbody/tr[*]/td/span[2]', Proporcao_de_Sobras)

Pythonic way N2, using dicts:

paths = {
    '//*[@id="tablepress-6"]/tbody/tr[*]/td/span[2]': [],
    '//*[@id="tablepress-6"]/tbody/tr[*]/td/span[4]': [],
    ....
}

for k, v in paths.items():
    for x in driver.find_elements_by_xpath(k):
        v.append(x.text)

answered Aug 12, 2020 at 12:47

Raymond Reddington

1,8471 gold badge15 silver badges21 bronze badges

Comments

dh762 · Accepted Answer · 2020-08-12 12:51:07Z

2

You can use a function for it. I would advise to catch specific exceptions though.

def fill_elem(fill_list, xpath):
    try:
        for x in driver.find_elements_by_xpath(xpath)
            fill_list.append(x.text)
    except SomeException:
        pass
    else:
        return fill_list

proporcao_de_sobras = []
proporcao_de_sobras = fill_elem(proporcao_de_sobras, r'//*[@id="tablepress-6"]/tbody/tr[*]/td/span[2]')

answered Aug 12, 2020 at 12:51

dh762

2,4364 gold badges27 silver badges48 bronze badges

Collectives™ on Stack Overflow

Custom For Loop in Python

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related