Copy table using Selenium and Python

Question

I have some Python code that copies a table from a website using Selenium,to create a csv, but as I haven't used Selenium much, I have that irksome feeling in the back of my mind that there must be a better way. It's also quite a bit slower than I would like. Here's the relevant code:

# ...Navigate to proper page...

table = self.browser.find_element_by_id('data_table')
head = table.find_element_by_tag_name('thead')
body = table.find_element_by_tag_name('tbody')

file_data = []

file_header = []
head_line = head.find_element_by_tag_name('tr')
headers = head_line.find_elements_by_tag_name('th')
for header in headers:
    header_text = header.text.encode('utf8')
    file_header.append(header_text)
file_data.append(",".join(file_header))

body_rows = body.find_elements_by_tag_name('tr')
for row in body_rows:
    data = row.find_elements_by_tag_name('td')
    file_row = []
    for datum in data:
        datum_text = datum.text.encode('utf8')
        file_row.append(datum_text)
    file_data.append(",".join(file_row))

with open(srcFile, "w") as f:
    f.write("\n".join(file_data))

Ethan Bierlein · Accepted Answer · 2015-07-03 02:38:42Z

First off, I see a couple of things that can be shortened to generator expressions, rather than full-blown for loops. For example, this section:

file_header = []
head_line = head.find_element_by_tag_name('tr')
headers = head_line.find_elements_by_tag_name('th')
for header in headers:
    header_text = header.text.encode('utf8')
    file_header.append(header_text)
file_data.append(",".join(file_header))

Can be shortened immensely to the following:

head_line = head.find_element_by_tag_name("tr")
file_header = [header.text.encode("utf8") for header in head_line.find_elements_by_tag_name('th')]
file_data.append(",".join(file_header))

Finally, your other for loop can be shortened to a generator expression as well. For more on generator expressions, see PEP0289.

Stack Exchange Network

Copy table using Selenium and Python

1 Answer 1

You must log in to answer this question.

Hot Network Questions

Copy table using Selenium and Python

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions