Using Selenium and python to save a table

Question

I'm trying to use Selenium with Python to store the contents of a table. My script is as follows:

import sys
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("http://testsite.com")

value = selenium.getTable("table_id_10")

print value

driver.close()

This opens up the webpage I am interested in, and then should save the contents of the table that I want. I've seen the syntax in this question which uses browser.get_table(), but the beginning of that program begins with browser=Selenium(...) which I did not understand. I'm not sure what syntax I should be using as selenium.getTable("table_id_10") is incorrect.

EDIT:

I included a html snippet of the table that I am using:

<table class="datatable" cellspacing="0" rules="all" border="1" id="table_id_10" style="width:70%;border-collapse:collapse;">
    <caption>
        <span class="captioninformation right"><a href="Services.aspx" class="functionlink">Return to Services</a></span>Data
    </caption><tr>
        <th scope="col">Read Date</th><th class="numericdataheader" scope="col">Days</th><th class="numericdataheader" scope="col">Values</th>

    </tr><tr>
        <td>10/15/2011</td><td class="numericdata">92</td><td class="numericdata">37</td>
    </tr><tr class="alternaterows">
        <td>7/15/2011</td><td class="numericdata">91</td><td class="numericdata">27</td>
    </tr><tr>
        <td>4/15/2011</td><td class="numericdata">90</td><td class="numericdata">25</td>    
</table>

If you're looking to web-scrape (I think that's what you're doing), you might also want to look at mechanize. I've used it in the past and really like it, but unfortunately the documentation is lacking quite a bit and it's a little difficult to use. Just a thought, hope it isn't too off topic. — Naftuli Kay
– Naftuli Kay, Commented Nov 15, 2011 at 22:57
@TKKocheran I am interested in scraping, though in this case it is just one table. I would also be fine saving the html page and parsing it separately later. — djq
– djq, Commented Nov 15, 2011 at 23:00
You might want to look into using mechanize then. Once you get the hang of it, mechanize is uber-powerful at doing stuff. I once wrote a script which would log into my bank account, answer a security question, then proceed to grab my financial data using a form available in the banking application. Fun stuff. — Naftuli Kay
– Naftuli Kay, Commented Nov 15, 2011 at 23:06

unutbu · Accepted Answer · 2016-04-05 22:09:28Z

17

The old Selenium RC API included a get_table method:

In [14]: sel=selenium.selenium("localhost",4444,"*firefox", "http://www.google.com/webhp")
In [19]: sel.get_table?
Type:       instancemethod
Base Class: <type 'instancemethod'>
String Form:    <bound method selenium.get_table of <selenium.selenium.selenium object at 0xb728304c>>
Namespace:  Interactive
File:       /usr/local/lib/python2.7/dist-packages/selenium/selenium.py
Definition: sel.get_table(self, tableCellAddress)
Docstring:
    Gets the text from a cell of a table. The cellAddress syntax
    tableLocator.row.column, where row and column start at 0.

    'tableCellAddress' is a cell address, e.g. "foo.1.4"

Since you are using the newer Webdriver (a.k.a Selenium 2) API, that code does not apply.

Perhaps try something like this instead:

import selenium.webdriver as webdriver
import contextlib

@contextlib.contextmanager
def quitting(thing):
    yield thing
    thing.close()
    thing.quit()

with quitting(webdriver.Firefox()) as driver:
    driver.get(url)
    data = []
    for tr in driver.find_elements_by_xpath('//table[@id="table_id_10"]//tr'):
        tds = tr.find_elements_by_tag_name('td')
        if tds: 
            data.append([td.text for td in tds])
print(data)
# [[u'10/15/2011', u'92', u'37'], [u'7/15/2011', u'91', u'27'], [u'4/15/2011', u'90', u'25']]

edited Apr 5, 2016 at 22:09

answered Nov 15, 2011 at 21:26

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

djq Over a year ago

Thanks for explaining why get_table did not work. I added a snippet of the html table - I'm trying to pull out all of the values from the cells.

RattleyCooper Over a year ago

+1 for including contextlib. I didn't realize this was a thing until now. I'm going to have to convert all my scripts to work this way :D

unutbu Over a year ago

@DuckPuncher: For some versions of webdriver, you may need to call both close and quit. I've updated the code above to show a context manager which does both.

RattleyCooper Over a year ago

@unutbu, this is great!

Collectives™ on Stack Overflow

Using Selenium and python to save a table

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related