Extract data from HTML table using Python

Question

I want to extract data from HTML table using Python script and save it as variables(that I can later use in same script after loading them in if they exist) into a separate file. Also I want the script to ignore the first row of table(Component, Status, Time / Error). I would prefer not to use external libraries.

The output into a new file should be like so:

SAVE_DOCUMENT_STATUS = "OK"
SAVE_DOCUMENT_TIME = "0.408"
GET_DOCUMENT_STATUS = "OK"
GET_DOCUMENT_TIME = "0.361"
...

And heres the input to the script:

<table border=1>
<tr>
<td><b>Component</b></td>
<td><b>Status</b></td>
<td><b>Time / Error</b></td>
</tr>
<tr><td>SAVE_DOCUMENT</td><td>OK</td><td>0.408 s</td></tr>
<tr><td>GET_DOCUMENT</td><td>OK</td><td>0.361 s</td></tr>
<tr><td>DVK_SEND</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>DVK_RECEIVE</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>GET_USER_INFO</td><td>OK</td><td>0.135 s</td></tr>
<tr><td>NOTIFICATIONS</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>ERROR_LOG</td><td>OK</td><td>0.001 s</td></tr>
<tr><td>SUMMARY_STATUS</td><td>OK</td><td>0.913 s</td></tr>
</table>

I tried to do it in bash, but since I need to compare *_TIME variables to maximum time, then it fails, because they're float numbers.

"One more thing, I don't want to use external libraries". Epic Fail. You really must use Beautiful Soup. It's the best for this kind of thing. — S.Lott
– S.Lott, Commented Jul 29, 2011 at 19:25
Ok, if there aren't any other way out I will use Beautiful Soup then:) — Marko
– Marko, Commented Jul 29, 2011 at 19:27

unutbu · Accepted Answer · 2011-07-29 19:46:15Z

4

Using lxml:

import lxml.html as lh

content='''\
<table border=1>
<tr>
<td><b>Component</b></td>
<td><b>Status</b></td>
<td><b>Time / Error</b></td>
</tr>
<tr><td>SAVE_DOCUMENT</td><td>OK</td><td>0.408 s</td></tr>
<tr><td>GET_DOCUMENT</td><td>OK</td><td>0.361 s</td></tr>
<tr><td>DVK_SEND</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>DVK_RECEIVE</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>GET_USER_INFO</td><td>OK</td><td>0.135 s</td></tr>
<tr><td>NOTIFICATIONS</td><td>OK</td><td>0.002 s</td></tr>
<tr><td>ERROR_LOG</td><td>OK</td><td>0.001 s</td></tr>
<tr><td>SUMMARY_STATUS</td><td>OK</td><td>0.913 s</td></tr>
</table>
'''
tree=lh.fromstring(content)
for key, status, t in zip(*[iter(tree.xpath('//td/text()'))]*3):
    print('''{k}_STATUS = "{s}"
{k}_TIME = "{t}"'''.format(k=key,s=status,t=t.rstrip(' s')))

yields

SAVE_DOCUMENT_STATUS = "OK"
SAVE_DOCUMENT_TIME = "0.408"
GET_DOCUMENT_STATUS = "OK"
GET_DOCUMENT_TIME = "0.361"
DVK_SEND_STATUS = "OK"
DVK_SEND_TIME = "0.002"
DVK_RECEIVE_STATUS = "OK"
DVK_RECEIVE_TIME = "0.002"
GET_USER_INFO_STATUS = "OK"
GET_USER_INFO_TIME = "0.135"
NOTIFICATIONS_STATUS = "OK"
NOTIFICATIONS_TIME = "0.002"
ERROR_LOG_STATUS = "OK"
ERROR_LOG_TIME = "0.001"
SUMMARY_STATUS_STATUS = "OK"
SUMMARY_STATUS_TIME = "0.913"

answered Jul 29, 2011 at 19:46

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

brandizzi Over a year ago

lxml is an external library, no? Depending on @Marko Python version, he ca use ElementTree instead, or - urgh.... - xml.dom.minidom

unutbu Over a year ago

@brandizzi: In the comments Marko expressed a willingness to use BeautifulSoup. I took that to mean a willingness to use any external library.

Marko Over a year ago

Here's my final code thanks to unutbu, I still used the external library:

if (not os.path.exists(Filename)):         download = urllib2.urlopen(MonitorURL)         data = download.read()         fileObj = open(Filename, "w")         tree = lh.fromstring(data)         for key, status, time in zip(*[iter(tree.xpath('//td/text()'))]*3):             fileObj.writelines('''{k}_STATUS = "{s}"\n'''.format(k=key,s=status))             fileObj.writelines('''{k}_TIME = "{t}"\n'''.format(k=key,t=time.rstrip(' s')))         fileObj.close()

brandizzi · Accepted Answer · 2011-07-29 19:41:07Z

2

Well, if your HTML document really has such a stable structure (which makes me scratch my head because it is pretty rare) you can use regexes:

>>> import re
>>> r = re.compile('<tr><td>(.*)</td><td>(.*)</td><td>(.*) s</td></tr>')

The regex below groups the values you want to show in the result. Then you use the sub() method of the object. If the text is in a variable (such as content) just execute it this way:

r.sub(r'\1_STATUS = "\2"\n\1_TIME = \3', content)

The result:

>>> print r.sub(r'\1_STATUS = "\2"\n\1_TIME = \3', content)
<table border=1>
<tr>
<td><b>Component</b></td>
<td><b>Status</b></td>
<td><b>Time / Error</b></td>
</tr>
SAVE_DOCUMENT_STATUS = "OK"
SAVE_DOCUMENT_TIME = 0.408
GET_DOCUMENT_STATUS = "OK"
GET_DOCUMENT_TIME = 0.361
DVK_SEND_STATUS = "OK"
DVK_SEND_TIME = 0.002
DVK_RECEIVE_STATUS = "OK"
DVK_RECEIVE_TIME = 0.002
GET_USER_INFO_STATUS = "OK"
GET_USER_INFO_TIME = 0.135
NOTIFICATIONS_STATUS = "OK"
NOTIFICATIONS_TIME = 0.002
ERROR_LOG_STATUS = "OK"
ERROR_LOG_TIME = 0.001
SUMMARY_STATUS_STATUS = "OK"
SUMMARY_STATUS_TIME = 0.913
</table>

Sure, there is a lot of garbage in the string yet, but it gives the idea :)

If your HTML documents are not that stable, however, you should really consider some XML parser or, better yet, BeautifulSoup, because it would be a heck of a job to process an unstably structured HTML file by hand.

answered Jul 29, 2011 at 19:41

brandizzi

27.4k9 gold badges111 silver badges171 bronze badges

3 Comments

SingleNegationElimination Over a year ago

-1 stackoverflow.com/questions/1732348/…

Marko Over a year ago

The HTML I gave is stable :) The script is needed for making this old HTML edible to Nagios.

brandizzi Over a year ago

@TokenMacGuy although I agree in general, I am with this answer in this case :)

Collectives™ on Stack Overflow

Extract data from HTML table using Python

2 Answers 2

3 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related