Use Python to parse html data which contains "&"

Question

I'm using the python library SGMLParser to parse some html. I encounter an html tag of the form

<td class="school">Texas A&amp;M</td>

I'd like to read out "Texas A&M". But when handle_data gets called, it gets called with "Texas A", and then, separately, "M" (quotes for clarity).

How do I replace the

&amp;

string with an & before the call, without replacing all special ampersands in the whole string (some of which I may need).

Thanks!

Fred Foo · Accepted Answer · 2011-09-14 12:57:07Z

4

If you switch from the deprecated SGMLParser to a modern alternative such as LXML (which also handles HTML), this becomes trivial:

>>> etree.fromstring('''<td class="school">Texas A&amp;M</td>''').text
'Texas A&M'

answered Sep 14, 2011 at 12:57

Fred Foo

365k80 gold badges765 silver badges852 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

phihag Over a year ago

SGMLParser is being deprecated because nobody cares about SGML (and most people use it to parse HTML, case in point). XMLParser has the same interface and is not being deprecated. lxml should really go into the stdlib.

mdeland Over a year ago

Yes, I didn't care about SGML either, it just seemed like an "easy" way to read data from html. I will look into lxml, thanks.

plaes · Accepted Answer · 2011-09-14 12:59:52Z

2

SGMLParser has convert_entityref() method, but instead of deprecated SGMLParser I would recommend using lxml or Beautiful Soup which have better parser API.

answered Sep 14, 2011 at 12:59

plaes

32.9k11 gold badges95 silver badges89 bronze badges

Comments

phihag · Accepted Answer · 2011-09-14 12:57:40Z

1

Entity references like & are handled by handle_entity. Check that this method knows how to translate &. The default implementation should call handle_data('&'), but you may have accidentally overwritten it.

Also, if possible, consider using the far more advanced lxml instead.

answered Sep 14, 2011 at 12:57

phihag

289k75 gold badges475 silver badges489 bronze badges

2 Comments

mdeland Over a year ago

I don't think I overwrote that... but then handle_data gets called three times with 'Texas A', '&', and 'M' right? Is there a way to have the data joined (if you know what I mean)? It looks like everyone suggests lxml, so I will look into it.

phihag Over a year ago

@mdeland Precisely. You have to join the data yourself; SGMLParser is a very low-level interface.

Collectives™ on Stack Overflow

Use Python to parse html data which contains "&"

3 Answers 3

2 Comments

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related