Parse XHTML5 with undefined entities

Question

Please consider this:

import xml.etree.ElementTree as ET

xhtml = '''<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
        <html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
        <head><title>XHTML sample</title></head>
            <body>
                <p>&nbsp;Sample text</p>
            </body>
        </html>
'''
parser = ET.XMLParser()
parser.entity['nbsp'] = '&#x00A0;'
tree = ET.fromstring(xhtml, parser=parser)
print(ET.tostring(tree, method='xml'))

which renders nice text representation of xhtml string.

But, for same XHTML document with HTML5 doctype:

xhtml = '''<!DOCTYPE html>
        <html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
        <head><title>XHTML sample</title></head>
            <body>
                <p>&nbsp;Sample text</p>
            </body>
        </html>
'''

I get Exception:

xml.etree.ElementTree.ParseError: undefined entity: line 5, column 19

so the parser can't handle it, although I added nbsp to entities dict.

Same happens if I use lxml:

from lxml import etree
parser = etree.XMLParser(resolve_entities=False)
tree = etree.fromstring(xhtml, parser=parser)
print etree.tostring(tree, method='xml')

raises:

lxml.etree.XMLSyntaxError: Entity 'nbsp' not defined, line 5, column 26

although I've set the parser to ignore entities.

Why is this, and how to make parsing of XHTML files with HTML5 doctype declaration possible?

Partial solution for lxml is to use recoverer:

parser = etree.XMLParser(resolve_entities=False, recover=True)

but I'm still waiting for better one.

The problem is, XML doesn't have  , &nbsp is a HTML thing, not XML. Parsing HTML as as XML requires the use of XHTML5 (see: stackoverflow.com/questions/1415394/…) — Martin Tournoij
– Martin Tournoij, Commented Jul 7, 2014 at 8:04

Wladimir Palant · Accepted Answer · 2014-07-07 08:00:46Z

The problem here is, the Expat parser used behind the scenes won't usually report unknown entities - it will rather throw an error, so the fallback code in xml.etree.ElementTree you were trying to trigger won't even run. You can use the UseForeignDTD method to change this behavior, it will make Expat ignore the doctype declaration and pass all entity declarations to xml.etree.ElementTree. The following code works correctly:

import xml.etree.ElementTree as ET

xhtml = '''<!DOCTYPE html>
        <html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
        <head><title>XHTML sample</title></head>
            <body>
                <p>&nbsp;Sample text</p>
            </body>
        </html>
'''
parser = ET.XMLParser()
parser._parser.UseForeignDTD(True)
parser.entity['nbsp'] = u'\u00A0'
tree = ET.fromstring(xhtml, parser=parser)
print(ET.tostring(tree, method='xml'))

The side-effect of this approach: as I said, the doctype declaration is completely ignored. This means that you have to declare all entities, even the ones supposedly covered by the doctype.

Note that the values you put into ElementTree.XMLParser.entity dictionary have to be regular strings, text that the entity will be replaced by - you can no longer refer to other entities there. So it should be u'\u00A0' for  .

Collectives™ on Stack Overflow

Parse XHTML5 with undefined entities

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related