Python's libxml2 can't parse unicode strings

Question

OK, the docs for Python's libxml2 bindings are really ****. My problem:

An XML document is stored in a string variable in Python. The string is a instance of Unicode, and there are non-ASCII characters in it. I want to parse it with libxml2, looking something like this:

# -*- coding: utf-8 -*-
import libxml2

DOC = u"""<?xml version="1.0" encoding="UTF-8"?>
<data>
  <something>Bäääh!</something>
</data>
"""

xml_doc = libxml2.parseDoc(DOC)

with this result:

Traceback (most recent call last):
  File "test.py", line 13, in <module>
    xml_doc = libxml2.parseDoc(DOC)
  File "c:\Python26\lib\site-packages\libxml2.py", line 1237, in parseDoc
    ret = libxml2mod.xmlParseDoc(cur)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 46-48:
ordinal not in range(128)

The point is the u"..." declaration. If I replace it with a simple "..", then everything is ok. Unfortunately it doesn't work in my setup, because DOC will definitely be a Unicode instance.

Has anyone an idea how libxml2 can be brought to parse UTF-8 encoded strings?

u"..." is a unicode string, its internal encoding is not something you care about, it's not UTF-8. — u0b34a0f6ae
– u0b34a0f6ae, Commented Oct 14, 2009 at 21:48

u0b34a0f6ae · Accepted Answer · 2009-10-14 21:49:58Z

9

It should be

# -*- coding: utf-8 -*-
import libxml2

DOC = u"""<?xml version="1.0" encoding="UTF-8"?>
<data>
  <something>Bäääh!</something>
</data>
""".encode("UTF-8")

xml_doc = libxml2.parseDoc(DOC)

The .encode("UTF-8") is needed to get the binary representation of the unicode string with the utf8 encoding.

edited Oct 14, 2009 at 21:49

u0b34a0f6ae

50k14 gold badges97 silver badges102 bronze badges

answered Oct 14, 2009 at 21:35

Peter Hoffmann

59.1k15 gold badges78 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Boldewyn Over a year ago

Thanks for the answer, it works perfectly. Andrey was first, though.

Andrey Vlasovskikh · Accepted Answer · 2009-10-15 11:39:09Z

6

XML is a binary format, despite of looking like a text. An encoding is specified in the beginning of the XML file in order to decode the XML bytes into the text.

What you should do is to pass str, not unicode to your library:

xml_doc = libxml2.parseDoc(DOC.encode("UTF-8"))

(Although some tricks are possible with site.setencoding if you are interested in reading or writing unicode strings with automatic conversion via locale.)

Edit: The Unicode article by Joel Spolsky is good guide to string characters vs. bytes, encodings, etc.

edited Oct 15, 2009 at 11:39

answered Oct 14, 2009 at 21:34

Andrey Vlasovskikh

16.9k8 gold badges49 silver badges65 bronze badges

1 Comment

Boldewyn Over a year ago

Thanks for the answer! OK, I guess I have to think again about strings in Python (although it would be nice of libxml2, if it would accept basestring instances).

Collectives™ on Stack Overflow

Python's libxml2 can't parse unicode strings

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related