XML Unicode strings with encoding declaration are not supported

Question

Trying to do the following...

from lxml import etree
from lxml.etree import fromstring

if request.POST:
    parser = etree.XMLParser(ns_clean=True, recover=True)
    h = fromstring(request.POST['xml'], parser=parser)
    return HttpResponse(h.cssselect('itagg_delivery_receipt status').text_content())

but it give this error:

[Fri Apr 05 10:27:54 2013] [error] Internal Server Error: /sms/status_postback/
[Fri Apr 05 10:27:54 2013] [error] Traceback (most recent call last):
[Fri Apr 05 10:27:54 2013] [error]   File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 115, in get_response
[Fri Apr 05 10:27:54 2013] [error]     response = callback(request, *callback_args, **callback_kwargs)
[Fri Apr 05 10:27:54 2013] [error]   File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 77, in wrapped_view
[Fri Apr 05 10:27:54 2013] [error]     return view_func(*args, **kwargs)
[Fri Apr 05 10:27:54 2013] [error]   File "/srv/project/livewireSMS/sms/views.py", line 42, in update_delivery_status
[Fri Apr 05 10:27:54 2013] [error]     h = fromstring(request.POST['xml'], parser=parser)
[Fri Apr 05 10:27:54 2013] [error]   File "lxml.etree.pyx", line 2754, in lxml.etree.fromstring (src/lxml/lxml.etree.c:54631)
[Fri Apr 05 10:27:54 2013] [error]   File "parser.pxi", line 1569, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:82659)
[Fri Apr 05 10:27:54 2013] [error] ValueError: Unicode strings with encoding declaration are not supported.

this is the XML

 <?xml version="1.1" encoding="ISO-8859-1"?>
<itagg_delivery_receipt>
<version>1.0</version>
<msisdn>447889000000</msisdn>
<submission_ref>
845tgrgsehg394g3hdfhhh56445y7ts6</
submission_ref>
<status>Delivered</status>
<reason>4</reason>
<timestamp>20050709120945</timestamp>
<retry>0</retry>
</itagg_delivery_receipt>

I don't have control over the xml document this comes from the SMS company.

From the lxml FAQ: Why can't lxml parse my XML from unicode strings? — Stevoisiak
– Stevoisiak, Commented Feb 13, 2018 at 17:06
Possible duplicate of parsing XML file gets UnicodeEncodeError (ElementTree) / ValueError (lxml) — Stevoisiak
– Stevoisiak, Commented Feb 13, 2018 at 17:14

Pavel Anossov · Accepted Answer · 2013-04-05 09:41:30Z

45

You'll have to encode it and then force the same encoding in the parser:

from lxml import etree
from lxml.etree import fromstring

if request.POST:
    xml = request.POST['xml'].encode('utf-8')
    parser = etree.XMLParser(ns_clean=True, recover=True, encoding='utf-8')
    h = fromstring(xml, parser=parser)

    return HttpResponse(h.cssselect('delivery_reciept status').text_content())

answered Apr 5, 2013 at 9:41

Pavel Anossov

63.3k16 gold badges156 silver badges125 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Stevoisiak Over a year ago

Can a standalone example be provided where the XML data is defined in the code?

mokko Over a year ago

this solution doesn't help with encoding problems.

Scott · Accepted Answer · 2020-12-02 04:42:21Z

24

The following solution from kernc worked for me:

from lxml import etree

xml = u'<?xml version="1.0" encoding="utf-8" ?><foo><bar/></foo>'
xml = bytes(bytearray(xml, encoding='utf-8'))  # ADDENDUM OF THIS LINE (when unicode means utf-8, e.g. on Linux)
etree.XML(xml)

# <Element html at 0x5b44c90>

edited Dec 2, 2020 at 4:42

Scott

5,9468 gold badges48 silver badges79 bronze badges

answered Jul 7, 2016 at 11:24

Aykut Kllic

94710 silver badges15 bronze badges

1 Comment

Pintun Over a year ago

Interesting. Even after reading the page suggesting the workaround, I sill miss why decoding fails... :S

Ryan · Accepted Answer · 2021-10-24 13:08:54Z

24

3 Comments

HansHirse Over a year ago

You might add some explanations to your solution, so that later users can follow your code more easily.

Kelly Bang Over a year ago

What type is r ?

Tim_h11 Over a year ago

The variable r would be from the request library. So you would have r = requests.get(URL).

Collectives™ on Stack Overflow

XML Unicode strings with encoding declaration are not supported

3 Answers 3

2 Comments

1 Comment

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related