4

What does it actually do? On my very basic level of understanding XML is just a formatted text. So there is no binary<->text transformation involved.

I highly suspect that the only difference between UTF-8 and ASCII encoding is that ASCII encoding will make XML writer work harder by converting all the non-ASCII characters into XML entities as opposed to just reserved XML characters. So ASCII encoded XML can still contain UTF-8 characters, except it is going to be slightly longer and uglier.

Or is there some other function to it?

Update:

I perfectly understand how individual characters are converted into byte(s) by means of encoding. However XML is just text markup and at no point does that.

The question really is why XML encoding value is stored in the XML? Or what is the case where XML reader would need to know which encoding was used for any particular XML document?

Improve this question
3
  • 1
    There is no such thing as “UTF-8 characters”. You probably mean non-ASCII Unicode characters. And the reason for the declaration is that there is lots of different encodings, like windows-1250 or latin-2. UTF-8 and ASCII are not the only two options. Commented Sep 26, 2011 at 18:37
  • ok, but why does XML declaration need encoding in the first place? Commented Sep 26, 2011 at 19:39
  • 1
    Because when you want to read the file and, for example, display it to the user, you need to know how to decode the bytes into characters. Commented Sep 26, 2011 at 19:44

3 Answers 3

Reset to default
9
+50

See Appendix F in the XML specification, "Autodetection of Character Encodings".

In particular, "XML encoding value is stored in the XML" because, by default, XML processors must assume the content is in UTF-16 or UTF-8, in the absence of external metadata found outside of the XML document. The XML declaration is designed for such cases where such metadata is not present.

Another advantage to how XML handles encodings is that this way, an XML processor need support only two encodings, namely UTF-8 and UTF-16. If the processor discovers, either in external metadata or in the XML declaration, that the document is in an encoding it does not support, it can fail sooner than it would if it continues to read the document (long after the declaration) and encounters an unexpected byte sequence for the encoding detected using implementation-dependent heuristics.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

owww... that is actually kinda scary. So when reading a file (byte stream) XML encoding can be used as a hint to determine the file encoding. Seems awfully magical in terms of implementation. Though I suppose, since file encoding would have to support character set no smaller than one used for XML encoding, it would be a valid assumption to make.
It's less magical than the alternatives. If we read a comma separated .txt from disk, there's absolutely no standard way to definitively know what the encoding is, it has to be application-specific. With XML, I can provide you a doc in UTF-8, one in ISO-8859-1, and one in UTF-16, and with proper xml declarations, I can be 100% sure that your properly implemented xml api will have no trouble reading them.
@PeterO. What if the file is sent over http with Content-Type: text/xml; charset=ISO-8859-1 yet the xml document itself claims that it is utf-8 ?
@Pacerier: In that case, the HTTP header's content type declaration takes precedence over any character encoding declaration in the document itself. Because of this, the document may be misinterpreted. See Authoritative Metadata for more information: "Metadata received in an encapsulating container [such as HTTP headers] MUST be considered authoritative and used in preference to metadata found by inspection of the data [such as the document's self-declared encoding]."
4

I'd highly, HIGHLY recommend reading The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!). You're saying XML is "just text" as if that makes everything simple, but even knowing that it's text as opposed to some structured binary format doesn't mean you know exactly how to read it or what characters are therein.

This isn't a "go read the manual!" answer, I believe establishing this baseline on how difficult text can be will help explain why the XML declaration exists.

why does XML declaration need encoding in the first place?

This is one of the ideas addressed in the article, but it's worth stressing here: All text has an encoding. There is no such thing as 'Plain Text'. ASCII is an encoding, even if we don't think about it most of the time. Historically we've often stuck our head in the sand and assumed everything is ASCII, but this isn't feasible in today's day & age. The XML declaration's encoding helps us out, where has a .txt file has nothing to indicate what its encoding is.

Improve this answer

7 Comments

I still don't see how any of that is relevant to XML. From XML POV character is just a character. Does it have 1 byte 2 or 3? That is up for underlying framework to decide. If I declare my XML as <?xml version="1.0" encoding="ASCII" ?> does it mean it can't contain Unicode characters? Of course it can, they just going to be converted to XML entities. In fact I don't a reason why program that reads XML would ever need to pay attention the declaration encoding value. It is definitely important to the XML writer, but then why store it? Just for bookkeeping I guess?
From the perspective of defining tags and attributes & such, you're correct that it doesn't care about the character encoding. Often, XML libraries will work on the program-side exclusively with UTF-8 - when you give it data to create a document or get data out, it has to be UTF-8. Encoding comes in to play with XML serialization, which is also part of the spec. The XML spec coesn't exist completely separate from the idea of converting the data into a stream of bytes, it has rules for doing it too. For example, all XML parsers MUST support UTF-8.
What you say is true if referring to the abstract model of an XML document. But in practice an XML document has to be stored somewhere, in memory or in a file. So you need to choose a convention for string encoding. The XML declaration specifies which encoding is used when serializing an XML document into a text file.
@Jason I have update my question, since many people seem to be confused by it.
@ChrisJ Incorrect. I can easily write a code that uses ASCII XML encoding, but writes it to a file as UTF-8.
|
2

Yes, an XML file is a text file, i.e. a sequence of characters. A file is a sequence of bytes. So how are individual characters encoded, i.e. converted into a sequence of bytes? There are several ways to encode characters into bytes; the "encoding" declaration indicates which one is used.

As such, the "encoding" declaration plays a very significant role: one absolutely needs to know which encoding is used to be able to merely read the characters from a file. If no encoding is specified, XML has a set of default encodings, depending on the presence of a “byte order marker” (BOM). If there is no BOM, the default encoding is UTF-8.

ASCII is one of the simplest forms of encoding. It can only represent a span of 128 basic Latin characters. UTF-8 is more elaborate; it can represent all of the Unicode character set. So you're right, if you're using ASCII, you're obliged to use XML entities to represent the huge amount of characters that exist in Unicode but not in ASCII.

Improve this answer

8 Comments

I am still confused. What does XML declaration have to do with files? Or any way of storage including memory? XML is just text. How the string containing XML is encoded to write to a file seems completely unrelated to its content.
You say “XML is just text”, and that is absolutely true. But to store any portion of text in a computer you have to map each character to a code (in practice, a sequence of bytes). Such a mapping is called an encoding. So any text file is written according to an encoding. To read the file, you need to know which encoding was used for writing.
You are confusing two different levels of encoding. XML encoding does not convert characters to bytes, but rather characters to characters or XML entities.
No. The <?xml encoding... ?> declaration has nothing to do with entities directly. The only relation between them is indirect: if you choose an encoding in which some Unicode code points cannot be represented, then to use these ones you're obliged to use the corresponding entities.
Isn't that exactly what I said in the opening question?
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.