151

When doing shell scripting, typically data will be in files of single line records like csv. It's really simple to handle this data with grep and sed. But I have to deal with XML often, so I'd really like a way to script access to that XML data via the command line. What are the best tools?

1

14 Answers 14

108

I've found xmlstarlet to be pretty good at this sort of thing.

http://xmlstar.sourceforge.net/

Should be available in most distro repositories, too. An introductory tutorial is here:

http://www.ibm.com/developerworks/library/x-starlet.html

Sign up to request clarification or add additional context in comments.

3 Comments

Thought I'd point out that there are Windows binaries available on the Sourceforge site.
Doesn't support XQuery though, as far as I can tell.
@SteveBennett indeed it doesn't, but the features it adds on top of raw XPath are good enough to make it competitive with "grep and sed". If you want the fancy, fancy goodness of XQuery... well, that's more like an XML equivalent to perl or awk. :)
40

Some promising tools:

  • nokogiri: parsing HTML/XML DOMs in ruby using XPath & CSS selectors

  • hpricot: deprecated

  • fxgrep: Uses its own XPath-like syntax to query documents. Written in SML, so installation may be difficult.

  • LT XML: XML toolkit derived from SGML tools, including sggrep, sgsort, xmlnorm and others. Uses its own query syntax. The documentation is very formal. Written in C. LT XML 2 claims support of XPath, XInclude and other W3C standards.

  • xmlgrep2: simple and powerful searching with XPath. Written in Perl using XML::LibXML and libxml2.

  • XQSharp: Supports XQuery, the extension to XPath. Written for the .NET Framework.

  • xml-coreutils: Laird Breyer's toolkit equivalent to GNU coreutils. Discussed in an interesting essay on what the ideal toolkit should include.

  • xmldiff: Simple tool for comparing two xml files.

  • xmltk: doesn't seem to have package in debian, ubuntu, fedora, or macports, hasn't had a release since 2007, and uses non-portable build automation.

xml-coreutils seems the best documented and most UNIX-oriented.

Update 2025-03-09: I've been referring to https://github.com/dbohdan/structured-text-tools for CSV and other field-separated data (I like miller, sc-im & visidata). But notably it has many XML tools in https://github.com/dbohdan/structured-text-tools?tab=readme-ov-file#xml which have not been mentioned here.

2 Comments

Couldn't you create a wrapper script for the Ruby program, and pass in the arguments' array in the script to hpricot? E.g., in a PHP shell script, something like the following should work: <?php /path/to/hpricot $argv ?>
xmldiff is cool
26

There is also xml2 and 2xml pair. It will allow usual string editing tools to process XML.

Example. q.xml:

<?xml version="1.0"?>
<foo>
    text
    more text
    <textnode>ddd</textnode><textnode a="bv">dsss</textnode>
    <![CDATA[ asfdasdsa <foo> sdfsdfdsf <bar> ]]>
</foo>

xml2 < q.xml

/foo=
/foo=   text
/foo=   more text
/foo=   
/foo/textnode=ddd
/foo/textnode
/foo/textnode/@a=bv
/foo/textnode=dsss
/foo=
/foo=    asfdasdsa <foo> sdfsdfdsf <bar> 
/foo=

xml2 < q.xml | grep textnode | sed 's!/foo!/bar/baz!' | 2xml

<bar><baz><textnode>ddd</textnode><textnode a="bv">dsss</textnode></baz></bar>

P.S. There are also html2 / 2html.

4 Comments

@Joseph Holsten Yes. It allows hacking with XML without thinking through XPath things.
Nice! I had been focusing on tools that don't use an intermediate format, but the idea of a high-fidelity, line-oriented representation of xml seems like a great way to keep using real grep and sed. Have you tried pyxie? How does it compare? Any other line oriented representations? Would you consider this better than just replacing xml newlines with an entity (&#10;)? This would let you stick records on the same line at least. Oh, and could you edit your post to include a link to the project?
@Joseph Holsten No, I don't think pyxie format whould be more useful than xml2 format. xml2 provides "full path" in nested XML elements, so allow more line-oriented matching and substitution. Also 2xml can easily recreate XML from partial (filtered) xml2 output.
+1 I can't upvote this enough... cat foo.xml | xml2 | grep /bar | 2xml — gives you the same structure as the original, but all elements have been stripped except "bar" elements. Awesome.
25

To Joseph Holsten's excellent list, I add the xpath command-line script which comes with Perl library XML::XPath. A great way to extract information from XML files:

 xpath -q -e '/entry[@xml:lang="fr"]' *xml

1 Comment

This is installed by default in osx, but without -q -e options. Example, get attribute "package" value from the "manifest" node in "AndroidManifest.xml": xpath AndroidManifest.xml 'string(/manifest/@package)' 2> /dev/null
18

You can use xmllint:

xmllint --xpath //title books.xml

Should be bundled with most distros, and is also bundled with Cygwin.

$ xmllint --version
xmllint: using libxml version 20900

See:

$ xmllint
Usage : xmllint [options] XMLfiles ...
        Parse the XML files and output the result of the parsing
        --version : display the version of the XML library used
        --debug : dump a debug tree of the in-memory document
        ...
        --schematron schema : do validation against a schematron
        --sax1: use the old SAX1 interfaces for processing
        --sax: do not build a tree but work just at the SAX level
        --oldxml10: use XML-1.0 parsing rules before the 5th edition
        --xpath expr: evaluate the XPath expression, inply --noout

4 Comments

There is no --xpath argument to xmllint: manpagez.com/man/1/xmllint
@MiserableVariable: The man page is incorrect. I just looked at the man page for my version: the xpath argument is not listed. This is a documentation error. Try running the program, instead.
@MiserableVariable --xpath is a fairly recent addition and e.g. not in RHEL 6 versions of xmllint.
To be more precise, xmllint --xpath was introduced in libxml2 2.7.7 (in 2010).
9

If you're looking for a solution on Windows, Powershell has built-in functionality for reading and writing XML.

test.xml:

<root>
  <one>I like applesauce</one>
  <two>You sure bet I do!</two>
</root>

Powershell script:

# load XML file into local variable and cast as XML type.
$doc = [xml](Get-Content ./test.xml)

$doc.root.one                                   #echoes "I like applesauce"
$doc.root.one = "Who doesn't like applesauce?"  #replace inner text of <one> node

# create new node...
$newNode = $doc.CreateElement("three")
$newNode.set_InnerText("And don't you forget it!")

# ...and position it in the hierarchy
$doc.root.AppendChild($newNode)

# write results to disk
$doc.save("./testNew.xml")

testNew.xml:

<root>
  <one>Who likes applesauce?</one>
  <two>You sure bet I do!</two>
  <three>And don't you forget it!</three>
</root>

Source: https://serverfault.com/questions/26976/update-xml-from-the-command-line-windows

2 Comments

battled with various linux tools for a few hours before resorting to Powershell. I'm surprised this is so hard - linux cmd-line is normally really good but there seems to be a hole here. Note: Use case for me was: 1) locate nodes by xpath, 2) remove if found, 3) add new nodes, 4) save file. I was updating a bunch of solr configs. If anyone knows of an easy/reliable way to do this I'm all ears
Wow, this really tiptoes up to the line of an acceptable solution. But honestly, I'd probably accept it if it looked like xps $doc .root.one xps $doc 'AppendChild("three")' and xps $doc '.three.set_InnerText("And don't you forget it!")', which is clearly inferior!
8

There're also xmlsed & xmlgrep of the NetBSD xmltools!

http://blog.huoc.org/xmltools-not-dead.html

Comments

6

Depends on exactly what you want to do.

XSLT may be the way to go, but there is a learning curve. Try xsltproc and note that you can hand in parameters.

Comments

6

D. Bohdan maintains an open source GitHub repo that keeps a list of command line tools for structured text tools, there a section for XML/HTML tools:

https://github.com/dbohdan/structured-text-tools#xml-html

Comments

4

There's also saxon-lint from command line with the ability to use XPath 3.0/XQuery 3.0. (Other command-line tools use XPath 1.0).

EXAMPLES :

http/html:

$ saxon-lint --html --xpath 'count(//a)' http://stackoverflow.com/q/91791
328

xml :

$ saxon-lint --xpath '//a[@class="x"]' file.xml

Comments

3

XQuery might be a good solution. It is (relatively) easy to learn and is a W3C standard.

I would recommend XQSharp for a command line processor.

1 Comment

BaseX also has a command-line XQuery processor (in addition to its database mode), and stays up-to-date with bleeding-edge versions of the standard (following the evolving draft of XQuery 3.0 quite closely).
3

I first used xmlstarlet and still using it. When the query gets tough, i need XML's xpath2 and xquery feature support I turn to xidel http://www.videlibri.de/xidel.html

Comments

1

Grep Equivalent

You can define a bash function, say "xp" ("xpath") that wraps some python3 code. To use it you need to install python3 and python-lxml. Benefits:

  1. regex matching which you lack in e.g. xmllint.
  2. Use as a filter (in a pipe) on the commandline

It's easy and powerful to use like this:

xmldoc=$(cat <<EOF
<?xml version="1.0" encoding="utf-8"?>
<job xmlns="http://www.sample.com/">programming</job>
EOF
)
selection='//*[namespace-uri()="http://www.sample.com/" and local-name()="job" and re:test(.,"^pro.*ing$")]/text()'
echo "$xmldoc" | xp "$selection"
# prints programming

xp() looks something like this:

xp()
{ 
local selection="$1";
local xmldoc;
if ! [[ -t 0 ]]; then
    read -rd '' xmldoc;
else
    xmldoc="$2";
fi;
python3 <(printf '%b' "from lxml.html import tostring\nfrom lxml import etree\nfrom sys import stdin\nregexpNS = \"http://exslt.org/regular-expressions\"\ntree = etree.parse(stdin)\nfor e in tree.xpath('""$selection""', namespaces={'re':regexpNS}):\n  if isinstance(e, str):\n    print(e)\n  else:\n    print(tostring(e).decode('UTF-8'))") <<< "$xmldoc"
}

Sed Equivalent

Consider using xq which gives you the full power of the jq "programming language". If you have python-pip installed, you can install xq with pip install yq, then in below example we are replacing "Keep Accounts" with "Keep Accounts 2":

xmldoc=$(cat <<'EOF'
<resources>
    <string name="app_name">Keep Accounts</string>
    <string name="login">"login"</string>
    <string name="login_password">"password:"</string>
    <string name="login_account_hint">input to login</string>
    <string name="login_password_hint">input your password</string>
    <string name="login_fail">login failed</string>
</resources>
EOF
)
echo "$xmldoc" | xq '.resources.string = ([.resources.string[]|select(."#text" == "Keep Accounts") ."#text" = "Keep Accounts 2"])' -x

Comments

-1

JEdit has a plugin called "XQuery" which provides querying functionality for XML documents.

Not quite the command line, but it works!

1 Comment

While JEdit likely has a way to search through a file, that does not make it a competitor to grep(1).

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.