Shell Script Parse XML: 5 Easy Ways, Avoid Mistakes

A shell script parse xml task usually starts small: you have a file such as person.xml and you want the value inside one tag. The classic answer is a one-liner with grep, awk or sed, and it works as long as the file looks exactly like the example. This shell script parse xml guide keeps the three commands from the original post and tests them in a clean Debian 13 container (GNU grep 3.11, mawk 1.3.4, GNU sed 4.9, libxml2 20914, xmlstarlet), then shows where each one breaks and which tool to use instead.

The sample file for the shell script parse xml test

The sample is the person.xml file from the old post. Every value sits on its own line, which is the one condition all three line-based commands depend on:

<person>
  <sex>Male</sex>
  <firstname>Huu</firstname>
  <lastname>Phan</lastname>
  <sex>female</sex>
  <firstname>Miranda</firstname>
  <lastname>Kerr</lastname>
</person>

The goal is to print the two values of the firstname tag, Huu and Miranda. Note that the file repeats the tags without a wrapper element for each person; this is a flat example, not a realistic document, but it is enough to compare the tools.

Three classic ways to parse XML in a shell script

The three commands from the original article, run unchanged on the pretty-printed file, all printed the same two lines, Huu and Miranda:

grep -oP '(?<=<firstname>).*(?=</firstname)' person.xml
awk -F '[><]' '/firstname/{print $3}' person.xml
sed -n '/firstname/{s/.*<firstname>//;s/<\/firstname.*//;p;}' person.xml

The grep version uses a Perl regular expression with a look-behind and a look-ahead, so only the text between the tags is printed. It needs -P, which GNU grep has and the grep of BSD or macOS does not. The awk version splits each line on the characters > and <, so the third field of a line such as <firstname>Huu</firstname> is the value. The sed version deletes everything before the opening tag and everything after the closing tag, and prints what is left.

When a shell script parse xml one-liner fails

Real XML is often minified into one line by an API or a build tool. The same three commands then give different and wrong answers, with no error message. The figure shows the test on a one-line copy of the same data.

Shell script parse xml on a one-line file: grep, awk and sed return wrong output, xmllint is correct
The same shell script parse xml commands on a minified file: only the non-greedy grep and xmllint return both names.
  • grep with .* is greedy. On one line it printed Huu</firstname><lastname>Phan</lastname><firstname>Miranda, a single match that swallows everything between the first opening and the last closing tag. Changing the pattern to the non-greedy .*? fixed it and returned Huu and Miranda.
  • awk printed one empty line, because on a minified line the characters >< touch each other and the third field is empty. It cannot return the second name at all, since it prints one field per line.
  • sed printed only Miranda: the greedy s/.*<firstname>// removes everything up to the last tag on the line, so the first name is lost.

A second test file had attributes, a value split over three lines, a CDATA block and an entity. The grep pattern also returned the text fake from inside the CDATA block, which is not a real element, and returned Tom &amp; Jerry without decoding the entity. The awk command printed three blank lines for the value that was split across lines and for the CDATA line, and it never printed the name Zoe. The conclusion for any shell script parse xml job is simple: a shell script parse xml job with grep, awk or sed is only safe for a file whose layout you control.

Use xmllint or xmlstarlet for a safe shell script parse xml

An XML parser reads the structure, not the lines. Two small tools do that from a shell script: xmllint from the libxml2-utils package and xmlstarlet. Both take an XPath expression; the language is defined in the W3C XPath specification, and the options are listed in the xmllint manual and the xmlstarlet user guide.

xmllint --xpath '//firstname/text()' person.xml
xmllint --xpath 'string(//person[@id="2"]/firstname)' attr.xml
xmllint --xpath 'normalize-space(//person[@id="3"]/firstname)' attr.xml
xmlstarlet sel -t -v '//firstname' -n person.xml
xmlstarlet sel -t -m '//person' -v '@id' -o ': ' -v 'normalize-space(firstname)' -n attr.xml

In the test, xmllint --xpath '//firstname/text()' returned Huu and Miranda for both the pretty-printed and the one-line file. Selecting by attribute worked (@id="2" returned Miranda), normalize-space() removed the line breaks around Zoe, and string(//name) decoded the entity to Tom & Jerry. The xmlstarlet loop printed 1: Huu, 2: Miranda, 3: Zoe. Neither tool was fooled by the CDATA block: count(//firstname) returned 3, the real elements.

One detail matters when you store the result in a Bash array. With several matches, xmllint --xpath '//firstname/text()' prints the values joined together, one after another. Use xmlstarlet sel ... -n, which ends each value with a newline, and read it with mapfile -t names < <(...). The test printed [Huu] and [Miranda], as two array items.

Check the XML first and handle errors in a shell script parse xml

Line-based tools never tell you that the file is broken. On the test file <person><firstname>Huu</person> the grep command printed nothing and the exit code was 1, the same as for a missing tag. The parser stopped with parser error : Opening and ending tag mismatch: firstname line 1 and person and exit code 1, so it states the cause. To validate a file before you read it, run xmllint --noout file.xml; on the good sample it printed nothing and exited with 0.

For bigger scripts, add set -euo pipefail and test the exit code of each parse call. The posts on Bash script examples and writing portable shell scripts explain why. If the script must ask the operator a question first, see the Bash yes/no prompt.

Which tool should you pick for shell script parse xml?

  • A one-off command on a file you wrote yourself: grep, awk or sed is fine.
  • Output from an API, a config export or any file you do not control: xmllint --xpath.
  • Several fields per record, such as id and name together: xmlstarlet sel.
  • A very large file that cannot be loaded in memory: a streaming parser in Python or another language, which is outside this article.

FAQ

Is grep -P available everywhere? It is in GNU grep, but not in the grep that macOS and BusyBox ship, so a shell script parse xml command with -P is not portable.

Do I need to install anything for shell script parse xml? grep, awk and sed come with the system. On Debian the other two come from apt install libxml2-utils xmlstarlet.

Comments

Popular posts from this blog

How to Play Minecraft Bedrock Edition on Linux: A Comprehensive Guide for Tech Professionals

Linux desktop frozen? My 5 go-to tricks to try - before forcing a hard reboot

The Ultimate Guide: How to Set Up DXVK in Wine on Linux for Enhanced Gaming Performance