Parsing HTML page using bash

Using xpath expressions in xmllint with its built-in HTML parser would produce an output as

ipAddr=$(xmllint --html --xpath "string(//tr[1]/td[1])" html)
172.29.219.17

and for the time out value prediction, I did a manual calculation of the number of the td row containing the value, which turned out to be 24

xmllint --html --xpath "string(//tr[1]/td[24]/u[1])" html 

produces an output as

         L7TOUT in 1001ms
         Layer7 timeout:  at step 6 of tcp-check (expect string 'role:master')

removing the whitespaces and extracting out only the needed parts with Awk as

xmllint --html --xpath "string(//tr[1]/td[24]/u[1])" html | awk 'NF && /L7TOUT/{gsub(/^[[:space:]]*/,"",$0); print}'
L7TOUT in 1001ms

put in a variable as

timeOut=$(xmllint --html --xpath "string(//tr[1]/td[24]/u[1])" html | awk 'NF && /L7TOUT/{gsub(/^[[:space:]]*/,"",$0); print}'

Now you can print both the values together as

echo "${ipAddr} ${timeOut}"
172.29.219.17 L7TOUT in 1001ms

version details,

xmllint --version
xmllint: using libxml version 20902

Also there is an incorrect tag in your HTML input file </table> at the end just before <p> which xmllint reports as

htmlfile:147: HTML parser error : Unexpected end tag : table

remove the line before further testing.

Here is a list of command line tools that will help you parse different formats via bash; bash is extremely powerful and useful.

• JSON utilize jq
• XML/HTML utilize xq
• YAML utilize yq
• CSS utilize bashcss
  • I have tested all the other tools, comment on this one

If the code starts getting truly complex you might consider the naive answer below as coding languages with class support will assit.

naive - Old Answer

Parsing complex formats like JSON, XML, HTML, CSS, YAML, ...ETC is extremely difficult in bash and likely error prone. Because of this I recommend one of the following:

• PHP
• RUBY
• PYTHON
• GOLANG

because these languages are cross platform and have parsers for all the above listed formats.

If you want to parse HTML with regexes, then you have to make assumptions about the HTML formatting. E.g. you assume here that the a tag and its name attribute is on the same line. However, this is perfect HTML too:

<a
    name="redis/172.29.219.17">
    Some text
</a>

Anyway, let's sole the problem assuming that the a tags are on one line and the name is the first attribute. This is what I could come up with:

  sed 's/\(<a name="redis\)/\n\1/g' index.html | grep '^<a name="redis\/[0-9.]\+"' | sed -e 's/^<a name="redis\///g' -e 's/".*//g'

Explanation:

• The first sed command makes sure that all <a name="redis text goes to a separate line.
• Then the grep keeps only those lines that start with `
• The last sed contains two expressions:
  • The first expressions removes the leading <a name="redis/ text
  • The last expression removes everything that comes after the closing "