I have the following HTML code:
<textarea name="command" class="setting-input fixed-width" rows="9">1</textarea><textarea name="command" class="setting-input fixed-width" rows="5">2</textarea>
I would like to parse it to receive such output:
1
2
Currently I am using:
xmllint --xpath '//textarea[@name="command"]/text()' --html
but it does not append a newline after each match.
On
I did the following, ugly trick, please feel free to provide a better solution.
Changed the HTML code by replacing </textarea> with \n</textarea> using the following command:
sed 's/\<\/textarea/\'$'\n\<\\/textarea/g' f
Try this patch, which provides 2 options:
--xpath: same as old --xpath, with nodes separated by \n.
--xpath0: same as old --xpath, with nodes separated by \0.
Test input (a.html):
<textarea name="command" class="setting-input fixed-width" rows="9">1</textarea><textarea name="command" class="setting-input fixed-width" rows="5">2</textarea>
Test command 1:
# xmllint --xpath '//textarea[@name="command"]/text()' --html a.html
Test output 1:
1
2
Test command 2:
# xmllint --xpath0 '//textarea[@name="command"]/text()' --html a.html | xargs -0 -n1
Test output 2:
1
2
Newlines can legitimately appear in xml data. A more robust approach would be to delimit xpath results by a character that is guaranteed to not occur in XML data. The Null character, U+0000 in the Universal Coded Character Set, is one such character.
Note that the code point U+0000, assigned to the null control character, is the only character encoded in Unicode and ISO/IEC 10646 that is always invalid in any XML 1.0 and 1.1 document.
– https://en.wikipedia.org/wiki/Valid_characters_in_XML
@Cyker's merge request for xmllint included the addition of an -xpath0 option that would delimit xpath results by NUL. A new feature request for this functionality was opened as well.
Hopefully, xmllint will gain this feature soon.
In the mean time, another xpath command line tool, xmlstarlet, can be coaxed into achieving this goal now. xmlstarlet does not currently support output of NULs directly, but we can make it output U+FFFF, which, like NUL, is guaranteed to not occur in XML data (source). We then just need to translate U+FFFF to U+0000 and we'll have NUL delimited xpath results.
In the following examples, I'll use the following partial html file. It's the same example from the OP's question, except I added newlines for testing purposes.
cat >data.html <<'EOF'
<textarea name="command" class="setting-input fixed-width" rows="9">1
newline</textarea>
<textarea name="command" class="setting-input fixed-width" rows="5">2
newline</textarea>
EOF
Here is how to use xmlstarlet and sed to delimit the xpath results with NULs:
xmlstarlet fo -H -R data.html \
| xmlstarlet sel -t -m '//textarea[@name="command"]' -v '.' -o $'\uffff' \
| sed s/$'\uFFFF'/\\x00/g
perl could be used instead of sed, if you prefer: perl -CS -0xFFFF -l0 -pe ''
Note: I ran the HTML through xmlstarlet fo -H -R as shown in @TheDudeAbides answer.
Now that the xpath results are delimited by NULs, we can process the results with the help of xargs -0. Example:
xmlstarlet fo -H -R data.html \
| xmlstarlet sel -t -m '//textarea[@name="command"]' -v '.' -o $'\uffff' \
| sed s/$'\uFFFF'/\\x00/g \
| xargs -0 -n 1 printf '%q\n'
Result:
'1 '$'\n'' newline'
'2 '$'\n'' newline'
or load it into a bash array:
mapfile -t -d '' a < <(
xmlstarlet fo -H -R data.html \
| xmlstarlet sel -t -m '//textarea[@name="command"]' -v '.' -o $'\uffff' \
| sed s/$'\uFFFF'/\\x00/g
)
declare -p a
Result:
declare -a a=([0]=$'1 \n newline' [1]=$'2 \n newline')
Same technique using saxon instead of xmlstarlet:
xmllint --html data.html --dropdtd --xmlout \
| java -cp "$CP" net.sf.saxon.Query -s:- -qs:'//textarea[@name="command"]' !method=text !item-separator=$'\uFFFF' \
| sed s/$'\uFFFF'/\\x00/g \
| xargs -0 -n 1 printf '%q\n'
Below is a wrapper script intended exactly to the purpose of newlines delimited output (for old releases of xmllint).
Create a file xmllint2.sh with the contents. Then execute chmod u+x xmllint2.sh, finally run like:
./xmllint2.sh --xpath --html '//textarea[@name="command"]/text()' 2>/dev/null
(the last part of the command is to hide the output of warnings that occurs with html)
#!/bin/bash
# wrapper script to
# - have newline delimited output on Xpath querys
# - implements --xpath on very old releases
/usr/bin/xmllint --xpath &>/dev/null
implements_xpath=$?
newlines_delimited_xmllint_version=20909
current_version=$(xmllint --version |& awk 'NR==1{print $NF;exit}')
args=( "$@" )
if [[ $@ == *--xpath* ]]; then
# iterate over positional parameters
for ((i=0; i<${#args}; i++)); do
if [[ ${args[i]} == --xpath ]]; then
xpath="${args[i+1]}"
unset args[i+1]
unset args[i]
break
fi
done
if [[ ($implements_xpath==0 && $current_version>=20909) || $file == - || $file == /dev/stdin || $xpath == / || $xpath == string\(* ]]
then
exec /usr/bin/xmllint "$@"
else
exec /usr/bin/xmllint "${args[@]}" --shell <<< "cat $xpath" | sed '1d;$d;s/^ ------- *$//;/^$/d'
fi
else
exec /usr/bin/xmllint "$@"
fi
Check latest revision: https://github.com/sputnick-dev/xmllint
Debian Buster in June 29 2020 have version 2.9.4 which is 4 years old.
Debian testing/experimental have 2.9.10, which is the fixed version.
Another way to install 2.9.10 with Debian last stable: https://serverfault.com/a/1022826/120473 (without taking the risk of crashing the apt system)
Hello from the year 2020!
As of v2.9.9 of libxml, this behavior has been fixed in
xmllintitself.However, if you're using anything older than that, and don't want to build libxml from source just to get the fixed
xmllint, you'll need one of the other workarounds here. As of this writing, the latest CentOS 8, for example, is still using a version of libxml (2.9.7) that behaves the way the OP describes.As I gather from this SO answer, it's theoretically possible to feed a command into the
--shelloption of older (<2.9.9) versions ofxmllint, and this will produce each node on a separate line. However, you end up having to post-process it withsedorgrepto remove the visual detritus of shell mode's (human-oriented) output. It's not ideal.XMLStarlet, if available, offers another solution, but you do need to use
xmlstarlet foto format your HTML fragment into valid XML before usingxmlstarlet selto extract nodes:If the
Attempt to load network entitymessage from the secondxmlstarletinvocation annoys you, just add2>/dev/nullat the very end to suppress it (at the risk of suppressing other messages printed to standard error).The XMLStarlet options explained (see also the user's guide):
fo -H -R— format the output, expecting HTML input, and recovering as much bad input as possible<html>root node, making the fragment in the OP's example valid XMLsel -T -t -v //xpath -n— select nodes based on XPath//xpath-T) instead of XML-t) that returns the value (-v) of the node rather than the node itself (allowing you to forgo usingtext()in the XPath expression)-n)Edit(s): Removed half-implemented
xmllint --shellsolution because it was just bad. Added an XMLStarlet example that actually works with the OP's data.