Discrepency in Python's lxml's equivalent parsing methods on HTML: cssselect vs xpath

Question

I was attempting to parse example.com's home page with xpath and cssselect but it seems as though either I don't know how xpath works, or lxml's xpath is broken, as it is missing matches.

Here's the quick and dirty code.

from lxml.html import *
mySearchTree = parse('http://www.example.com').getroot()
for a in mySearchTree.cssselect('tr a'):
    print 'found "%s" link to href "%s"' % (a.text, a.get('href'))

print '-'*8 +'Now for Xpath' + 8*'-'
# Find all 'a' elements inside 'tr' table rows with xpath
for a in mySearchTree.xpath('.//tr/*/a'):
    print 'found "%s" link to href "%s"' % (a.text, a.get('href'))

Results:

found "About" link to href "/about/"
found "Presentations" link to href "/about/presentations/"
found "Performance" link to href "/about/performance/"
found "Reports" link to href "/reports/"
found "Domains" link to href "/domains/"
found "Root Zone" link to href "/domains/root/"
found ".INT" link to href "/domains/int/"
found ".ARPA" link to href "/domains/arpa/"
found "IDN Repository" link to href "/domains/idn-tables/"
found "Protocols" link to href "/protocols/"
found "Number Resources" link to href "/numbers/"
found "Abuse Information" link to href "/abuse/"
found "Internet Corporation for Assigned Names and Numbers" link to href "http://www.icann.org/"
--------Now for Xpath--------
found "Presentations" link to href "/about/presentations/"
found "Performance" link to href "/about/performance/"
found "Reports" link to href "/reports/"
found "Root Zone" link to href "/domains/root/"
found ".INT" link to href "/domains/int/"
found ".ARPA" link to href "/domains/arpa/"
found "IDN Repository" link to href "/domains/idn-tables/"
found "Abuse Information" link to href "/abuse/"
found "Internet Corporation for Assigned Names and Numbers" link to href "http://www.icann.org/"

Basically the xpath found every link it was supposed to, except for those that were bolded by Example.com. However, shouldn't the asterisk wildcard have allowed for this in the xpath match './/tr/*/a'?

Community · Accepted Answer · 2020-06-20 09:12:55Z

3

Possibly something else is going on (I didn't examine the sample document closely), but your CSS selector and XPath are not equivalent.

CSS tr a is //tr//a in XPath. .//tr/*/a means (conceptually, not precisely):

.: current node
//: all descendants of current node
tr: all tr elements among all descendants of current node
/: all children of found tr elements
*: any element among children of found tr elements
/: all children of any child element of found tr elements
a: all a elements which which are element children of element children of a tr element

In other words, given the following HTML:

<ul>
    <li><a href="link1"></a><li>
    <li><b><a href="link2"></a></b><li>
</ul>

//ul/*/a will only match link1.

XPath Primer

In reality, an "XPath" is a series of Location Steps separated by slashes. A Location Step consists of:

An axis (e.g. child::)
A node test (either the node's name, or one of the special node types, e.g. node(), text())
Optional predicates (surrounded by []. A node is only matched if all the predicates are true.)

If we were to decompose .//tr/*/a into its Location Steps it would look like this:

.
(the "space" between the slashes in "//")
tr
*
a

It's probably not evident what the heck I am talking about. This is because XPath has an abbreviated syntax. Here is the expression with abbreviations expanded (axis and node-test are separated by a ::, steps by a /):

self::node()/descendent-or-self::node()/child::tr/child::*/child::a

(Notice that self::node() is redundant.)

Conceptually what happens in a step is:

given a set of context nodes (default is the current node or '/' for the root node)
For each context node, create a set of nodes which satisfy the Location Step
Union all per-context-node sets into one node set
Pass that set to the next Location Step as its given context nodes.
Repeat until out of steps. The set left after the final step is the set for the entire path.

Note that this is still a simplification. Read the XPath Standard for the gory details if you want them.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Dec 23, 2011 at 22:29

Francis Avila

31.8k7 gold badges63 silver badges99 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Nomad Coder Over a year ago

Thanks for the answer and in your description answering 3 other questions I hadn't even had yet!

Dimitre Novatchev Over a year ago

Francis, your interpretation of the meaning of the XPath operators is quite different from their real meaning. According to your interpretation expressions like .// or .//tr/ should be valid, while in fact, these are syntactically illegal. Please, either remove, or correct the explanation.

Francis Avila Over a year ago

By correcting the explanation (i.e. introducing the concept of Location Steps) it may become less clear to the OP. I will add a second more precise explanation.

jfs · Accepted Answer · 2011-12-23 22:26:57Z

1

'tr a' -> '//tr//a'

answered Dec 23, 2011 at 22:26

jfs

417k210 gold badges1k silver badges1.7k bronze badges

Collectives™ on Stack Overflow

Discrepency in Python's lxml's equivalent parsing methods on HTML: cssselect vs xpath

2 Answers 2

XPath Primer

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

XPath Primer

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related