Get element text using RSelenium

Question

I have been struggling with the following lines of code for a few hours now and I still seem to be no closer to a solution. My code is as follows:

#create a list of all the question elements on the page
questions <- remDr$findElements(using = 'xpath', "//div[@class='question-text']")

#get the first question element in the list
question <- questions[1]

#get the text of the question element
question$getElementText()

When I debug using RStudio, it looks as though the 'questions' list is populated correctly with all 'question' elements; the 'question' item is populated correctly with the first 'question' element in the list; but many variations on the next line of code, intended to get the text within the question element, all seem to fail, giving the following error:

Error in evalq({ : attempt to apply non-function

It is possible that the error comes from a different part of the code, but very unlikely, as commenting out that line seems to leave everything else working perfectly.

I would be very grateful for any help you guys and gals might be able to provide. I'm programming in R using RSelenium - as you can probably tell, I am new to R although I do have very limited experience using Selenium in other environments.

Thanks in advance for your thoughts!

Welcome to SO. Please hover over the R tag - it asks you to provide a minimal reproducible example. Your example is not reproducible: executing the code yields "Error: object 'remDr' not found". — lukeA
– lukeA, Commented May 7, 2017 at 16:18
Apologies Luke, I was trying to provide the shortest possible example to illustrate the problem but obviously I neglected to give a reproducible one. Will try to do better next time. — thunderbuck88
– thunderbuck88, Commented May 7, 2017 at 17:41
Unless you state that you're scraping an internal job resource that can't be shared, posting the URL you're going after wld help folks know if they're assisting someone engaged in theft of intellectual property or not. — hrbrmstr
– hrbrmstr, Commented May 8, 2017 at 1:03
Thanks hrbrmstr, I will bear that in mind, too. It is difficult to get away from the spectre of intellectual property theft when dealing with scraping code, but this is a project designed to perform some natural language processing over public data (OGL, I think) that is readily available online - and I think there is even an API for those who know how to interact with such things. Hope this helps people feel more comfortable with providing help. — thunderbuck88
– thunderbuck88, Commented May 10, 2017 at 18:09

lukeA · Accepted Answer · 2017-05-07 16:16:26Z

9

question does not have a function called getElementText; it's a list object instead of a webElement object. You need [[ instead of [ - check out this example:

library(RSelenium)
rD <- rsDriver(port=4444L, browser = "phantomjs")
remDr <- rD[["client"]]
remDr$navigate(
  "http://stackoverflow.com/questions/43833649/get-element-text-using-rselenium")
elems <- remDr$findElements(using = 'xpath', "//a")
elem <- elems[1]
class(elem)
# [1] "list"
elem$getElementText()
# Error: attempt to apply non-function

Now

elem <- elems[[1]]
class(elem)
# [1] "webElement"
elem$getElementText()
# [[1]]
# [1] "Stack Overflow"
elem$getElementText()[[1]]
# [1] "Stack Overflow"
remDr$close()
rD[["server"]]$stop()

answered May 7, 2017 at 16:16

lukeA

54.4k5 gold badges102 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Get element text using RSelenium

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related