961 questions
2
votes
1
answer
191
views
Extract tables from website with dynamic content with R
I'm trying to extract tables from this site:
https://www.dnb.com/business-directory/company-information.beverage_manufacturing.br.html
As you can see, the complete table has 14,387 rows and each page ...
1
vote
1
answer
64
views
Rselenium won't switch to another tab
Using RSelenium, I'm trying to open a new tab and scroll down. Unfortunately, I can't make this work because my browser stays at the first tab. Does anyone know how to solve this?
This is a simplied ...
- 35
9
votes
1
answer
2k
views
RSelenium error in open.connection(con, "rb") with rsDriver()
I'm using RSelenium to scrape some data. It was working for me no problem yesterday but when I got back to it today I'm getting an error I don't understand. Here's what I'm doing:
library(rJava)
...
- 99
0
votes
0
answers
53
views
RSelenium / Undefined error in httr call / could not determine server status
I am trying to run a simple code in R Studio on my new laptop, and I am trying desperately to get my code to run, by failing since hours ... any help would be highly appreciated!
I run this code:
...
- 9
0
votes
0
answers
61
views
Using R Selenium inside a selenium/standalone-firefox Docker container. Connecting to the server
I am using a selenium/standalone-firefox docker container that I have installed R into. The aim is to set up a daily scrape of a webpage using GCS. I have an R selenium script that runs well locally. ...
- 1
4
votes
1
answer
100
views
How to extract text from a website that uses Javascript with rvest?
At this link (https://portraits.ouranos.ca/fr/spatial?a=0&c=0&discrete=1&e=CMIP6&i=tg_mean&p=50&r=mrc001&s=annual&scen=ssp370&w=0&yr=2071) there is a tag (...
- 3,843
0
votes
1
answer
65
views
Web scraping on tipti page that requires login
I'm trying to extract the name and prices of the AKI supermarket in Ecuador. There is a page called tipti that gathers products from several supermarkets.
However, it requires login and the page seems ...
- 2,339
0
votes
1
answer
147
views
No longer able to set up an rsDriver using Rselenium
It's been a few months since I used rSelenium but now what used to work doesn't. I updated both java and the rselenium package thinking that must be it.
Here's my code:
library(tidyverse)
library(...
- 689
0
votes
1
answer
133
views
Using RSelenium, problem loading driver, error 504
I am trying to open firefox with RSelenium and I am getting this error:
Warning: cannot open URL 'https://api.bitbucket.org/2.0/repositories/ariya/phantomjs/downloads?pagelen=100': HTTP status was '...
- 1
0
votes
1
answer
43
views
R Selenium wait for "next" button and click until no longer there
I'm dealing with a dynamically loaded page.
After the load, the "next" button appears, which I want to click until the full list of divs is loaded (= until the button is no longer there).
...
- 1,542
0
votes
1
answer
59
views
The Input password field is not displaying password but not throwing error
I'm trying to automate login with Rselenium in Rstudio please see below
remote_driver <- rsDriver(browser = "firefox",
geckover = "0.33.0",
...
- 424
0
votes
0
answers
102
views
scraping a dynamic webpage from R - RSelenium issue
I am trying to scrape dynamically filled webpages like this, on R.
I am trying to do that with RSelenium, but I am open to alternatives. For example, I would happy to do everything with rvest only.
...
- 141
0
votes
2
answers
88
views
Scraping PDF or pdf text when the file is embedded in html [closed]
Using R, I am trying to get the text (ideally, with some formatting) of a pdf embedded in html. THe url, as an example, is
"https://www.nycourts.gov/courts/ad2/Handdowns/2024/10-October/10-02-...
- 153
1
vote
0
answers
39
views
Return the year of the first mention of a string in Google using RSelenium
I have a list of strings (slightly over a thousand), and I would like to find out the year when each string was mentioned in the web for the first time and save the link of the webpage where this &...
- 57
4
votes
1
answer
263
views
Web scraping in R, page doesn't respond to request
I'm unable, most of the time, to make request to the following website:
https://www.adondevivir.com/proyectos-etapa-pre-venta-en-construccion.html
library(rvest);library(tibble);library(httr2)
...
- 117
1
vote
1
answer
72
views
Scraping text in whitebox
I am trying to collect some Dutch historical election data. Below you see the code I have been using. I still need to figure out how to iterate the process for every 'Gemeente', but my main problem ...
- 115
0
votes
1
answer
90
views
Looping over multiple pages with RSelenium
I've managed to get RSelenium to work and so far I've been successful in extracting some data from the following website: https://www.immobiliare.it/vendita-case/belluno-provincia/?criterio=rilevanza
...
0
votes
1
answer
69
views
How can I see Chrome Network activity i R? [httr, selenium?]
I'm wondering how I can in R visit a website and see the Network activity and responses as I would in Google Chrome and copy their associated attributes as url?
Dont even know where to start but I ...
- 2,579
0
votes
0
answers
59
views
Setup for Rselenium
I am using Azul jdk to set Rselenium if is good until when I go into R
remote_driver <- rsDriver(browser = "chrome", chromever = "114.0.5735.90", ...
0
votes
1
answer
49
views
Why does getElementValueOfCssProperty return a digit and not the correct value in Selenium?
I'm trying to get get a background color of a element and see if its bold using getElementValueOfCssProperty. For the color it returns "rgba(255, 222, 173, 1)", which I can convert to hex, ...
- 2,579
0
votes
1
answer
70
views
Can not get information from a website using RSelenium
I would like to use RSelenium to scrape a webpage to check if a certain text exists.
I need to click on Staatsangehörigkeitsangelegenheiten followed by clicking 02. Antrag Einbürgerung [sorry as the ...
- 1
1
vote
1
answer
1k
views
How to install and specify correct driver version for RSelenium for Chrome v115.0 and later?
Most of the instructions for setting up Chrome servers with RSelenium advise users set the version of Chrome to the most recent version or the version found in their Chrome drivers folder. For example:...
- 4,337
1
vote
1
answer
80
views
simulate scrolling in Rselenium or selenium in python
I am trying to scrape this website. You need to click on the magnifying galss icon in the search bar to see the records I want to extract. The issue is that the website is dynamic and I need to scroll ...
2
votes
2
answers
202
views
RSelenium: Unable to extract hrefs from page after button click
I'm trying to automate web scraping using RSelenium in R. I've successfully located and clicked a button on a webpage using RSelenium, but I'm having trouble extracting href attributes from the page ...
- 4,132
0
votes
2
answers
178
views
Scrape google knowledge graph with Rselenium
I am trying to access the elements on the rhs of the google search, sometimes called the knowledge graph. In particular I am interested in the short bio (normally a wikipedia snippet) and the external ...
- 1,750
1
vote
2
answers
127
views
Difficult Web scraping a dynamic content website with R
library(readr)
library(tidyverse)
library(rvest)
salary_link <- "https://www.spotrac.com/mlb/rankings/salary/pitching/"
salary_page <- read_html(salary_link)
salaries <- html_text(...
- 11
0
votes
1
answer
96
views
netTimeout when navigating more than once with RSelenium
I'm getting this netTimeout error when I try to navigate more than once with RSelenium. The first time I use navigate after loading the browser, it always works. The second time I use navigate, I ...
- 1,248
0
votes
1
answer
100
views
RSelenium - Not able to perform operations after hanging on a website
I am webscraping using RSelenium. After hanging on a website for some minutes, I get the following error message:
"_Error in .Call(R_curl_fetch_memory, enc2utf8(url), handle, nonblocking) :
...
- 1
2
votes
2
answers
126
views
Multiple tags getting captured while web scraping Reddit in R using RSelenium
enter image description hereI was writing code to web scrape the post title, comments and author names from a reddit post for a project.
I am able to web scrape the post title, author names but the ...
- 33
1
vote
0
answers
19
views
How to load multiple chrome extensions with RSelenium?
I found a way in which one can load a single Extension directly with RSelenium. The question here is how to load multiple extensions? Anyone?
ZoteroConnector="C:/Users/Documents/URLsToZotero/...
- 449
0
votes
1
answer
101
views
Why is RSelenium unable to find my xpath?
I have to perform a search repeatedly on a website (https://franklin.genoox.com/clinical-db/home)
A long time ago I found the Rselenium package and I have used it successfully in previous times (After ...
0
votes
1
answer
128
views
how to use executeScript in Rselenium as a way to simulate pressing hotkeys
RSelenium is a great package for web interface using R, but lacks functionality on the use of hotkeys as a way to run external functions or for programmatically activate extensions.
One possibility I ...
- 449
1
vote
1
answer
265
views
Download pdf from javascript onclick attribute using R
I would like to download a pdf from this website using R. The problem is that you first have to click on the "Maak een pdf" button on the website. Because this is an javascript onclick ...
- 42.8k
2
votes
0
answers
95
views
How can I access file downloaded using RSelenium in Github Actions?
I have a test repo in GitHub and I am using GitHub Actions to visit a URL and download a .zip file, using RSelenium and a Docker container running Selenium Firefox. As shown in the code below, I run ...
0
votes
1
answer
93
views
How to use shortcuts with RSelenium
How are shortcuts used in RSelenium?
In the example below, I use a common shortcut to display the browsing history in Firefox. That is: push and hold Control key + press H key.
While the shortcut ...
- 449
0
votes
2
answers
127
views
How to load extensions using Rselenium
Can anyone give me a hand to figure out how to activate a chrome extension with RSelenium?
Extensions are located in the tab of the browsers, but are not preloaded when using RSelenium.
- 449
0
votes
0
answers
51
views
RSelenium - For loop through multiple webpages, grab data and paste it into data.frame
I'm trying to loop through a job listing website to grab their job listing and do text analysis. For this job I use RSelenium.
The code I am working on is as follows:
#### REMOTE.COM ####
remDR$...
- 120
1
vote
0
answers
79
views
Undefined error in httr call. httr output: Failed to connect to localhost port #####: Connection refused (RSelenium error)
I am learning about RSelenium.
Does anyone have a solution for this error:
Undefined error in httr call. httr output: Failed to connect to localhost port #####: Connection refused.
I am surprised by ...
0
votes
1
answer
689
views
Scrape leaflet map coordinates from a dynamic website
I am trying to scrape the marker coordinates from a website containing a leaflet map (osm data). I have been trawling the web for answers and it appears that a simple query to the parsed html will not ...
- 607
1
vote
0
answers
60
views
Could not open Chrome browser rselenium in shinyapps.io
I've been working on a Shiny app using RSelenium. Locally, everything works, but when I try to host it on shinyapps.io, I encounter an error:
"Connecting to remote server"
Could not open ...
0
votes
1
answer
147
views
Undefined error in httr call. httr output: Failed to connect to localhost port 4445 after 0 ms: Connection refused
I was run shiny app through docker. But it works on RStudio but when I run it through docker it gives an error:
Undefined error in httr call. httr output: Failed to connect to localhost port 4445 ...
- 29
5
votes
3
answers
94k
views
How to find a chrome driver for my chrome version?
I am trying to install RSelenium for webscraping but am not able to get a chromedriver that matches with my current chrome version. I do not want to downgrade to an older google chrome as that seems ...
- 53
0
votes
1
answer
151
views
Rselenuim/Docker - unable to connect to remote server for firefox
My Rselenium suddenly stopped. I looked for answers, tried all of them without any luck.I then uninstalled docker and reinstalled, restarted my machine and all of that. Finally have managed to get a ...
- 1,906
0
votes
1
answer
115
views
Need to use rvest to scrap dynamic content
I have to scrap data from a auctions website called Unicorn Auctions.
When I try to do it using rvest everything that I can get is the auction title and URL, but I also need it's start and end dates. ...
1
vote
0
answers
45
views
RSelenium still running?
Hi used RSelenium for some web-scraping a couple of weeks back. I finished the task but every time I open my RStudio I get this error in my console
Warning: namespace ‘RSelenium’ is not available and ...
- 11
0
votes
1
answer
33
views
Getting data from nested nodes in RSelenium
I am doing literature review, so I am trying to get the list of papers under certain search terms in Google scholar. I have trouble getting the PDF links when available. Some references have PDFs, ...
- 449
1
vote
1
answer
69
views
How can click on the keyboard right button using R selenium?
I want to click on the right button of keyboard using Rselenum. But unfortunately, I get this error.
x <- driver$findElement("xpath", "/html/body/c-wiz[2]/div/c-wiz/div[1]/div[1]/div[...
- 29
2
votes
0
answers
688
views
Undefined error in httr call. httr output: length(url) == 1 is not TRUE in Rselenium
I get this error when my google chrome update from 118 to 119. I have been changed the chromedriver also.
Error in checkError(res) :
Undefined error in httr call. httr output: length(url) == 1 is not ...
- 29
2
votes
1
answer
462
views
R: Is it possible to download the latest version of Google Chrome directly in R?
I am working with the R programming language.
I am using the Selenium library.
Earlier (in August 2023), my code used to run without any problem - but now (November 2023) I am getting the following ...
- 6,845
2
votes
0
answers
163
views
How can I use package RSelenium in a Shiny app hosted on shinyapps.io?
Hello smart people of the internet,
I have written an app in R Shiny that uses the package RSelenium (and specifically, its rsDriver function). It works fine on my local machine. However, when I try ...
- 377