2

I am scraping an interactive map on this website (https://collegecrisis.shinyapps.io/dashboard/) using Python and Selenium. Specifically, I want to get the text found in the pop-ups from the map.

After first trying to do this using Scrapy, I changed strategies because the information I am seeking is in the pop-ups and the clickable parts of the map overlap and aren't all accessible when looping through all relevant elements.

I get the exception: ElementClickInterceptedException element click intercepted: Element is not clickable at point (391, 500). Other element would receive the click: (Session info: chrome=Xxxx)

Hence I found some recommendations to try interacting with the map using javascript via this method:

driver.execute_script('arguments[0].click();', element)

However, I am getting the error (in other similar problems, respondents have highlighted that the error suggests that the parentheses and/or colon are missing, They are not in the code, the error simply shows the code without them below) : JavascriptException: javascript error: arguments[0].click is not a function (Session info: chrome=85.0.4183.83)

I have tried using the driver.execute_script('arguments[0].click();', element) approach to see if this works at all in the map. It does when clicking the 'zoom' button:

zoom_btn = driver.find_element_by_class_name("leaflet-control-zoom-in")
driver.execute_script("arguments[0].click();", zoom_btn)

I can click on individual dots on the map using .click() without javascript:

specific_node = driver.find_element_by_xpath('.//*[contains(@d, "M421.5685916444444,133.94770476315125a1,1 0 1,0 2,0 a1,1 0 1,0 -2,0 ")]')
specific_node.click()

But I can't loop through, click on every element (and then grab the text in the pop up) as mentioned at the top of the question due to overlaps in the dots on the map.

The code I have used so far is below. The last for loop is unfinished as I can't get the click part to work.

# import packages
from selenium import webdriver
from selenium.common.exceptions import ElementClickInterceptedException

# setup drivers
PATH = "/Applications/chromedriver"
driver = webdriver.Chrome(PATH)
driver.implicitly_wait(10) # seconds
driver.get("https://collegecrisis.shinyapps.io/dashboard/")

# find all class elements =leaflet-interactive
nodes = driver.find_elements_by_class_name("leaflet-interactive")

# loop through clickable elements and get text from pop-up
for node in nodes:
        node.click()
        # get text from popups (code to be finished). node.find_elements_by_xpath('.//div[@class = "leaflet-popup-content"]')

A working solution using ActionChains:

# import packages
from selenium import webdriver
from selenium.common.exceptions import ElementClickInterceptedException
from selenium.webdriver.common.action_chains import ActionChains
from time import sleep

# setup drivers
PATH = "/Applications/chromedriver"
driver = webdriver.Chrome(PATH)
driver.implicitly_wait(10) # seconds
driver.get("https://collegecrisis.shinyapps.io/dashboard/")

# find all class elements =leaflet-interactive
nodes = driver.find_elements_by_class_name("leaflet-interactive")

# use actionchains
nodelist = []

# loop through each node
for node in nodes:
    ActionChains(driver).move_to_element(node).click().perform() # Used actionchains class to click to open popup
    sleep(.5)
    nodelist.append(driver.find_element_by_class_name('leaflet-popup-content').text)
    ActionChains(driver).move_to_element(node).click().perform() #click to close popup

If anyone can provide further clarity around why the ActionChain method worked without encountering the overlapping dots that would be great.

Similarly, why doesn't the driver.execute_script('arguments[0].click();', element) method doesn't work?

1
  • Try to move find_element into for loop. Commented Sep 11, 2020 at 8:27

1 Answer 1

3

As all the dots are overlapping with each other its a better idea to use ActionChains class methods to mimic moving mouse cursor over them and then click:

ActionChains(driver).move_to_element(node).click().perform() # Used actionchains class to click to open popup
time.sleep(.5)
contents.append(driver.find_element_by_class_name('leaflet-popup-content').text)
ActionChains(driver).move_to_element(node).click().perform() #click to close popup
Sign up to request clarification or add additional context in comments.

1 Comment

That worked a treat. FYI for others - had to import ActionChains, time.sleep and used the code above in a for loop. Will add the complete working code to my post

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.