How to scrape all the search results using Selenium webdriver and Python

Question

I'm trying to scrape all CRD# from the search result from this site https://brokercheck.finra.org/search/genericsearch/list

(You'll need to redo the search when you click on the link, just type some random stuff for the Individual search)

I'm using driver.find_elements_by_xpath to target all CRD numbers on each result page. However, I've been playing around with the paths for a while but the webdriver still can't pick up the CRDs from the site.

I currently have (in Python)

crds = driver.find_elements_by_xpath("//md-list-item/div/div/div/div/div/bc-bio-geo-section/div/div/div/div/div/span")

But the result is always empty.

They have similar span classes of spacerright ng-binding. driver.find_elements_by_xpath("//span[@class = 'spacerright ng-binding']")); Then loop and print .getText() maybe? — Arundeep Chohan
– Arundeep Chohan, Commented May 19, 2020 at 3:55

undetected Selenium · Accepted Answer · 2020-05-19 18:22:15Z

To print all the CRD# from the search results within the website https://brokercheck.finra.org/search/genericsearch/grid using Selenium you have to induce WebDriverWait for the visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR and get_attribute():

print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "span.ng-binding[ng-bind-html='vm.item.id']")))])

Using XPATH and text:

print([my_elem.text for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[starts-with(., 'CRD')]//following-sibling::span[1]")))])

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

frianH · Accepted Answer · 2020-05-19 05:10:04Z

1

Try to use .find_elements_by_css_selector like this:

crds = driver.find_elements_by_css_selector("span[ng-bind-html='vm.item.id']")

answered May 19, 2020 at 5:10

frianH

7,5916 gold badges26 silver badges49 bronze badges

Collectives™ on Stack Overflow

How to scrape all the search results using Selenium webdriver and Python

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related