Scraping javascript data from table or grid

Question

I need to scrape data from a grid. I need all the data within the grid - (data containing Ticker , Score, Company, Exchange, and so on.

I am using Selenium to open the page and can't figure out what I am doing wrong. I've tried some tutorials but the code seems different. I've tried to scrape using the table "maintable-multi-1067-body" and I've also tried the grid "gridview-1070" but nothing is working. If anyone could show me how to do this, I'd appreciate it.

Code

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
import pandas as pd
    
options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
ser = Service("./chromedriver.exe")
browser = driver = webdriver.Chrome(service=ser)

driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
  "source": """
    Object.defineProperty(navigator, 'webdriver', {
      get: () => undefined
    })
  """
})
driver.execute_cdp_cmd("Network.enable", {})
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
wait = WebDriverWait(driver, 30)
driver.get("https://stockrover.com")
wait.until(EC.visibility_of_element_located((By.XPATH, "/html/body/div[1]/div/section[2]/div/ul/li[2]"))).click()
user = driver.find_element(By.NAME, "username")
password = driver.find_element(By.NAME, "password")
user.clear()
user.send_keys("test11964")
password.clear()
password.send_keys("test21964")
driver.find_element(By.NAME, "Sign In").click()
wait = WebDriverWait(driver, 30)


wait = WebDriverWait(driver,30)

table = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'tbody#maintable-multi-1067-body tr')))

for tab in table:
  print(tab.text)

Prophet · Accepted Answer · 2022-02-19 17:37:42Z

1

You are very close to the working code but there are still several issues here:

Your table locator is wrong.
Instead of

'tbody#maintable-multi-1067-body tr'

Try using Instead of

'#maintable-multi-1067-body tr'

It will give you 502 results.
2) No need to define 3 times the wait object

wait = WebDriverWait(driver,30)

answered Feb 19, 2022 at 17:37

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

J R Over a year ago

Wow, you're the best! Now, I just need to figure out how to get it a readable format in a csv file and get it to do all the pages. Thanks again

Prophet Over a year ago

You are sincerely welcome! As about how to put that in a csv file and paging please open a new questions. SO rules saying each question should be about a single, concrete problem.

Collectives™ on Stack Overflow

Scraping javascript data from table or grid

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related