Fetching table data from webpage using selenium in python

Question

I am very new to web scraping. I have the following url:

https://www.bloomberg.com/markets/symbolsearch

So, I use Selenium to enter the Symbol Textbox and press Find Symbols to get the details. This is the code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("https://www.bloomberg.com/markets/symbolsearch/")
element = driver.find_element_by_id("query")
element.send_keys("WMT:US")
driver.find_element_by_name("commit").click()

It returns the table. How can I retrieve that? I am pretty clueless.

Second question, Can I do this without Selenium as it is slowing down things? Is there a way to find an API which returns a JSON?

bloomberg.com/professional/support/api-library

Thomas
– Thomas

2017-05-08 09:34:50 +00:00
Commented May 8, 2017 at 9:34 — Thomas
– Thomas, Commented May 8, 2017 at 9:34

Exprator · Accepted Answer · 2017-05-08 09:59:02Z

1

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from bs4 import BeautifulSoup
import requests
driver = webdriver.Firefox()
driver.get("https://www.bloomberg.com/markets/symbolsearch/")
element = driver.find_element_by_id("query")
element.send_keys("WMT:US")
driver.find_element_by_name("commit").click()
time.sleep(5)
url = driver.current_url
time.sleep(5)

parsed = requests.get(url)


soup = BeautifulSoup(parsed.content,'html.parser')
a = soup.findAll("table", { "class" : "dual_border_data_table" })

print(a)

here is the total code by which you can get the table you are looking for. now do what you need to do after getting the table. hope it helps

answered May 8, 2017 at 9:59

Exprator

27.8k6 gold badges54 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Fetching table data from webpage using selenium in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related