Web Scraping Javascript Table using BeautifulSoup

Question

I am relatively new to web scraping and prototyping using various websites. I am having difficulties with scraping what seems to be Javascript loaded tables. Any help would be much appreciated. The following is my code:

import requests
from bs4 import BeautifulSoup


url='https://onlineservice.cvo.org/webs/cvo/register/#/search/
toronto/0/1/0/10'

response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
tables = soup.find_all(class_='table')
print(tables)

That table is loaded with js, so you first need a client that's able to run the js on that page and then parse the result...I believe selenium is the recommended tool for doing that. But you can also check this tutorial and see if it helps: youtube.com/watch?v=FSH77vnOGqU — Amr Noman
– Amr Noman, Commented Apr 30, 2018 at 21:12

SIM · Accepted Answer · 2021-06-08 14:54:53Z

1

Try the below url to get all the information with the blink of an eye. You can retrieve that url by using chrome dev tools at xhr request under network tab. Give it a shot:

import requests

URL = 'https://onlineservice.cvo.org/rest/public/registrant/search/?query=%20toronto&status=0&type=1&skip=0&take=427'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36',
}

response = requests.get(URL,headers=headers,verify=False)

for items in response.json()['result']:
    lastname = items['lastName']
    firstname = items['firstName']
    commonname = items['commonName']
    status = items['registrationStatus']['name']
    print(lastname,firstname,commonname,status)

Partial results:

Aadoson Andres Andres Active
Aarabi Alireza Allen Active
Aarnes Turi Turi Expired
Abbasi Tashfeen Tashfeen Active
Abbott Jonathan Jonathan Resigned
Abd El Nour Emad Emad Active
Abdel Hady Medhat Hady Active
Abdelhalim Khaled Khaled Active

edited Jun 8, 2021 at 14:54

answered Apr 30, 2018 at 21:30

SIM

22.5k6 gold badges45 silver badges116 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

the_dummy Over a year ago

I am at the Network tab and I clicked on the xhr request but I can't find the url

SIM Over a year ago

Go through this site to understand how it works.

Collectives™ on Stack Overflow

Web Scraping Javascript Table using BeautifulSoup

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related