Web-scraping a javascript table with python BueatifulSoup

Question

I can't get one javascript table with BueatifulSoup, returning empty array

I tried to get data from this page. https://www.hkex.com.hk/Mutual-Market/Stock-Connect/Statistics/Historical-Daily?sc_lang=en#select4=1&select5=2&select3=0&select2=3&select1=24

import requests, json
text = requests.get("https://www.hkex.com.hk/Mutual-Market/Stock-Connect/Statistics/Historical-Daily?sc_lang=en#select4=0&select5=2&select3=0&select2=3&select1=24")
data = json.loads(text)

print(data['Scty'])

the webpage you are downloading contains html not json, so Python throws a error. Also where is your code using Bueatiful Soup? — Raymond
– Raymond, Commented Apr 26, 2019 at 3:46
I started with BeutifulSoap, but didnt work. So I tried with Json — padulla
– padulla, Commented Apr 26, 2019 at 3:48

QHarr · Accepted Answer · 2019-04-26 04:25:45Z

1

There is another url you can use - found by looking at the network tab. A little string manipulation on the response text and you have a string that can be loaded with json and contains everything on the page (including for all 4 drop down geographies). There is no need for bs4. You can extract everything you want with json library.

Explore it here.

import requests
import json

r = requests.get('https://www.hkex.com.hk/eng/csm/DailyStat/data_tab_daily_20190425e.js?_=1556252093686')
data = json.loads(r.text.replace('tabData = ',''))

For example, path to first row of table on landing page:

edited Apr 26, 2019 at 4:25

answered Apr 26, 2019 at 4:19

QHarr

84.5k14 gold badges58 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Web-scraping a javascript table with python BueatifulSoup

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related