Importing HTML code into CSV using python

Question

I have an HTML with data that I want to bring into python and put into a CSV. I'm not sure which package and program will allow me to complete this as I've tried a few different ones with no success (bs4 and urllib).

This is the HTML link:

https://www.cmegroup.com/CmeWS/mvc/Volume/Details/F/8478/20200807/F?tradeDate=20200807

Out of interest, what kind of HTML link is this? It appears to almost be in CSV format already. Apologies if this is a silly question. I've tried to search file types on the internet too.

I tried a URL request on this web link but received an error when trying to make the request:

from urllib.request import urlopen as uReq

cme_url = "https://www.cmegroup.com/CmeWS/mvc/Volume/Details/F/8478/20200807/F?tradeDate=20200807"

#opening up connection
uClient = uReq(cme_url)

I have scoured StackOver for examples which could solve my questions, but I was unsuccessful. For example, this example didn't help because it's using a specifically CSV file already: Importing CSV into Python

I really appreciate your assistance.

Mike67 · Accepted Answer · 2020-09-09 04:09:14Z

You can read json from a URL and convert it to csv in a couple steps:

Use requests to get the json text and convert it to a dictionary
Use pandas to convert the dictionary to a csv file

I assume you only want the month data.

Here's the code:

import requests
import pandas as pd

url = 'https://www.cmegroup.com/CmeWS/mvc/Volume/Details/F/8478/20200807/F?tradeDate=20200807'
r = requests.get(url)
dj = r.json()
        
df = pd.DataFrame(dj['monthData'])
df.to_csv('out.csv', index=False)

Output (out.csv)

month,monthID,globex,openOutcry,totalVolume,blockVolume,efpVol,efrVol,eooVol,efsVol,subVol,pntVol,tasVol,deliveries,opnt,aon,atClose,change,strike,exercises
AUG 20,AUG-20-Calls,"10,007",0,"10,007",0,0,0,0,0,0,0,0,0,-,-,"9,372","-1,103",0,0
SEP 20,SEP-20-Calls,"1,316",0,"1,316",0,0,0,0,0,0,0,0,0,-,-,"2,899",47,0,0
OCT 20,OCT-20-Calls,115,0,115,0,0,0,0,0,0,0,0,0,-,-,614,32,0,0
NOV 20,NOV-20-Calls,16,0,16,0,0,0,0,0,0,0,0,0,-,-,68,6,0,0
DEC 20,DEC-20-Calls,13,0,13,0,0,0,0,0,0,0,0,0,-,-,105,-3,0,0
JAN 21,JAN-21-Calls,6,0,6,0,0,0,0,0,0,0,0,0,-,-,5,4,0,0
DEC 21,DEC-21-Calls,0,0,0,0,0,0,0,0,0,0,0,0,-,-,1,0,0,0

lim1718 · Accepted Answer · 2021-10-12 03:37:37Z

0

The data format in the URL you provided is almost in JSON.

Your question is "How to convert JSON file to CSV" in fact.

Python itself can solve this problem, with JSON encoder and decoder.

edited Oct 12, 2021 at 3:37

answered Sep 9, 2020 at 3:24

lim1718

12 bronze badges

Collectives™ on Stack Overflow

Importing HTML code into CSV using python

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related