How to construct data frame from Web Scraping in Python

Question

I can fetch data from web page thru web scraping in Python. My data is fetched into a list. But don't know how to transform that list into a data frame. Is there any way I could web scrape and fetch data directly to a df? Here is my code:

import pandas as pd
import requests
from bs4 import BeautifulSoup
from tabulate import tabulate
from pandas import DataFrame
import lxml

# GET the response from the web page using requests library
res = requests.get("https://www.worldometers.info/coronavirus/")

# PARSE and fetch content using BeutifulSoup method of bs4 library
soup = BeautifulSoup(res.content,'lxml')
table = soup.find_all('table')[0]
df = pd.read_html(str(table))

# Here dumping the fetched data to have a look
print( tabulate(df[0], headers='keys', tablefmt='psql') )
print(df[0])

αԋɱҽԃ αмєяιcαη · Accepted Answer · 2020-04-03 09:38:28Z

1

import requests
import pandas as pd

r = requests.get("https://www.worldometers.info/coronavirus/")
df = pd.read_html(r.content)[0]

print(type(df))

# <class 'pandas.core.frame.DataFrame'>

df.to_csv("data.csv", index=False)

Output: view

answered Apr 3, 2020 at 9:38

αԋɱҽԃ αмєяιcαη

11.6k3 gold badges23 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jamil Rahman Over a year ago

Thanks for your solution. Very straight forward, learned something new today.

UJIN · Accepted Answer · 2020-04-03 09:37:52Z

1

Well read_html returns a list of DataFrames (as per documentation), so you have to get the "first" (and only) element of that list.

I would just add at the end (after you call read_html):

df = df[0]

Then you can inspect its info getting:

df.info()

# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 207 entries, 0 to 206
# Data columns (total 10 columns):
# Country,Other       207 non-null object
# TotalCases          207 non-null int64
# NewCases            59 non-null object
# TotalDeaths         144 non-null float64
# NewDeaths           31 non-null float64
# TotalRecovered      154 non-null float64
# ActiveCases         207 non-null int64
# Serious,Critical    112 non-null float64
# Tot Cases/1M pop    205 non-null float64
# Deaths/1M pop       142 non-null float64
# dtypes: float64(6), int64(2), object(2)
# memory usage: 16.3+ KB

answered Apr 3, 2020 at 9:37

UJIN

1,78816 silver badges34 bronze badges

1 Comment

Jamil Rahman Over a year ago

I could select only one solution, and both of solutions works for me. Thanks anyway.

Collectives™ on Stack Overflow

How to construct data frame from Web Scraping in Python

2 Answers 2

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related