A problem with web scraping using python ,BeautifulSoup and pandas 'read_html'

Question

Thank you for the helpers !

I am scraping a table of data about covid19 and push it into a pandas data frame , it was working until this morning .

That the code :

import pandas as pd
import requests
from bs4 import BeautifulSoup


url = 'https://www.worldometers.info/coronavirus/'

req = requests.get(url)

page = BeautifulSoup(req.content, 'html.parser')

table = page.find_all('table',id="main_table_countries_today")[0]

print(table)

df = pd.read_html(str(table))[0]

This morning I starting to get the next error :

ValueError: No tables found matching pattern '.+'

Can you please help me figure it out ?

This code is working fine for me. Returning a list of dataframes. — Anshul
– Anshul, Commented May 29, 2020 at 11:29
I am getting the same error on different ide , can you please show me what you get from the code ? — Mark
– Mark, Commented May 29, 2020 at 11:33
Install covid19 library. It is better pip install COVID19Py https://pypi.org/project/COVID19Py/ — Osadhi Virochana
– Osadhi Virochana, Commented May 29, 2020 at 11:35
@ZurHanin - Not sure whats the issue. Can you check the lib versions for lxml and html5lib, once? — Anshul
– Anshul, Commented May 29, 2020 at 11:38
Thank You ,your api is a really good source , but its importune to me to understand what i did wrong in my code .. — Mark
– Mark, Commented May 29, 2020 at 11:39

screenpaver · Accepted Answer · 2020-05-29 14:47:13Z

3

Try changing the last line to: df = pd.read_html(str(table), displayed_only=False)[0] The table header at the url has changed its style attribute to style="width:100%;margin-top: 0px !important;display:none;". Previously it did not have the 'display' tag set.

answered May 29, 2020 at 14:47

screenpaver

1,1508 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Mark Over a year ago

Thank you very very much , your solution work ! can you please guide me on which subject i can read for deep understanding about that subject ?

screenpaver Over a year ago

Pandas read_html simply provides the option of processing elements with display:none set (see pandas.pydata.org/pandas-docs/stable/reference/api/…). See more on the display property here w3schools.com/css/css_display_visibility.asp. By the way, your original code works again today (May 31), so they changed again.

Collectives™ on Stack Overflow

A problem with web scraping using python ,BeautifulSoup and pandas 'read_html'

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related