Trying to scrape HTML table and convert into dataframe in Python. Code not working correctly

Question

I'm trying to scrape the coinmarketcap website for its historical daily data (in a HTML table), but not getting the correct result. Below is the code. The code only returns the last row of the table. I'm doing something wrong with the loop...any help is greatly appreciated!

import requests
from bs4 import BeautifulSoup
import pandas as pd

data = requests.get('https://coinmarketcap.com/currencies/ethereum/historical-data') 

soup = BeautifulSoup(data._content, 'lxml')

table = soup.find_all('table')[0]

#the table has 7 columns, about 30 rows
new_table = pd.DataFrame(columns=range(0,7), index = [0]) 

row_marker = 0
for row in table.find_all('tr'):    
    column_marker = 0
    columns = row.find_all('td')
    for column in columns:
        new_table.iat[row_marker,column_marker] = column.get_text()
        column_marker += 1
print (new_table.head())

jezrael · Accepted Answer · 2017-11-10 07:59:52Z

1

Use read_html:

url = 'https://coinmarketcap.com/currencies/ethereum/historical-data'

df = pd.read_html(url, parse_dates=[0])[0]
print (df.head())
        Date    Open    High     Low   Close     Volume   Market Cap
0 2017-11-09  308.64  329.45  307.06  320.88  893250000  29509000000
1 2017-11-08  294.27  318.70  293.10  309.07  967956000  28128700000
2 2017-11-07  298.57  304.84  290.77  294.66  540766000  28533300000
3 2017-11-06  296.43  305.42  293.72  298.89  579359000  28322700000
4 2017-11-05  300.04  301.37  295.12  296.26  337658000  28661500000

answered Nov 10, 2017 at 7:59

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Trying to scrape HTML table and convert into dataframe in Python. Code not working correctly

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related