How to fetch historical ESG data from Yahoo?

Question

I'm trying to scrape historical ESG data from Sustainalytics on Yahoo Finance using Python. Specifically, let's say I want the last 10 years of ESG scores of a given list of constituents.

The following code lines provides up-to-date ESG scores. But I would like to scrape past-ESG performance. I am essentially looking for yearly (monthly, if possible) ESG from Jan 2010 till Dec 2020. I would like to automate the scraping and save the data in a txt or csv file.

# import yfinance, pandas and os
import yfinance as yf
import pandas as pd
import os

Code for a single ticker:

msft = "MSFT"
msft_y = yf.Ticker(msft)
esg_data = pd.DataFrame.transpose(msft_y.sustainability)
esg_data['company_ticker'] = str(msft_y ticker)

It returns a dataframe of 27 rows covering ESG-relevant information for Microsoft.

Code for the S&P 500:

# Import list of tickers from file
os.chdir("C:\...")
sp_500 = pd.read_csv('SP_500_tickers.csv')
# Retrieve Yahoo! Finance Sustainability Scores for each ticker
for i in sp_500['ticker_code']:
    # print(i)
    i_y = yf.Ticker(i)
    try:
        if i_y.sustainability is not None:
            temp = pd.DataFrame.transpose(i_y.sustainability)
            temp['company_ticker'] = str(i_y.ticker)
            # print(temp)
            esg_data = esg_data.append(temp)
    except IndexError:
        pass

It returns a dataframe of ESG data for the S&P500 constituents, which we can use for analysis. The idea behind is to create portfolios of “good” and “bad” ESG companies and compare performance to see how share prices have performed during different historical time periods.

So far, these codes cannot fetch ESG data for past dates.

Does yfinance come with documentation regarding historic periods? — QHarr
– QHarr, Commented May 4, 2021 at 5:47

PANsg · Accepted Answer · 2021-10-20 11:21:15Z

3

I was not able to use requests.get as the connection was forbidden. Error 403. So I tried using urllib.request.urlopen from the StackOverflow board below. Visit Python; urllib error: AttributeError: 'bytes' object has no attribute 'read'

import pandas as pd
from datetime import datetime as dt
import urllib.request
import json
dataframes = []
url = "https://query2.finance.yahoo.com/v1/finance/esgChart?symbol=A"

connection = urllib.request.urlopen(url)

data = connection.read()
data_2 = json.loads(data)
Formatdata = data_2["esgChart"]["result"][0]["symbolSeries"]
Formatdata_2 = pd.DataFrame(Formatdata)
Formatdata_2["timestamp"] = pd.to_datetime(Formatdata_2["timestamp"], unit="s")

Preview prints the data as shown by putty

print(Formatdata_2.head())

The rest of the code was adapted from putty

edited Oct 20, 2021 at 11:21

answered Oct 20, 2021 at 10:57

PANsg

463 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

putty · Accepted Answer · 2021-05-04 15:49:50Z

1

There's a Yahoo Finance endpoint you can use that should give you monthly ESG scores, governance score, environment score, and social score. Don't think it goes back that far though:

import pandas as pd
import requests

# Read in your symbols
sp_500 = pd.read_csv("SP_500_tickers.csv")

# Endpoint
url = "https://query2.finance.yahoo.com/v1/finance/esgChart"

# List of dataframes
dataframes = []

for symbol in sp_500["ticker_code"]:
    response = requests.get(url, params={"symbol": symbol})
    if response.ok:
        df = pd.DataFrame(response.json()["esgChart"]["result"][0]["symbolSeries"]
        df["symbol"] = symbol
        dataframes.append(df)

df = pd.concat(dataframes)
df["timestamp"] = pd.to_datetime(df["timestamp"], unit="s")

Preview:

>>> df.head()
   timestamp  esgScore  governanceScore  environmentScore  socialScore symbol
0 2014-09-01      61.0             62.0              74.0         45.0   AAPL
1 2014-10-01      60.0             62.0              74.0         45.0   AAPL
2 2014-11-01      61.0             62.0              74.0         45.0   AAPL
3 2014-12-01      61.0             62.0              74.0         45.0   AAPL
4 2015-01-01      61.0             62.0              74.0         45.0   AAPL

answered May 4, 2021 at 15:49

putty

7641 gold badge6 silver badges15 bronze badges

2 Comments

pinpss Over a year ago

This is amazing! Thank you

anonim Over a year ago

Could you please send me the SP_500_tickers.csv file?

jason hill · Accepted Answer · 2023-05-07 11:27:41Z

0

1 Go to Yahoo Finance. 2 Enter a quote into the search field. 3 Select a quote in the search results to view it. 4 Click Historical Data. 5 Select a Time Period, data to Show, and Frequency. 6 Click Apply. 7 To use the data offline, click Download.

answered May 7, 2023 at 11:27

jason hill

11 bronze badge

Collectives™ on Stack Overflow

How to fetch historical ESG data from Yahoo?

3 Answers 3

Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related