Appending multiple pandas dataframe using for loop but returns an empty dataframe

Question

I am to download a number of .csv files which I convert to pandas dataframe and append to each other.

The csv can be accessed via url which is created each day and using datetime it can be easily generated and put in a list.

I am able to open these individually in the list.

When I try to open a number of these and append them together I get an empty dataframe. The code looks like this so.

#Imports
import datetime 
import pandas as pd


#Testing can open .csv file
data = pd.read_csv('https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin01022018.csv')
data.iloc[:5]


#Taking heading to use to create new dataframe 
data_headings = list(data.columns.values)


#Setting up string for url
path_start = 'https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin'
file = ".csv"


#Getting dates which are used in url 
start = datetime.datetime.strptime("01-02-2018", "%d-%m-%Y")
end = datetime.datetime.strptime("04-02-2018", "%d-%m-%Y")
date_generated = [start + datetime.timedelta(days=x) for x in range(0, (end-start).days)]


#Creating new dataframe which is appended to
for heading in data_headings:
    data = {heading: []}

df = pd.DataFrame(data, columns=data_headings)


#Creating list of url
date_list = []

for date in date_generated:
    date_string = date.strftime("%d%m%Y")

    x = path_start + date_string + file
    date_list.append(x)


#Opening and appending csv files from list which contains url
for full_path in date_list:
    data_link = pd.read_csv(full_path)
    df.append(data_link)

print(df)

I have checked that they are not just empty csv but they are not. Any help would be appreciated.

Cheers, Sandy

Try commenting out the for loop of data_headings. You are deleting your data in that part of the code. I'm not sure what you are trying to accomplish with that section. — Cilantro Ditrek
– Cilantro Ditrek, Commented Feb 12, 2019 at 19:10
The loop for data_headings is only used to create the dataframe which is used to append the dataframes that are downloaded from url — Alexander Caskie
– Alexander Caskie, Commented Feb 12, 2019 at 19:13
You say "I am to download a number of .csv files which I convert to pandas dataframe and append to each other." Is it that you want to append the three csvs that you put in date_list together? — Cilantro Ditrek
– Cilantro Ditrek, Commented Feb 12, 2019 at 19:27
Yes, that is what I would like to do. Sorry if this was not cleat. — Alexander Caskie
– Alexander Caskie, Commented Feb 12, 2019 at 19:27

Cilantro Ditrek · Accepted Answer · 2019-02-12 19:35:34Z

3

You are never storing the appended dataframe. The line:

df.append(data_link)

Should be

df = df.append(data_link)

However, this may be the wrong approach. You really want to use the array of URLs and concatenate them. Check out this similar question and see if it can improve your code!

answered Feb 12, 2019 at 19:35

Cilantro Ditrek

1,1071 gold badge16 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alexander Caskie Over a year ago

My god! Always something simple. Cheers chief.

Cilantro Ditrek Over a year ago

Glad I could help! If I've answered your question, can you go ahead and accept the answer? That way, folks will no your issue has been resolved.

Lucas Hort · Accepted Answer · 2019-02-12 19:22:24Z

0

I really can't understand what you wanted to do here:

#Creating new dataframe which is appended to
for heading in data_headings:
    data = {heading: []}

df = pd.DataFrame(data, columns=data_headings)

By the way, try this:

for full_path in date_list:
    data_link = pd.read_csv(full_path)
    df.append(data_link.copy())

answered Feb 12, 2019 at 19:22

Lucas Hort

8547 silver badges10 bronze badges

Collectives™ on Stack Overflow

Appending multiple pandas dataframe using for loop but returns an empty dataframe

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related