Pandas - Appending data from one Dataframe to

Question

I have a Dataframe (called df) that has list of tickets worked for a given date. I have a script that runs each day where this df gets generated and I would like to have a new master dataframe (lets say df_master) that appends values form df to a new Dataframe. So anytime I view df_master I should be able to see all the tickets worked across multiple days. Also would like to have a new column in df_master that shows date when the row was inserted.

Given below is how df looks like:

I tried to perform concat but it threw an error

TypeError: first argument must be an iterable of pandas objects, you passed an object of type "Series"

Update

df_ticket = tickets['ticket']
df_master = df_ticket
df_master['Date'] = pd.Timestamp('now').normalize()
L = [df_master,tickets] 
master_df = pd.concat(L)
master_df.to_csv('file.csv', mode='a', header=False, index=False)

What did you try so far? also please post a sample output. for appending dfs generated in a loop see this — anky
– anky, Commented Dec 21, 2018 at 7:16

jezrael · Accepted Answer · 2018-12-21 08:28:00Z

2

I think you need pass sequence to concat, obviously list is used:

objs : a sequence or mapping of Series, DataFrame, or Panel objects

If a dict is passed, the sorted keys will be used as the keys argument, unless it is passed, in which case the values will be selected (see below). Any None objects will be dropped silently unless they are all None in which case a ValueError will be raised

L = [s1,s2] 
df = pd.concat(L)

And it seems you pass only Series, so raised error:

df = pd.concat(s)

For insert Date column is possible set pd.Timestamp('now').normalize(), for master df I suggest create one file and append each day DataFrame:

df_ticket = tickets[['ticket']]
df_ticket['Date'] = pd.Timestamp('now').normalize()
df_ticket.to_csv('file.csv', mode='a', header=False, index=False)

df_master = pd.read_csv('file.csv',  header=None)

edited Dec 21, 2018 at 8:28

answered Dec 21, 2018 at 7:17

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

dark horse Over a year ago

thanks for the help. I have one issue however. I see the time current date gets added to the same column as the ticket. I am unable to add a new column to the Dataframe df_master..

jezrael Over a year ago

@darkhorse - I think df_master with concat is not necessary, only add column with date and append to file. And if need all data use df_master = pd.read_csv('file.csv', header=None), check edited answer.

dark horse Over a year ago

I tried the above and it still does the same. Adds the timestamp as a new row and not as a column.

jezrael Over a year ago

@darkhorse - you are right, need df_ticket = tickets[['ticket']] instaed df_ticket = tickets['ticket'] for one column DataFrame

Collectives™ on Stack Overflow

Pandas - Appending data from one Dataframe to

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related