Pandas - Concatenating Dataframes

Question

I have a script with if statements that has 14 possible dataframes

['result_14', 'result_13', 'result_12', 'result_11', 'result_10', 'result_9', 'result_8', 'result_7', 'result_6', 'result_5', 'result_4', 'result_3', 'result_2', 'result_1']

Not all dataframes are created every time I run the script. It is dependent on a secondary input variable. I am now attempting to concatenate dataframes but run into issue with those that do not exist.

pd.concat(([result_14, result_13, result_12, result_11, result_10, result_9, result_8, result_7, result_6, result_5, result_4, result_3, result_2, result_1]), ignore_index=True)

NameError: name 'result_13' is not defined

I have tried finding all dfs that exist in my python memory and parsing the results but this creates a list rather than a list of dataframes

alldfs = [var for var in dir() if isinstance(eval(var), pd.core.frame.DataFrame)]
SelectDFs = [s for s in alldfs if "result" in s]
SelectDFs

['result_14', 'result_15', 'result_12', 'result_11', 'result_10', 'result_9', 'result_8', 'result_7', 'result_6', 'result_5', 'result_4', 'result_3', 'result_2', 'result_1']

pd.concat(([SelectDFs]), ignore_index=True)
TypeError: cannot concatenate object of type '<class 'list'>'; only Series and DataFrame objs are valid

BENY · Accepted Answer · 2022-06-14 13:41:24Z

2

You can try

%who_ls  DataFrame
# %whos DataFrame

In your case

l = %who_ls  DataFrame
pd.concat([eval(dfn) for dfn in l if dfn.startswith('result')], ignore_index=True)

edited Jun 14, 2022 at 13:41

answered Jun 14, 2022 at 13:28

BENY

324k22 gold badges176 silver badges250 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

SMar3552 Over a year ago

I have other dataframes loaded in memory that I would like to ignore. is there a way to only choose those that have the 'result' prefix?

BENY Over a year ago

@JSimon check the update

SMar3552 Over a year ago

thanks for your help. though it is now being read as a string - throwing an error for concatenating:

SMar3552 Over a year ago

TypeError: cannot concatenate object of type '<class 'str'>'; only Series and DataFrame objs are valid

BENY Over a year ago

@JSimon that is use df name which is str , to convert to the df ~

|

GodWin1100 · Accepted Answer · 2022-06-14 13:24:22Z

0

You are passing list of string and not Dataframe object.

And once you re able to get DF Object you can pass SelecteDFs without bracket.

pd.concat(SelectDFs, ignore_index=True)

answered Jun 14, 2022 at 13:24

GodWin1100

1,4301 gold badge8 silver badges14 bronze badges

Comments

hessam_kk · Accepted Answer · 2022-06-14 13:25:36Z

0

Have you tried to convert them into DFs? I mean when you want to concat them, it raise an error which says your data need to be dfs rahter than lists, so have you tried to convert your lists into DFs?

this link may help you: Convert List to Pandas Dataframe Column

answered Jun 14, 2022 at 13:25

hessam_kk

3191 silver badge12 bronze badges

1 Comment

SMar3552 Over a year ago

they are already dataframes which is how I am detecting the names in my loaded memory. the issue is that the code above reads them as a list

Collectives™ on Stack Overflow

Pandas - Concatenating Dataframes

3 Answers 3

7 Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related