0

I would like to create a csv file from a list of pandas dataframes. My data maintains all of the same column names besides for 1 column which is swapped each time. The following is what my list looks like based on the index of the list:

index 0:

height weight age isFunny
  4      2    21     0

index 1:

height weight age isCool
  10     30   81    34

index 2:

height weight age isDumb
  4      2    34     10

index 3:

height weight age isTalker
  4      12   25     3

I thought that the best way to save this data was in a list of pandas dataframes but if there is a better alternative, then please suggest any.

Improve this question
4
  • What do you want the CSV to look like? Commented Jul 26, 2019 at 16:07
  • Something that clearly depicts the information above. I would like to maintain the objects at each index so that I could easily identify associated information. Commented Jul 26, 2019 at 16:12
  • Are these all the same samples in different dataframes that you'd like to merge into one file or does each dataframe have separate samples that are also measured differently? Commented Jul 26, 2019 at 16:42
  • All of them maintain the exact same column names besides for the last column. Commented Jul 26, 2019 at 16:49

2 Answers 2

Reset to default
3

You can use reduce to create a single dataframe and save it to a csv.

df_1 = pd.DataFrame([[4,2,21,0]],columns='height weight age isFunny'.split())
df_2 = pd.DataFrame([[10,30,81,35]],columns='height weight age isCool'.split())
df_3 = pd.DataFrame([[4,2,34,10]],columns='height weight age isDumb'.split())
df_4 = pd.DataFrame([[4,12,25,3]],columns='height weight age isTalker'.split())
l = [df_1, df_2, df_3, df_4]

from functools import reduce
>>> reduce(lambda x, y: x.append(y), l)

   age  height  isCool  isDumb  isFunny  isTalker  weight
0   21       4     NaN     NaN      0.0       NaN       2
1   81      10    35.0     NaN      NaN       NaN      30
2   34       4     NaN    10.0      NaN       NaN       2
3   25       4     NaN     NaN      NaN       3.0      12
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

This isn't what I want though. If I wanted this, all you would do is convert the list of dataframes back to a pandas.core.series.Series then pass the list name in to cast the entire list as a pandas dataframe. Simply put you can do: pd.DataFrame(myDF) and you'll achieve the same outcome. But, I don't want that. I want to keep my separate dataframes all held together in a single csv.
1

Assuming myDF is your list of DataFrames you can serialize it like this:

import pickle
pickle.dump(myDF, "my_file.pickle")

Then you can load it whenever with this:

import pickle
myDF = pickle.load("my_file.pickle")

What you want to do can technically be done with a CSV file but it would be unwise to have a CSV with multiple headers. You could do that, though:

my_file_name = "my_file.csv"
[df.to_csv(my_file_name, mode="a+", index=False) for df in myDF]

Don't do this. This is dumb.

Based on your additional context, I would suggest doing this (which is similar in concept to the other answer):

melted_df = pd.concat([df.melt(id_vars = ['height', 'weight', 'age']) for df in myDF])
melted_df.to_csv("my_file.csv")

You may wish to unroll the MultiIndex into columns again.

Improve this answer

8 Comments

Okay, if you don't recommend this then what's the best alternative.
Save separate data sets into separate CSV files or serialize your list into a pickled object file. Cramming these dataframes into one CSV file for no reason makes no sense.
Well I am working with patient data and analyzing different variables of hundreds of patients. We maintain variables that will always be present like demographics, but there are also variables that are swapped out. So, if you have any alternative ideas, I would gladly opt to use them.
Try melting them. That's pretty much the best answer because you can pack them into a single dataframe and CSV file.
It's in the answer.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.