Creating a pandas dataframe from an unknown number of lists of columns

Question

I have limited my requirements to 5 columns and 3 rows for easy explanation. My column header will come to string and my rows will come to a string. I want all the rows to be added to a dataframe. Here is what I have tried

import pandas as pd

Column_Header = "Col1,Col2,Col3,Col4,Col5" # We have upto 500 columns
df = pd.DataFrame(columns=Column_Header.split(","))


#we will get upto 100000 rows from a server response
Row1 = "Val11,Val12,Val13,Val14,Val15"
Row2 = "Val21,Val22,Val23,Val124,Val25"
Row3 = "Val31,Val32,Val33,Val34,Val35"
df_temp = pd.DataFrame(data = Row1.split(",") , columns = Column_Header.split(","))
pd.concat(df,df_temp)
print(pd)

Every time you received a Row, add it by: df.loc[-1] = Row.split(","), then adjust the index: df.index = df.index + 1 — Ian
– Ian, Commented Sep 19, 2017 at 7:13

jezrael · Accepted Answer · 2017-09-19 07:17:21Z

2

The best and fastest is create list of all data by list comprehension and call DataFrame constructor only once:

Column_Header = "Col1,Col2,Col3,Col4,Col5"
Row1 = "Val11,Val12,Val13,Val14,Val15"
Row2 = "Val21,Val22,Val23,Val124,Val25"
Row3 = "Val31,Val32,Val33,Val34,Val35"

rows = [Row1,Row2,Row3]
L = [x.split(',') for x in rows]

print (L)
[['Val11', 'Val12', 'Val13', 'Val14', 'Val15'], 
 ['Val21', 'Val22', 'Val23', 'Val124', 'Val25'],
 ['Val31', 'Val32', 'Val33', 'Val34', 'Val35']]


df = pd.DataFrame(data = L , columns = Column_Header.split(","))
print (df)
    Col1   Col2   Col3    Col4   Col5
0  Val11  Val12  Val13   Val14  Val15
1  Val21  Val22  Val23  Val124  Val25
2  Val31  Val32  Val33   Val34  Val35

edited Sep 19, 2017 at 7:17

answered Sep 19, 2017 at 7:11

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

cs95 · Accepted Answer · 2017-09-19 07:20:13Z

1

If this is a viable option, it would be simpler to leave all the data munging to pd.read_csv. Convert all your strings to a single multiline string, and pass it through a StringIO buffer to read_csv.

import io    
data = '\n'.join([Column_Header, Row1, Row2, Row3])
df = pd.read_csv(io.StringIO(data))
df

    Col1   Col2   Col3    Col4   Col5
0  Val11  Val12  Val13   Val14  Val15
1  Val21  Val22  Val23  Val124  Val25
2  Val31  Val32  Val33   Val34  Val35

If you're on python2.x, the io module is available as the cStringIO module, so you'd have to import it as:

import cStringIO as io

answered Sep 19, 2017 at 7:20

cs95

406k106 gold badges744 silver badges797 bronze badges

Collectives™ on Stack Overflow

Creating a pandas dataframe from an unknown number of lists of columns

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related