multiple array to dataframe pandas

Question

So, I am iterating through a dictionary and taking a bunch of values out as a array - Trying to make a Dataframe with each observation as a separate row.

X1 =[]
for k,v in DF_grp:
    date = v['Date'].astype(datetime)
    usage = v['Usage'].astype(float)
    comm = v['comm'].astype(float)
    mdf = pd.DataFrame({'Id' : k[0],'date':date,'usage':usage, 'comm':comm})
    mdf['used_ratio'] = ((mdf['used']/mdf['comm']).round(2))*100
    ts = pd.Series(mdf['usage'].values, index=mdf['date']).sort_index(ascending=True)
    ts2 = pd.Series(mdf['used_ratio'].values, index = mdf['date']).sort_index(ascending=True)
    ts2 = ts2.dropna()
    data = ts2.values.copy()
    if len(data) == 10:
        X1 =np.append(X1,data, axis=0)
        print(X1)

[0,0,0,0,1,0,0,0,1]
[1,2,3,4,5,6,7,8,9]
[0,5,6,7,8,9,1,2,3]
....

similarly, so the question is how do I capture all these arrays in a single DataFrame so that it looks like below:

[[0,0,0,0,1,0,0,0,1]] --- #row 1 in dataframe 
[[1,2,3,4,5,6,7,8,9]] --- #row 2 in dataframe

If the same task can be divided further ? There are more thank 500K arrays in the dataset. Thank You

Neha Bhushan · Accepted Answer · 2018-12-05 00:05:19Z

3

I hope below mentioned code helps you:

arr2 = [0,0,0,0,1,0,0,0,1]
arr3 = [1,2,3,4,5,6,7,8,9]
arr4 = [0,5,6,7,8,9,1,2,3]

li = [arr2, arr3, arr4]
pd.DataFrame(data = li, columns= ["c1", "c2", "c3", "c4", "c5","c6", "c7", "c8", "c9"])

You can make it more dynamic by simply creating one temp_arr and appending that array to list. and creating data frame from generated list of arrays. Also, you can add name to columns(shown above) or avoid naming them(just remove column detailing). I hope that solves your problem

answered Dec 5, 2018 at 0:05

Neha Bhushan

1431 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

harold_noobie Over a year ago

Edited the Question now, I have more than 500K arrays.

Neha Bhushan Over a year ago

So to understand it correctly, you have 500, 000 arrays with each array having how many elements?

harold_noobie Over a year ago

the number of elements are different but currently Im looking at just with len == 10.

meW · Accepted Answer · 2018-12-05 00:30:39Z

0

Declare an empty dataframe in second line i.e. below X1=[] with code df = pd.DataFrame(). Next, inside your IF statement pass the following after appending values to X1:

df = pd.concat([df, pd.Series(X1)]).T

Or,

df = pd.DataFrame(np.NaN, index=range(3), columns=range(9))
for i in range(3):
    df.iloc[i,:] = np.random.randint(9)    # <----- Pass X1 here
df
#     0 1   2   3   4   5   6   7   8
# 0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0
# 1 7.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0
# 2 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0

edited Dec 5, 2018 at 0:30

answered Dec 5, 2018 at 0:07

meW

3,97710 silver badges27 bronze badges

5 Comments

harold_noobie Over a year ago

Thanks, I had tried a similar approach but his breaks at the transpose with --- 'Reindexing only valid with uniquely valued Index objects'

meW Over a year ago

then declare a dataframe with known shape and NaN values. Later use iloc command to fill it. Cut the unnecessary rows later.

harold_noobie Over a year ago

Could you provide an example, I did not get it.

meW Over a year ago

check my modified answer. If you know the number of arrays beforehand then pass the value inside index=range(NUMBER_OF_ARRAYS)

harold_noobie Over a year ago

Tried above now the error is -- ' only size-1 arrays can be converted to Python scalars'

Collectives™ on Stack Overflow

multiple array to dataframe pandas

2 Answers 2

3 Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related