Extract array (column name, data) from Pandas DataFrame

Question

This is my first question at Stack Overflow.

I have a DataFrame of Pandas like this.

        a   b   c   d
one     0   1   2   3
two     4   5   6   7
three   8   9   0   1
four    2   1   1   5
five    1   1   8   9

I want to extract the pairs of column name and data whose data is 1 and each index is separate at array.

[ [(b,1.0)], [(d,1.0)], [(b,1.0),(c,1.0)], [(a,1.0),(b,1.0)] ]

I want to use gensim of python library which requires corpus as this form.

Is there any smart way to do this or to apply gensim from pandas data?

Andy Hayden · Accepted Answer · 2015-01-15 05:52:45Z

1

Many gensim functions accept numpy arrays, so there may be a better way...

In [11]: is_one = np.where(df == 1)

In [12]: is_one
Out[12]: (array([0, 2, 3, 3, 4, 4]), array([1, 3, 1, 2, 0, 1]))

In [13]: df.index[is_one[0]], df.columns[is_one[1]]
Out[13]:
(Index([u'one', u'three', u'four', u'four', u'five', u'five'], dtype='object'),
 Index([u'b', u'd', u'b', u'c', u'a', u'b'], dtype='object'))

To groupby each row, you could use iterrows:

from itertools import repeat

In [21]: [list(zip(df.columns[np.where(row == 1)], repeat(1.0)))
          for label, row in df.iterrows()
          if 1 in row.values]  # if you don't want empty [] for rows without 1
Out[21]:
[[('b', 1.0)],
 [('d', 1.0)],
 [('b', 1.0), ('c', 1.0)],
 [('a', 1.0), ('b', 1.0)]]

In python 2 the list is not required since zip returns a list.

answered Jan 15, 2015 at 5:52

Andy Hayden

378k110 gold badges640 silver badges546 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Zero · Accepted Answer · 2017-08-09 20:00:54Z

0

Another way would be

In [1652]: [[(c, 1) for c in x[x].index] for _, x in df.eq(1).iterrows() if x.any()]
Out[1652]: [[('b', 1)], [('d', 1)], [('b', 1), ('c', 1)], [('a', 1), ('b', 1)]]

answered Aug 9, 2017 at 20:00

Zero

77.4k22 gold badges153 silver badges153 bronze badges

Collectives™ on Stack Overflow

Extract array (column name, data) from Pandas DataFrame

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related