how to implement logical and on array length?

Question

I need to translate the matlab code

indexSelect0 = a.index1==0 & a.index2==wRange;

into a fast python style. My try is:

idx1=np.array(np.where(a['index2'][:,0]==wIndex2))
idx=np.array(np.where(a['index1'][:,0]==wIndex1))
indexSelect0 = ma.masked_array(idx,mask=[not (i in idx1[0,:]) for i in idx[0,:]])

but it takes a while as the array is pretty long (more than 5M of samples).

The problem can be stated as : I have an array of data that is composed of different observations. I have 2 indices that allow me to know where is what. But I am not able to find the right way to combine two options to filter the data.

Hope it is clear.

Thanks for your help

What is your a? Is it dataframe? Could you provide sample data? — Anton Protopopov
– Anton Protopopov, Commented Mar 9, 2016 at 14:13
a is a dictionnary. Could it be easier if a is a dataframe ? — Oscar H.
– Oscar H., Commented Mar 9, 2016 at 14:22

Oscar H. · Accepted Answer · 2016-03-09 15:17:53Z

2

For backup I found the answer. I thank Anton for having directed me to DataFrame

import pandas as pd
d = {'index1': a['index1'][:,0].squeeze(), 'index2': a['index2'][:,0].squeeze(), 'data': x}
df= pd.DataFrame(data=d)
y = df[(df.index1==wIndex1) & (df.index2==wIndex2)]

So I use the DataFrame of the pandas module and the boolean operator for indexing and selecting the data (more here http://pandas.pydata.org/pandas-docs/stable/indexing.html). It works fine : readable, simple for coding and much faster.

answered Mar 9, 2016 at 15:17

Oscar H.

313 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

TheBlackCat · Accepted Answer · 2016-03-24 22:26:16Z

1

The exact equivalent of your MATLAB, with the struct converted to a dict, is:

indexSelect0 = (a['index1'] == 0) & (a['index2'] == wRange)

On my 5-year-old laptop I was able to process 10 million samples in just a few seconds:

n = 10000000
a = {'index1': np.random.randint(0, 10, n), 'index2': np.random.randint(0, 10, n)}
wRange = 5
indexSelect0 = (a['index1'] == 0) & (a['index2'] == wRange)

answered Mar 24, 2016 at 22:26

TheBlackCat

10.4k3 gold badges26 silver badges32 bronze badges

Collectives™ on Stack Overflow

how to implement logical and on array length?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related