Mean and Standard deviation across multiple arrays using numpy

Question

I’m importing 2-D matrix data for a multi year climate time series testing on a 5 year annual dataset. I’ve created a for loop to import the 2D matrix data by year into a series of 5 separate arrays of size (1500, 3600). I append the matrix time series data into a single combined (5, 1500, 3600) array with each year being one dimension in the array. I then run the np.mean and np.std to create (1500, 3600) matrices calculating the 5 year mean and stddev of the data at each matrix point. Code is below. The numbers look to be coming out correctly when I test this but I would like to know ..

Is there a faster way to do this? I will eventually need to run this type of analysis for daily data over an 18 time span which would be building and operating on a (6570, 1500, 3600) array. Any suggestions? I’m fairly new to Python and still finding my way.

StartYear=2009
EndYear=2014
for x in range(StartYear, EndYear):
    name = "/dir/climate_variable" + str(x) + ".gz"
    Q_WBM = rg.grid(name)
    Q_WBM.Load()
    q_wbm = Q_WBM.Data  # .flatten()
    q_wbm[np.isnan(q_wbm)] = 0
    if x == StartYear:
        QTS_array = q_wbm
    else:
        QTS_array = np.append(QTS_array, q_wbm, axis=0)

DischargeMEAN = np.mean(QTS_array, axis=0)
DischargeSTD = np.std(QTS_array, axis=0)

Paul Panzer · Accepted Answer · 2018-12-07 01:31:43Z

1

Unlike list.append which is amortized O(1) numpy.append is pretty much O(n), meaning your loop is O(n^2) and will be no fun to use on your full problem.

On top of that, 6570 x 1500 x 3600 x itemsize is actually quite large and won't fit into memory unless you have a lot of that.

If all you want are mean and SD, then you can sidestep both these problems by summing on the fly. You would replace the end of your code by something like

    if x == StartYear:
        mom1 = q_wbm
        mom2 = q_wbm**2
    else:
        mom1 += q_wbm
        mom2 += q_wbm**2

DischargeMEAN = mom1 / n
DischargeSTD = np.sqrt(mom2 / n - DischargeMEAN**2)

answered Dec 7, 2018 at 1:31

Paul Panzer

53.3k3 gold badges59 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Mean and Standard deviation across multiple arrays using numpy

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related