Increase speed of numpy nested for loop for huge matrix

Question

I have a python code, which has nested for loop, for a huge matrix. Here rho0, alp0, bet0 are constant. and E is a symmetric matrix. How can i increase is the speed of for loop?

N = 20000
Q = np.zeros(shape=(N,N))               
for i in tqdm(range(0,N)):
    for j in range(0,N):
        nom    = rho0*alp0**E[i,j]*(1-alp0)**(M-E[i,j]);
        dnom   = nom + ( (1-rho0)*bet0**E[i,j]*(1-bet0)**(M-E[i,j]));
        Q[i,j] = nom/dnom;

PyTorch is not magic, and it's not going to speed up your code. It's a numerical computation library just like numpy. — Arya McCarthy
– Arya McCarthy, Commented Feb 22, 2021 at 6:12
Edited my problem statement. is there anyway the speed of loop increased? — Sushmita
– Sushmita, Commented Feb 22, 2021 at 6:27
Yes, we've each provided one way to do that. (Accidentally, the same way, about 15 second apart—it's the natural way to do it!) — Arya McCarthy
– Arya McCarthy, Commented Feb 22, 2021 at 6:29

Arya McCarthy · Accepted Answer · 2021-02-22 06:24:04Z

1

You can tremendously speed your code up by replacing Python looping with hardware-accelerated vector operations and tight C or Fortran loops provided by numpy.

(You didn't define several variables, so I took the liberty of defining them.)

import numpy as np
from tqdm import tqdm

# Set these to the correct values.
rho = 0.6
alp0 = 1.4
bet0 = 2.5
M = 3.6
_E = np.random.rand(N, N)
E = _E + _E.T  # symmetric matrix.

# Look, all 20000 * 20000 values computed in one move!
nom = rho0 * alp0 ** E * (1 - alp0) ** (M - E)  # Shape is 20000x20000
dnim = nom + ((1 - rho0) * bet0 ** E * (1 - bet0) ** (M - E))  # Shape is 20000x20000
Q = nom / dnom

answered Feb 22, 2021 at 6:24

Arya McCarthy

8,8544 gold badges39 silver badges59 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sushmita Over a year ago

Thanks for the solution. yes it improved the speed tremendously.

hpaulj · Accepted Answer · 2021-02-22 06:28:46Z

0

If everything is scalars constants except E, then this should work

    nom    = rho0*alp0**E*(1-alp0)**(M-E)
    dnom   = nom + ( (1-rho0)*bet0**E*(1-bet0)**(M-E))
    Q = nom/dnom

A numpy array E can be multiplied, subtracted etc as a whole. The key to speed in numpy is to work with the compiled whole-array methods (and operators) where possible. Those methods do iterate, but they do so at compiled speeds, rather than the (much) slower Python speed.

There are tools for compiling larger expressions, like numba, but work with the core numpy for a start. Do take some time read numpy basics; it will save you time later.

edited Feb 22, 2021 at 6:28

answered Feb 22, 2021 at 6:23

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Collectives™ on Stack Overflow

Increase speed of numpy nested for loop for huge matrix

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related