Build a list of list efficiently in python

Question

I have the following code:

neig = []
for _ in xrange(number):
    r = random.randint(0,self._tam-1)
    s = random.randint(0,self._tam-1)
    neig.append([r,s, self.deltaC(r, s)])

I am trying to get rid of the for loop using map, a generator or something to optimize the build of the list. I've checked this answer, but in my case I need to give values to the elements.

Would it be possible to use a generator and delay the call to deltaC? I think using map is not possible because the body of the loop has more than one statement.

What do you mean "delay the call to deltaC? When will you need that value; will you need it for all values in neig, and more than once? — jonrsharpe
– jonrsharpe, Commented Apr 13, 2014 at 8:59
I will be using the value of deltaC later in the program. If I have an element of this list of list [1,2,deltaC(1,2)], ¿Could it be evaluate later?. Maybe I misunderstood the concept of generators. If delay the call is not possible, How could I generate this array more efficiently? — Alejandro Alcalde
– Alejandro Alcalde, Commented Apr 13, 2014 at 9:04
Will you use the value more than once? If not, you could just store r and s in the list, then call deltaC(r, s) where you need that value. What is more important - run time or memory use? — jonrsharpe
– jonrsharpe, Commented Apr 13, 2014 at 9:06
"Generate this array more efficiently" -- could you clarify what do you mean by more efficient, i.e. what's the problem with the current approach? Is it because deltaC returns a very huge structure, or that number is very large, such that it is not feasible to retain the whole list into memory? If this is the case, generators might help. — YS-L
– YS-L, Commented Apr 13, 2014 at 9:13

Roland Smith · Accepted Answer · 2014-04-13 09:12:46Z

1

Try using list comprehensions;

In [1]: import random

In [2]: number = 20

In [3]: tam = 13

In [4]: r = [random.randint(0, tam) for _ in range(number)]

In [5]: s = [random.randint(0, tam) for _ in range(number)]

In [6]: zip(r, s)
Out[6]: [(3, 12), (6, 12), (7, 12), (5, 1), (1, 0), (4, 12), (4, 5), (6, 2), (10, 6), (6, 11), (12, 6), (10, 2), (5, 2), (3, 2), (3, 11), (13, 2), (0, 2), (7, 0), (9, 13), (0, 12)]

In [7]: def deltaC(a, b):
   ...:     return a + b
   ...: 

In [8]: neig = [[p, q, deltaC(p, q)] for p, q in zip(r, s)]

In [9]: neig
Out[9]: [[3, 12, 15], [6, 12, 18], [7, 12, 19], [5, 1, 6], [1, 0, 1], [4, 12, 16], [4, 5, 9], [6, 2, 8], [10, 6, 16], [6, 11, 17], [12, 6, 18], [10, 2, 12], [5, 2, 7], [3, 2, 5], [3, 11, 14], [13, 2, 15], [0, 2, 2], [7, 0, 7], [9, 13, 22], [0, 12, 12]]

answered Apr 13, 2014 at 9:12

Roland Smith

43.8k3 gold badges69 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

unutbu · Accepted Answer · 2014-04-13 09:26:07Z

1

You could use a generator expression:

neig = ((r, s, self.deltaC(r, s))
        for r, s in ([random.randint(0,self._tam-1),
                      random.randint(0,self._tam-1)]
                     for _ in xrange(number)))

provided you only need to iterate over neig once.

This will delay the calls to self.deltaC(r, s) until they are needed as neig is iterated over.

edited Apr 13, 2014 at 9:26

answered Apr 13, 2014 at 9:14

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

1 Comment

Alejandro Alcalde Over a year ago

Thank you, but I didn't remember that I need to sort all items in the list by the value of deltaC, so it seems a generator is not going to work here.

U2EF1 · Accepted Answer · 2014-04-13 09:30:14Z

1

The best way I recommend would be to use numpy:

import numpy as np
r, s = np.random.randint(0, self._tam, (2, number))
neig = np.concatenate([r, s, self.deltaC(r, s)])

This assumes deltaC can be written to accept vectors of r's and s's, which should be pretty easy. Note that np.random.randint is exclusive on the upper bound (unlike random.randint).

answered Apr 13, 2014 at 9:30

U2EF1

13.4k3 gold badges38 silver badges38 bronze badges

1 Comment

Alejandro Alcalde Over a year ago

I will try it later, now I have to go, deltaC accept two integers representing index in a list. If you have experience with numpy I would thank you if you can help me optimizing deltaC (stackoverflow.com/q/23033416/1612432) Because I have a huge bottleneck there.

Collectives™ on Stack Overflow

Build a list of list efficiently in python

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related