minimize memory consumption when dealing with python list assignment

Question

I have two high-dimension lists of integers. The dimension of each list is over 200000. My task is like this: given two list

a = [None, 0, None, None, 0, None, None, None, 0, None, None, None, None, None, 0,...]
b = [7 1 4 8 2 1 1 1 1 6 1 6 1 4 4 1 1 6 6 7 4 4 7 5...]

the number of Noneequals to the length of b

My question is how to replace all None of a with each element in busing minimum memory.

My method is

j = 0
for i in range(len(a)):
    if a[i] == None:
        while j < len(b):   
            a[i] = b[j]
            break
        j = j+1

Since my program consumes memory much, I really need to save memory use. Could you please give me some ideas on how to save the memory in this situation?

to check the memory maybe can be useful this package github.com/fabianp/memory_profiler — JuanPablo
– JuanPablo, Commented Jun 24, 2015 at 14:36
Which version of Python are you using, 2 or 3? If this is Python 2, then you should use xrange instead of range, since that's the only thing in your code that uses any significant amount of memory. The arrays a and b occupy memory too, of course, but since those are the inputs you can't avoid that in this code. — Steve Jessop
– Steve Jessop, Commented Jun 24, 2015 at 14:43

Cory Kramer · Accepted Answer · 2015-06-24 14:43:39Z

3

You can make an iterator from b, then in a list comprehension if the current element of a is None, you can next(b_iter) to grab the next item.

b_iter = iter(b)
a = [next(b_iter) if i is None else i for i in a]

As an example

>>> a = [None, 0, None, None, 0, None, None, None, 0, None, None, None, None, None, 0]
>>> b = [7, 1, 4, 8, 2, 1, 1, 1, 1, 6, 1]
>>> b_iter = iter(b)
>>> [next(b_iter) if i is None else i for i in a]
[7, 0, 1, 4, 0, 8, 2, 1, 0, 1, 1, 1, 6, 1, 0]

edited Jun 24, 2015 at 14:43

answered Jun 24, 2015 at 14:34

Cory Kramer

119k19 gold badges176 silver badges233 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

allenwang Over a year ago

Thank you for your kind answer. But could you please explain more on why using iterator makes memory consumption minimum?

Cory Kramer Over a year ago

@allenwang Because an iterator only accesses one element at a time. It doesn't create an entire copy of the list. It simply walks (or iterates) along a sequence, and retrieves an element each time you request one, in this case by using the function next

Collectives™ on Stack Overflow

minimize memory consumption when dealing with python list assignment

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related