Pass large data structure in Python

Question

In C++, it's good practice to pass references to large data structures to avoid the overhead of copying the whole thing by value. Python works very differently, though, and I'm wondering how best to pass a large data structure.

Here's the simple version. Is this what I should use, or is there something better?

foo(hugeArray):
    # Change values in hugeArray
    return hugeArray

hugeArray = foo(hugeArray)

Solution (Kudos to jonrsharpe!)

foo(hugeArray)
    hugeArray[0] = "New!"

myArray[0] = "Old."
foo(myArray)
print myArray[0] # Prints "New!"

The function changes the original array, so there's no need to assign the return value to the original to change it.

If foo mutates hugeArray you would usually (implicitly or explicitly) return None, and simply call foo(hugeArray). See e.g. list.append. As Python passes references (see e.g. here), there is no performance implication to returning hugeArray, though. — jonrsharpe
– jonrsharpe, Commented Aug 8, 2014 at 16:52
For "huge arrays" I would look into generators if at all possible. If not, then look into the numpy's arrays. — James Mertz
– James Mertz, Commented Aug 8, 2014 at 16:53
Have you looked at generators in python? If you are worried about performance implications while passing a large array, generators could be useful. — Santanu C
– Santanu C, Commented Aug 8, 2014 at 17:07
@SantanuC If I were truly concerned about performance, I wouldn't use Python. :P Generators look fascinating, though, and I'll definitely read up. — Qaz
– Qaz, Commented Aug 8, 2014 at 17:12
@KronoS Arrays are just an example, but the code I'm working on does indeed use numpy for them, and I'll definitely look into generators. Thanks! — Qaz
– Qaz, Commented Aug 8, 2014 at 17:21

score 1 · Accepted Answer · 2021-03-10 10:53:29Z

1

No need to copy stuff

def foo(array):
    array[0] = 3  # do stuff to the array in place

huge_array = [1, 2, 3]
foo(huge_array)    # returns None
print(huge_array)  # [3, 2, 3]

edited Mar 10, 2021 at 10:53

answered Aug 8, 2014 at 16:59

user3064538

Sign up to request clarification or add additional context in comments.

Comments

ideasman42 · Accepted Answer · 2014-08-08 16:55:33Z

1

Python will do this for you, unless you explicitly copy it or modify.

foo(hugeArray):
    # Change values in hugeArray
    return hugeArray

hugeArray2 = foo(hugeArray)
print("Was Python Clever? - ", hugeArray2 is hugeArray)

Note that if you want to re-assign hugeArray without making a copy, you can use slicing. so

hugeArray[:] = hugeNewArray

Instead of

hugeArray = hugeNewArray

answered Aug 8, 2014 at 16:55

ideasman42

49.3k53 gold badges228 silver badges367 bronze badges

1 Comment

ideasman42 Over a year ago

Not sure what you mean, hugeArray[:] = hugeNewArray fully replaces hugeArray's contents, not a shallow copy.

Collectives™ on Stack Overflow

Pass large data structure in Python

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related