Is python deepcopy more efficient than custom code, or less efficient/predictable?

Question

I have a dict where the values are non-nested lists (specifically, the keys are ints and the values are lists of ints). I'd like to make a deep copy of it so that I don't modify the lists in the original dict.

I know I can use

copied = copy.deepcopy(original)

However since I know the form of the data structure I can also use something like

copied = {key:valuelist[:] for (key,valuelist) in original.iteritems()}

Is one of these solutions better? More efficient? Less likely to lead to nasty surprises?

I have been told that deepcopy() comes with some gotchas but I don't really understand what. I would also like to understand whether using deepcopy() is less efficient than my code (maybe because it's a more general solution?) or more efficient (maybe it's optimised at a lower level?).

Have you actually tried both solution to see which one is faster? — Maciej Gol
– Maciej Gol, Commented Dec 25, 2013 at 12:40
To be honest, I'm not even sure of good ways to time python code. I know about the time command in ipython. Using that makes me think deepcopy is a lot slower for my specific case (IF I'm testing it right) - but does that mean it's slower for similar cases? An answer based on understanding why it ought to be slower would still be valuable to me. I'd also really like to know if there are reasons other than efficiency to avoid using it. — user2428107
– user2428107, Commented Dec 25, 2013 at 12:45

Maciej Gol · Accepted Answer · 2013-12-25 13:21:47Z

7

As you might have expected, copy.deepcopy is way slower than your second solution:

$ python -m timeit "original = {x: range(10) for x in xrange(10)}; copy = {x: v[:] for x,v in original.iteritems()}"
100000 loops, best of 3: 5.41 usec per loop

$ python -m timeit "original = {x: range(1000) for x in xrange(1000)}; copy = {x: v[:] for x,v in original.iteritems()}"
100 loops, best of 3: 17.1 msec per loop

$ python -m timeit "import copy; original = {x: range(10) for x in xrange(10)}; c = copy.deepcopy(original)"
10000 loops, best of 3: 86.4 usec per loop

$ python -m timeit "import copy; original = {x: range(1000) for x in xrange(1000)}; c = copy.deepcopy(original)"
10 loops, best of 3: 1.4 sec per loop

The reasons why deepcopy is so much slower than the dict comprehension + list copy are:

deepcopy is multi-purpose function - it works for mostly any kind of object
deepcopy is implemented in python whilst dict comprehension and list slicing is done at lower level

And most imporantly

deepcopy makes copies of the elements inside containers recursively, whilst your dict comprehension does not.

Example:

>>> import copy
>>> obj = object()
>>> original = {x: [obj] * 10 for x in xrange(10)}
>>> copy1 = {x:v[:] for x,v in original.iteritems()}
>>> copy2 = copy.deepcopy(original)
>>> copy1[0][0] is original[0][0]
True
>>> copy2[0][0] is original[0][0]
False

As you can see, deepcopy copied the obj contained in the original so that copy2 lists contain it's copy, not the obj itself. Unlike your dict comprehension, which preserves elements in the lists whilst created new list objects.

answered Dec 25, 2013 at 13:21

Maciej Gol

15.9k4 gold badges35 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user2428107 Over a year ago

Thanks, this is really helpful. In this particular case I know that the lists will not contain anything other than immutable types, but it's good to know what's going on under the hood for the future. I'd still like to know if there are any non-efficiency-related reasons to avoid deepcopy. Are there cases where its behaviour could be surprising? Possibly the reason no-one's commented on this is that there aren't...

user2428107 Over a year ago

And.. is there any way to delete my 'extra' comment above? Stackoverflow apparently won't let me delete or edit because I left it for more than 5 minutes. It's just clutter though.

Ever · Accepted Answer · 2013-12-25 13:07:46Z

2

I used below code, and get some result

import copy
import time

def go(loop):
    original = {x: [y for y in range(x)] for x in xrange(100)}
    print loop

    start = time.time()
    for x in xrange(loop):
        copied = copy.deepcopy(original)
    print 'deepcopy %ss' % (time.time() - start)

    start = time.time()
    for x in xrange(loop):
        copied = {k: v[:] for (k,v) in original.iteritems()}
    print 'custome  %ss' % (time.time() - start)

    print ''

for x in (100, 1000, 10000):
    go(x)


result

100
deepcopy 0.47200012207s
custome  0.00699996948242s

1000
deepcopy 4.69200015068s
custome  0.0620000362396s

10000
deepcopy 47.7449998856s
custome  0.677999973297s

obviously, copy.deepcopy is much worse. I think it handle much more than the custom method

edited Dec 25, 2013 at 13:07

answered Dec 25, 2013 at 13:02

Ever

1,1841 gold badge9 silver badges13 bronze badges

Comments

gitaarik · Accepted Answer · 2013-12-25 12:47:52Z

0

My advice is to use deepcopy for now, if it turns out to be too slow you can always replace it with a custom function.

It might get slow if your dict is really really big, but as long as you don't work with really big data I won't worry too much.

answered Dec 25, 2013 at 12:47

gitaarik

47k12 gold badges101 silver badges109 bronze badges

Collectives™ on Stack Overflow

Is python deepcopy more efficient than custom code, or less efficient/predictable?

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related