Python: Does the interpreter optimize statements like any([implicit for loop])?

Question

I was wondering what type of optimizations are done automatically by the python interpreter. E.g. I have the following code to check whether a string starts with any substrings in startList:

def startswithAny(string, startList):
    return any([string.startswith(x) for x in startList])

Let's assume startList contains 1000 entries, but string.startswith(startList[0]) already yields true. What will happen? Will [string.startswith(x) for x in startList] still be fully evaluated? Or will the interpreter recognize that any() yields true already?

If not, than IMHO it makes more sense to write codes like these non-pythonic, like this:

def startswithAny(string, startList):
    for x in startList:
        if string.startswith(x): 
            return True
    return False

Thanks

Serge Ballesta · Accepted Answer · 2020-03-23 10:37:03Z

1

Well, any will honour short circuit. The problem is that as you use a list, you first compute the full list, and only then call any on the full list.

If you really want to use the short circuit, you should use a generator instead of a list:

def startswithAny(string, startList):
    return any((string.startswith(x) for x in startList))

(square brackets replaced with parentheses)

Demo using timeit:

>>> timeit.timeit('any([i.startswith("ab") for i in lst])',
                        'lst = [ "abc" + str(i) for i in range(10000)]', number=10000)
17.005268767956352
>>> timeit.timeit('any((i.startswith("ab") for i in lst))',
                        'lst = [ "abc" + str(i) for i in range(10000)]', number=10000)
0.006302962841857607

answered Mar 23, 2020 at 10:37

Serge Ballesta

150k13 gold badges137 silver badges267 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

tim Over a year ago

Oh yes, I must have missed that. Thanks for pointing it out. Can you think of any usage of any([list comprehension]) which cannot be replaced by a generator expression? Or you think it's safe to say that this combination can always be replaced by a generator expression like in your example?

Joseph Chotard · Accepted Answer · 2020-03-23 10:23:40Z

1

Yes the any function is guaranteed to short circuit that means that the moment it finds an element that's True it terminates. The any() code is essentially equivalent to the following:

def any(iterable):
    for element in iterable:
        if element:
            return True
    return False

As you can see the moment an element is True, the loop is stopped.

So yes the code you wrote at the end is essentially equivalent to running any() on it. any() is the more pythonic way.

answered Mar 23, 2020 at 10:23

Joseph Chotard

6855 silver badges15 bronze badges

2 Comments

tim Over a year ago

Thanks for the reply - I guess I wasn't specific enough w.r.t my question. What I meant was: Will my implicit for-loop [string.startswith(x) for x in startList] still be evaluated fully before it is passed to any as a fully evaluated list (yes, any() obviously will short-circuit)? Or can any() already interfere into the for loop so that it's not necessary to evaluate it completely?

Joseph Chotard Over a year ago

Oh, I'm sorry. jizhilong and Serge Ballesta have good answers to that question

jizhilong · Accepted Answer · 2020-03-23 10:39:50Z

1

According to the document, any guarantees short circuit.

But your usage will break that optimization, as a list comprehension expression will get each element evaluated, so you should pass a generator expression instead, which means the more optimized way is any(string.startswith(x) for x in startList)

answered Mar 23, 2020 at 10:39

jizhilong

1245 bronze badges

Collectives™ on Stack Overflow

Python: Does the interpreter optimize statements like any([implicit for loop])?

3 Answers 3

1 Comment

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related