Using an anonymous function in Python

Question

I've got some code which downloads a list of data from numerous URLs, then calls another function, passing in each result. Something like...

def ShowUrls(self, url):
    Urls = self.Scraper.GetSubUrls(url)
    for Url in Urls:
        self.UI.addLink(
          Url[0],
          Url[1])

This works fine but there's a long delay while self.Scraper.GetSubUrls runs, then all the UI calls are made very rapidly. This causes the UI to show "0 Urls added" for a long time, then complete.

What I'd like is to be able to pass the self.UI.addlink method in to the self.Scraper.GetSubUrls method so that it can be called as soon as each URL is retrieved. This should make the UI show the correct count as soon as each url is retrieved.

Is this possible? If so, what's the correct syntax?

If I were in Javascript, I'd do something like....

getSubUrls(url, function(x, y) {UI.addLink(x, y)})

and then, inside getSubUrls do

SomeParamMethod(Pram1, Param2)

Is this possible? If so, what's the correct syntax?

Ry- · Accepted Answer · 2012-11-18 00:55:53Z

7

You can use lambda, but it's usually a better idea to make a separate function and pass it.

self.Scraper.GetSubUrls(url, lambda url: self.UI.addLink(url[0], url[1]))

or

def addLink(url):
    self.UI.addLink(url[0], url[1])

self.Scraper.GetSubUrls(url, addLink)

answered Nov 18, 2012 at 0:55

Ry-♦

226k56 gold badges496 silver badges504 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Basic Over a year ago

Thanks for the quick response. Can you clarify slightly - what's the preferred syntax and why?

Ry- Over a year ago

@Basic: Sorry, updated with a link to the documentation and the better version. (Note that you nest the definition inside the other one.) It's preferred because it looks better and usually makes for cleaner code.

Basic Over a year ago

Thanks very much, that seems to be be exactly what I was after. I'll accept when I can

ocket8888 Over a year ago

Just gonna throw in my two cents: lamda is generally best used in a place where a very simple functionality needs to happen just once. For instance, if you us it in something that loops like e.g. a list comprehension, Python will need to re-construct the function on every iteration, whereas if you pass a pre-defined function, it merely looks up the name.

Community · Accepted Answer · 2017-05-23 11:50:26Z

6

This suggestion is a bit more involved, but if you control GetSubUrls, a more Pythonic approach might be to make it a generator that yields each URL as it is retrieved. You can then process each URL outside the function in a for loop. For instance, I'm assuming GetSubUrls presumably looks something vaguely like this:

def GetSubUrls(self, url):
    urls = []
    document = openUrl(url)
    for stuff in document:
        urls.append(stuff)
    return urls

That is, it builds a list of URLs and returns the whole list. You can make it a generator:

def GetSubUrls(self, url):
    document = openUrl(url)
    for stuff in document:
        yield stuff

Then you can just do

for url in self.Scraper.GetSubUrls(url):
    self.UI.addlink(url[0], url[1])

Which is the same as before, but if GetSubUrls is a generator, it doesn't wait to collect all the suburls and then return them. It just yields one at a time, and your code can likewise process them one at a time.

One advantage of this over passing a callback is that you can store the generator and use it whenever you want, instead of having the calls made inside GetSubUrls. That is, you can do urls = GetSubUrls(url), save that for later, and still iterate over the URLs "on demand" at a later time, when they will be retrieved one by one. Using a callback approach forces the GetSubUrls function to process all the URLs right away. Another advantage is that you needn't create a bunch of small callbacks with little content; instead you can write these one-liners naturally as the body of the for loop.

Read up on Python generators for more info on this (for instance What does the "yield" keyword do in Python? ).

edited May 23, 2017 at 11:50

CommunityBot

11 silver badge

answered Nov 18, 2012 at 1:04

BrenBarn

253k39 gold badges421 silver badges392 bronze badges

1 Comment

Basic Over a year ago

Yep, that's a good assumption re: GetSubUrls() and I like this approach - reminds me of Yielding in other languages. I've implemented minitech's answer so have awarded to him but +1 as I think I'm mgoing to come back and try to get your approach working as soon as I get the chance. Thanks

Collectives™ on Stack Overflow

Using an anonymous function in Python

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related