Asynchronous API calls in loop python

Question

I am trying to store a list of objects using Tweepy's statuses_lookup api call. Each call to statuses_lookup takes in a list of IDs, and can contain up to 100 IDs.

This function below takes in a list of IDs, and I am trying to append all the metadata returned from the API call into the tweetData list.

def lookupTweets(self, tweetIds):
    tweetData = []
    i = 0
    while i < len(tweetIds):
        print(i)
        if len(tweetIds) - i > 0:
            statuses = self.status_lookup(tweetIds[i + 99])
        else:
            statuses = self.status_lookup(tweetIds[i, len(tweetIds) - i])

        tweetData.append(statuses)
        i += 100

    return tweetData

And here is the async function that makes the api call

async def status_lookup(self, tweets):
        return self.api.statuses_lookup(tweets)

And here is the main method:

if __name__ == "__main__":
    twitterEngine = TwitterEngine()
    tweets = twitterEngine.ingestData("democratic-candidate-timelines.txt")

    twitterData = twitterEngine.lookupTweets(tweets)
    loop = asyncio.get_event_loop()
    loop.run_until_complete(asyncio.wait(twitterData))
    print(twitterData)

When I print the result of twitterData, I get a list of coroutine objects. The output looks something like this: [<coroutine object TwitterEngine.status_lookup at 0x105bd16d0>]. However, I want the actual metadata and not the coroutine object.

I'm new to async programming in Python, and any guidance would be greatly appreciated!

Does tweepy library support async

Kay
– Kay

2019-04-28 09:13:17 +00:00
Commented Apr 28, 2019 at 9:13 — Kay
– Kay, Commented Apr 28, 2019 at 9:13

user4815162342 · Accepted Answer · 2018-04-21 15:42:53Z

3

When I print the result of twitterData, I get a list of coroutine objects.

Calling a coroutine function will just create the coroutine object, much like invoking a generator just creates the generator object. To get actual data from a coroutine object, you need to either await it from another coroutine, or run it in the event loop. In case of status_lookup, lookupTweets should itself be an async def coroutine, and it should await the statuses:

statuses = await self.status_lookup(tweetIds[i + 99])

The same goes for status_lookup:

async def status_lookup(self, tweets):
    return await self.api.statuses_lookup(tweets)

The return value of the outer-most coroutine will be returned by run_until_complete:

loop = asyncio.get_event_loop()
twitterData = loop.run_until_complete(twitterEngine.lookupTweets(tweets))    
print(twitterData)

edited Apr 21, 2018 at 15:42

answered Apr 21, 2018 at 14:01

user4815162342

159k22 gold badges349 silver badges418 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Varun Vu Over a year ago

When I set status_lookup to be async, I get the following error: TypeError: object ResultSet can't be used in 'await' expression

user4815162342 Over a year ago

@VarunVu It seems that status_lookup is not a coroutine. Is tweepy asyncio aware at all?

Michal Charemza · Accepted Answer · 2018-04-21 13:45:04Z

1

Coroutine objects, the result of calling async def functions, need to be associated with futures to get access to the returned value.

There are several ways of doing this, but if you have a list of coroutine objects, you can use asyncio.gather:

twitterData = twitterEngine.lookupTweets(tweets)
future = asyncio.gather(*twitterData)
loop = asyncio.get_event_loop()
loop.run_until_complete(future)
print(future.result())

answered Apr 21, 2018 at 13:45

Michal Charemza

27.3k15 gold badges114 silver badges191 bronze badges

6 Comments

Varun Vu Over a year ago

This looks promising, but the result from printing future.result() is list of empty lists: [[], [], [], [], [], [], [], [], [], []]

Michal Charemza Over a year ago

@VarunVu If you add print(tweetData) just before the return in lookupTweets, does it show non-empty lists?

Varun Vu Over a year ago

It shows a list of coroutine objects, like so:

[<coroutine object TwitterEngine.status_lookup at 0x105bda360>, <coroutine object TwitterEngine.status_lookup at 0x105bdaa98>...]

Michal Charemza Over a year ago

@VarunVu Ah ok... and just to confirm, self.api.statuses_lookup(tweets) does return something that isn't empty?

Varun Vu Over a year ago

Ah I made a dumb mistake with my slicing, I put a comma instead of a semicolon! Thanks!!

|

Collectives™ on Stack Overflow

Asynchronous API calls in loop python

2 Answers 2

2 Comments

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related