1

I am working on program, which sends a bunch of requests to one service asynchronously. Since now i had the part, where there was just request and awaiting for results (ok_function). Now I needed to add part which gets result, and if it is 202 waits for some time and try again since it return 200.

It looks like this:

async def async_make_requests(parsed_data, url):
    async with aiohttp.ClientSession() as session:
        tasks = []
        for x, y in parsed_data.items():

            if something:
                tasks.append(
                    asyncio.ensure_future(
                        function_with_sleep(
                            session, x, y,
                        )
                    )
                )
             else:
                 tasks.append(
                    asyncio.ensure_future(
                        ok_function(
                            session, x, y,
                        )
                    )
                )   
        results = await asyncio.gather(*tasks)
    return results

async def function_with_sleep(session, x, y):
    not_processed_yet = True
    while not_processed_yet:
        async with session.get(x, data=y) as r:
            response = await r.json()
            status_code = r.status
        if status_code == 202:
            await asyncio.sleep(10)
        else:
            not_processed_yet = False
   ...

async def ok_function(session, x, y)
    async with session.post(x, data=y) as r:
        response = await r.json()
        status_code = r.status
    ...

While testing it with:

    resp_processing = MockResponse(202, response)
    resp_ok = MockResponse(200, response)
    mocker.patch("aiohttp.ClientSession.get", side_effect=[resp_processing, resp_ok])
    return_value = await async_make_requests(parsed_data, "anyurl")

I am getting:

  results = await asyncio.gather(*tasks) RuntimeError: coroutine raised StopIteration

ok_function works fine, and function_with_sleep works ok only if it doesn't repeat itself, and doesn't sleep.

I have no idea what is wrong here.

1 Answer 1

2

The code seens to be right - although you don't say which http framework your session belongs too: If r.json raises an error when there is no json body in the request made, this error would be propagated - and if it is StopIteration, it will break asyncIO.gather as described (otherwise, the task exception would be set in the corresponding task).

Try to call await r.json() just after you have a 200 status code, and not before testing the status code. If the behavior you claim is indeed caused by the code you pasted in the question, this should fix it.

Indeed - it seems you are using aiohttp, or something that builds upon it: if a body is not available in the request object, it will call .read in the response - since the response is a mock object this is likely triggering the StopIteration exception. - https://docs.aiohttp.org/en/stable/client_reference.html

Trying to get the JSON body just when you have a meaningful response is the right thing to do, nonetheless, so there is no point in trying to add a correct read response to your mock.

Sign up to request clarification or add additional context in comments.

1 Comment

Thanks for answer. It came out that it wasn't the case - but your answer makes me dig deeper. As you have mentioned the code seems to be right - and it was true. It came out that the problem was with tests - I was testing performing 2 sets of data - So 1st was getting 1st 202, 2nd 200 response. But it came out for 2 set it wasn't resetting and that's why StopIteration was raised. all i needed to do was side_effect=itertools.cycle([resp_processing, resp_ok])

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.