Why does synchronous lambda invoke multiple times?

Question

I have a lambda function my-func which should only be run once at a time. Because of that, I set the Reserved Concurrency to 1. I am trying to invoke it with the command:

aws lambda invoke --function-name my-func --invocation-type RequestResponse --cli-binary-format raw-in-base64-out --payload '{\"recreate\":true}' response.json

However, it results in this error after a few seconds:

An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 2): Rate Exceeded.

It appears that it tries to invoke the function multiple times even though the original invocation never ran into an error. If I increase the Reserved Concurrency to a value like 5, then the single lambda invoke command results in multiple invocations even though the first invocation continues to execute without any problem.

Another thing that is throwing me off is that it works correctly from the AWS console GUI. I created a test event in the AWS lambda function on the console interface. It invokes my-func with the same payload I am using in the aws-cli command:

{
  "recreate": true
}

Invoking the function using this test event works flawlessly. It seems to just run the function once and doesn't cause a TooManyRequestsException. Does this mean something is wrong with my aws-cli command?

What is your function code? Maybe you try to run it recursively? — Marcin
– Marcin, Commented Feb 1, 2022 at 22:46
@Marcin I thought about that at first, but the function doesn't call itself anywhere. In fact it doesn't perform any lambda operations. It only configures some NAT gateway stuff using ec2 client in boto3 — porcelainsnake
– porcelainsnake, Commented Feb 1, 2022 at 22:51
It does not have to be explicit invocation. Maybe the function triggers something non-directly (e.g. store object in S3 and S3 event triggers the same function). — Marcin
– Marcin, Commented Feb 1, 2022 at 22:57
@Marcin I just did a test: I removed all code from the function and replaced it with time.sleep(60). Again, the function works correctly when I invoke it from the AWS console but gets TooManyRequestsException when I invoke from the aws-cli. Could it be something with the way I'm invoking it? — porcelainsnake
– porcelainsnake, Commented Feb 1, 2022 at 23:31

porcelainsnake · Accepted Answer · 2022-02-02 01:18:37Z

6

I've figured out the problem. Even though my function's timeout was set to 600 (5 minutes), the aws cli has it's own socket timeout which defaults to 60 seconds. Each time it reached this timeout, it must have triggered a retry. I fixed it by adding --cli-read-timeout 600 to my command like so:

aws --cli-read-timeout 600 lambda invoke --function-name my-func --invocation-type RequestResponse --cli-binary-format raw-in-base64-out --payload '{\"recreate\":true}' response.json

answered Feb 2, 2022 at 1:18

porcelainsnake

1539 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

jellycsc Over a year ago

Why would read take more than 60 seconds? Does the function have a large output and you have a slow connection?

Martin J Over a year ago

@jellycsc He mentioned that the lambda modifies NAT gateway resources. Creating NAT gateways is a slow operation that can easily take 60+ seconds. It is simply a very slow operation ion the AWS side. So this has nothing to do with his connection, it is working as expected.

Kuba Over a year ago

Thank You for that! when doing and invoke through e.g. python boto library the same aplies, the timeout can be changed in the config: botocore.amazonaws.com/v1/documentation/api/latest/reference/…

Alan Hay Over a year ago

Good find. I am seeing the same thing with lambda execution > 60 seconds.

Collectives™ on Stack Overflow

Why does synchronous lambda invoke multiple times?

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related