57

I am using the AWS lambda function to convert uploaded WAV files in a bucket to mp3 format and later move the file to another bucket. It is working correctly. But there's a problem with triggering. When I upload small WAV files, the lambda function is called once. But when I upload a large-sized wav file, this function is triggered multiple times.

I have googled this issue and found that it is stateless, so it will be called multiple times(not sure if this trigger is for multiple uploads or the same upload).

https://aws.amazon.com/lambda/faqs/

Is there any method to call this function once for a single upload?

Improve this question
3
  • "not sure this trigger is for multiple upload or a same upload" ... doesn't it seem like that should be the first thing you should work toward figuring out? Statelessness has nothing to do with it. Examine the actual event content, create a log of what you are receiving in the S3 event and store it for review, and the explanation should become obvious. Most likely, behavior in whatever you are using to originally upload the object is doing more S3 operations than you realize. Commented Aug 18, 2015 at 10:38
  • You need to expand this question with more specific information. Ideally a code sample. Commented Aug 19, 2015 at 13:11
  • 3
    Sounds like the upload is being broken into parts. You'll want to create an event specify that the event is only executed on s3:ObjectCreated:CompleteMultiPartUpload, this tutorial shows how to do that just replace s3:ObjectCreated:* with s3:ObjectCreated:CompleteMultiPartUpload. Commented Jan 24, 2016 at 16:16

8 Answers 8

Reset to default
58

Short version: Try increasing timeout setting in your lambda function configuration.

Long version:

I guess you are running into the lambda function being timed out here.

S3 events are asynchronous in nature and lambda function listening to S3 events is retried atleast 3 times before that event is rejected. You mentioned your lambda function is executed only once (with no error) during smaller sized upload upon which you do conversion and re-upload. There is a possibility that the time required for conversion and re-upload from your code is greater than the timeout setting of your lambda function.

Therefore, you might want to try increasing the timeout setting in your lambda function configuration.

By the way, one way to confirm that your lambda function is invoked multiple times is to look into cloudwatch logs for the event id (67fe6073-e19c-11e5-1111-6bqw43hkbea3) occurrence -

START RequestId: 67jh48x4-abcd-11e5-1111-6bqw43hkbea3 Version: $LATEST

This event id represents a specific event for which lambda was invoked and should be same for all lambda executions that are responsible for the same S3 event.

Also, you can look for execution time (Duration) in the following log line that marks end of one lambda execution -

REPORT RequestId: 67jh48x4-abcd-11e5-1111-6bqw43hkbea3  Duration: 244.10 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 20 MB

If not a solution, it will at least give you some room to debug in right direction. Let me know how it goes.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

+1 Not Specific to S3, For me, this is happening for CloudWatch Lambda Triggers as well. Is there an option prevent this other than increasing the timeout.
Mine is fired exactly 3 times, every time. It reads from dynamoDB and stores in another table. Every time 3 different request IDs
i think the answer here shows that its not always just a timeout: stackoverflow.com/questions/57286268/… & aws-labs themselves recently added some open source tools to standardize dealing with this issue of multiple executions: awslabs.github.io/aws-lambda-powertools-python/api/utilities/…
19

Any event Executing Lambda several times is due to retry behavior of Lambda as specified in AWS document.

Your code might raise an exception, time out, or run out of memory. The runtime executing your code might encounter an error and stop. You might run out concurrency and be throttled.

There could be some error in Lambda which makes the client or service invoking the Lambda function to retry.

Use CloudWatch logs to find the error and resolving it could resolve the problem.

I too faced the same problem, in my case it's because of application error, resolving it helped me.

Recently AWS Lambda has new property to change the default Retry nature. Set the Retry attempts to 0 (default 2) under Asynchronous invocation settings.

Improve this answer

3 Comments

In my case there is absolutely no error and my timeout is 10 minutes which is way above what I need. Do you think the use of finally might cause this behavior?
Resolving application errors is generally considered good practice.
My problem was a special instance of this scenario: I was returning results from the handler (so the very last thing I do in my function) that was not serializable causing an exception, hence the event's need to retry. CloudWatch logs is your friend here.
5

For some in-depth understanding on this issue, you should look into message delivery guarantees. Then you can implement a solution using the idempotent consumers pattern.

The context object contains information on which request ID you are currently handling. This ID won't change even if the same event fires multiple times. You could save this ID for every time an event triggers and then check that the ID hasn't already been processed before processing a message.

Improve this answer

4 Comments

thanks @maxpaj, Could you please let us know if there is any similar option for python handler. I am doing a requests .post from lamda function ,to my application endpoint .and if it takes more than 3 seconds to respond with the status code. it is retyring it , adding extract function triggers.
I had some issue with this implementation. I have a cloud watch event invoking a lambda function one time and noticed it would run the function multiple times randomly but each one had a different requestId associated to it.
can u check if your function was timing out or not? @skrusetvt
@neo73 I checked out my CloudWatch logs and there was no error thrown. I did however increase the timeout from 15 seconds to 3 minutes and I have not seen this behavior anymore.
3

Multiple retry can also happen due read time out. I fixed with '--cli-read-timeout 0'.

e.g. If you are invoking lambda with aws cli or jenkins execute shell:

aws lambda invoke --cli-read-timeout 0 --invocation-type RequestResponse \
   --function-name ${functionName} --region ${region} --log-type Tail \
   --payload {""} out --log-type Tail \
   --query 'LogResult' --output text | base64 -d
Improve this answer

Comments

1

In the Lambda Configuration look for "Asynchronous invocation" there is an option "Retry attempts" that is the maximum number of times to retry when the function returns an error.

Here you can also configure Dead-letter queue service

Improve this answer

1 Comment

I tried this and set to 0, but still my lambda retries.
1

Faced a similar problem when i invoked the lambda using an async lambda client. Verifying the following points helped me:

  • Make sure the lambda is not erroring out.(response of your lambda should be in 200 series)
  • under Asynchronous invocation there is an configuration Retry attempts you can try setting that to 0.
  • if you are using a Async Lambda Client make sure the --invocation-type configured in your client is Event. (this point helped me solve my problem)
Improve this answer

Comments

0

I was also facing this issue earlier, try to keep retry count to 0 under 'Asynchronous Invocations'.

Improve this answer

Comments

0

The multiple invocations for my case is related to the default sdk timeout behavior.

I am using python boto3 and the default connect_timeout and read_timeout are both 60s. Upon timeout, the default behavior is to retry up to 4 times every 60s. Hence, even when I increase the timeout configuration in lambda, I still keep seeing retries.

I solve this by adding additional config in the boto3.client declaration.

import boto3
from botocore.client import Config

LAMBDA_TIMEOUT_S = 300
session = boto3.Session()
config = Config(
    connect_timeout=LAMBDA_TIMEOUT_S, 
    read_timeout=LAMBDA_TIMEOUT_S, 
    retries={'max_attempts': 0}
    )
lambda_client = session.client("lambda", config=config)

reference: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.