AWS Lambda function getting called repeatedly

Question

I have written a Lambda function which gets invoked automatically when a file comes into my S3 bucket. I perform certain validations on this file, modify the particular and put the file at the same location. Due to this "put", my lambda is called again and the process goes on till my lambda execution times out. Is there any way to trigger this lambda only once?

I found an approach where I can store the file name in DynamoDB and can apply a check in lambda function, but can there be any other approach where DynamoDB's use can be avoided?

cementblocks · Accepted Answer · 2018-09-24 15:32:53Z

3

You have a couple options:

You can put the file to a different location in s3 and delete the original
You can add a metadata field to the s3 object when you update it. Then check for the presence of that field in s3 so you know if you have processed it already. Now this might not work perfectly since s3 does not always provide the most recent data on reads after updates.

edited Sep 24, 2018 at 15:32

answered Sep 24, 2018 at 14:15

cementblocks

4,64520 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Marcin · Accepted Answer · 2018-09-24 14:18:26Z

1

AWS allows different type of s3 event triggers. You can try playing s3:ObjectCreated:Put vs s3:ObjectCreated:Post.

answered Sep 24, 2018 at 14:18

Marcin

2512 gold badges6 silver badges13 bronze badges

Comments

Manoj Acharya · Accepted Answer · 2018-09-24 15:50:34Z

1

You can upload your files in a folder, say

s3://bucket-name/notvalidated

and store the validated in another folder, say

s3://bucket-name/validated.

Update your S3 Event notification to invoke your lambda function whenever there is a ObjectCreate(All) event in the /notvalidated prefix.

answered Sep 24, 2018 at 15:50

Manoj Acharya

1,4992 gold badges17 silver badges28 bronze badges

Comments

Frank · Accepted Answer · 2018-09-24 15:01:29Z

0

The second answer does not seem to be correct (put vs post) - there is not really a concept of update in S3 in terms of POST or PUT. The request to update an object will be the same as the initial POST of the object. See here for details on the available S3 events.

I had this exact problem last year - I was doing an image resize on PUT and every time a file was overwritten, it would be triggered again. My recommended solution would be to have two folders in your s3 bucket - one for the original file and one for the finalized file. You could then create the lambda trigger with the lambda prefix so it only checks the files in the original folder

edited Sep 24, 2018 at 15:01

answered Sep 24, 2018 at 14:56

Frank

4195 silver badges17 bronze badges

Comments

vikash singh · Accepted Answer · 2018-09-26 10:43:09Z

0

The events are triggered in S3 based on if the object is put/post/copy/complete Multipart Upload - All these operations corresponds to ObjectCreate as per AWS documentation .

https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html

The best solution is to restrict your S3 object create event to particular bucket location. So that any change in that bucket location will trigger lambda function.

You can do the modification in some other bucket location which is not configured to trigger lambda function when object is created in that location.

Hope it helps!

answered Sep 26, 2018 at 10:43

vikash singh

1,5091 gold badge20 silver badges30 bronze badges

Collectives™ on Stack Overflow

AWS Lambda function getting called repeatedly

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related