How to wait to trigger Lambda function until specific files are uploaded

Question

I'm new to AWS and working on Lambda function triggered by S3. I have two json files and need to use the both data to send queue to SQS. For example, two files, "file_a.json" and "file_b.json" are to be uploaded to S3 at the same time.

file_a.json
{"data": "123"}

file_b.json
{"data": "456"}

Then, Lambda will send queue message like ["123", "456"] which is created with the data from both files.

However, S3 invokes Lambda function every time the single file is uploaded and this makes it hard to send queue to SQS as I wrote above. I'm thinking if S3 can wait until the two files are uploaded to trigger Lambda, it can solve the issue. If anyone had faced the similar situation, I would like to know how to resolve this problem. Thanks!

@MisterSmith has answered correctly so I won't post the same answer again, but you may consider uploading the 2 files at once in an archive (zip) and triggering your lambda on that. — Ninad Gaikwad
– Ninad Gaikwad, Commented Jan 8, 2022 at 5:37

MisterSmith · Accepted Answer · 2022-01-08 01:50:27Z

3

S3 notifications are simple events in response to changes in S3, so you cant "make s3 wait" for both files, its upto you to handle that in your lambda.

You could store the message (or whatever data you need) - s3, rds, paramstore etc, or if you can deduce the other filename from the message you can query S3 and check if the other file exists yet and act accordingly.

answered Jan 8, 2022 at 1:50

MisterSmith

3,7041 gold badge12 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

shinyatk Over a year ago

Thank you for answering my question. Good to know that we cannot do it with S3 features only. I will try the approach as you advised. Thank you.

Collectives™ on Stack Overflow

How to wait to trigger Lambda function until specific files are uploaded

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related