1

I want to run a py script in my ec2 instance automatically whenever a new file is added to an s3 bucket. As of now, I have connected my bucket to a lambda function which is triggered with event notification whenever a file is added to the bucket. How do i further use this to trigger my ec2 instance to start executing the python file? I want to send that file from the bucket to the instance and the python program will use that file as input. I had read that i can trigger the lambda function when object is added. Now how do i use it or SQS to start my instance automatically and start running the python script inside it after the file has been sent from bucket to the instance?

Improve this question
2
  • Is the EC2 instance always running? Commented Feb 25, 2021 at 12:40
  • @mhawke It doesnt have to be, but if it makes it easier, then yeah, it can be running all the time. Ideally, i would want it to start running when that the bucket gets a new file Commented Feb 26, 2021 at 4:01

3 Answers 3

Reset to default
2

There are several options:

  • Configure Amazon S3 to trigger an AWS Lambda function, and then do all of the processing inside the Lambda function, or
  • Have the EC2 instance running continuously. Configure S3 to push a message into an Amazon SQS queue. An app on the EC2 instance should continually long-poll the SQS queue to check if a file has been received. If so, the SQS message will contain the file details. or
  • Configure S3 to trigger an AWS Lambda function that then starts ("wakes up") an EC2 instance if it has been stopped. The Lambda function should push a message onto an SQS queue so that it is accessible to the instance (especially if multiple files have been created). Put a script on the EC2 instance in /var/lib/cloud/scripts/per-boot/, which will be run automatically when the instance starts. The script should trigger the app, the same as the 2nd option above, which reads from the SQS queue. Once it has emptied the queue, the script could stop the instance to minimize costs. For more details, see: Auto-Stop EC2 instances when they finish a task - DEV Community
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

uploading file to s3 bucket in an event which can trigger lambda function, as you wrote. The lambda may be python code itself - in this case there is no need to run it on ec2 instance. However you have time limit (15 mins) for execution. If this is not sufficient and the ec2 instance is normally stopped, you could start your ec2 instance using boto3 (you could use lambda code described here to do this and have the instance configured to run the python script on start (for example @reboot entry in /etc/crontab)

Improve this answer

Comments

0

S3 can generate an event, when a new object is created https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html

One type of event destination may be an SNS Topic https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html

One type of SNS subscription type is an HTTP/S POST https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html

So if you script can implement a simple HTTP listener/api, it will be triggered by SNS nearly instantly.

Improve this answer

3 Comments

I had read that the lambda function which gets triggered when object is added to S3 can then ask SQS to trigger my instance or script. How do I do that?
How do i pass the path details and video file details from the bucket to the sqs queue and use the SQS to trigger my instance and script? I want those details to be passed to the instance so that it can transfer the file from the s3 to the ec2 instance
@akshayacharya you don't need a lambda function when the SNS does exactly the same - as I said above - it supports https endpoints, so it can trigger your code directly. All the details are in the json in the message, so you don't have to worry about it either.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.