0

I have created an AWS lambda trigger from S3 - create object notification. When lambda gets object creation notification it also launches the ec2 instance. For the next step, I want ec2 to access the object/data form s3 and do some processing on that data. How can I access that data from ec2?

Improve this question

2 Answers 2

Reset to default
2

When the Lambda function launches the EC2 instance, I would have it pass the S3 object key as part of the EC2 user-data.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

thank you for your response. But the point is the user-data script will run only when ec2 lunches. I need to copy every data uploaded to the s3 bucket. Are there any way of doing that?
Your question indicated you are launching a new EC2 instance each time an new object is uploaded to S3. Is that not the case?
If the instance is already up and running I don't want to lunch a new one. I just want already run EC2 to continue processing with newly uploaded objects.
Then it sounds like you should just have S3 push notifications to SQS, and have a pool of auto-scaled EC2 instance that read from SQS. You don't need AWS Lambda at all for that.
2

It appears that you are wanting to use an existing Amazon EC2 instance to process objects that are uploaded to an Amazon S3 bucket.

The standard method for doing this is:

  • Create an Amazon SQS queue
  • Configure the Amazon S3 bucket to send a message to the SQS queue when a new object is created
  • Write code on the Amazon EC2 instance that continuously polls the Amazon SQS queue, asking for a message (use WaitTimeSeconds=20 to reduce the number of calls, it will return immediately if a message is available)
  • Once a message is received, the code can download the S3 object (the Bucket and Key are provided in the message) and then process the file. Once the file is processed, your code should delete the SQS message

If there are long time-periods between object uploads and you wish to save money, you could add some logic to your code that does a Shutdown of the instance when no messages have been received for a given period of time. You could then create an Amazon CloudWatch alarm based on queue size that can trigger an AWS Lambda function to Start the instance if it is not currently Running.

To run a script every time that an Amazon EC2 instance starts, you can place it in /var/lib/cloud/scripts/per-boot/.

See also: Auto-Stop EC2 instances when they finish a task - DEV Community

Improve this answer

1 Comment

Big Thanks John it works for our case

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.