3

I have a sync job (in Node.js) which has to process several hundred of documents in one batch. For each of them also perform several tasks. As usually, after deployment, such job will become a blackbox: without propper logging it is impossible to find a problem.

Therefore, I log any reasonable information - which document job is being processed, what task is performing now etc. I use console.log / console.error for logging. This results in a very large log file, which is not that big problem when running localy.

Once deployed on AWS, is there any best practice / limitation for logging? I am afraid of costs also.

Thanks!

Improve this question

2 Answers 2

Reset to default
2

There are many ways to approach this.

If your main need is visualizing your data workflow, using AWS Step Functions would do it. They recently launched a cheap version called Express Workflows.

Break down into multiple Lambdas, one for each task. Step Functions will take care of all the orchestration, error handling, retrying, etc. Using AWS SQS may also be beneficial to smooth out the batching process.

For full visibility, a specialized tool will be required. In serverless, we don't have control over the infrastructure, so a different approach is required. I'd recommend to check out Dashbird.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

I don't think there is much limitation, for pricing you can find it here : https://aws.amazon.com/cloudwatch/pricing/

Also you can use loggers like winston to control logging level in prod, say debug in dev and info/warn in prod.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.