Using nested query in AWS log insights

Question

I am using AWS Lambdas to host some services. To mitigate issue of cold start, I'm using serverless plugin warmup.

Now using cloudwatch log insights, I intend to figure out p99/p95 of all the requests going to my lambda but exclude requests made by the plugin. For this one query which should help is , figure out requestIDs for plugin calls ( I'm logging a certain message based on which I can filter out those requests ). Now I want all those requests whose request IDS are not in these requests IDs. In sql this would transform to a nested query.

Rough version :

SELECT @duration from requestIDs where requestID not in ( SELECT RequestID where @message like ***** )

Do we have a way in which I can do such query in Log Insights?

Alex P · Accepted Answer · 2022-06-07 14:53:40Z

0

Yes, you can remove messages from the warmup plugin by piping all messages to the filter with not like condition and then to pct(fieldName: LogFieldValue, percent: number) function:

fields @duration
| filter @message not like "MESSAGE TO EXCLUDE"
| stats pct(@duration, 99)

edited Jun 7, 2022 at 14:53

answered Jun 7, 2022 at 13:50

Alex P

2851 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Kancha Over a year ago

Yeah that I get. But the time durations like @initDuration is part of another message and not part of console.log message.

Alex P Over a year ago

Got it, let me know if the corrected version of the answer solves your problem.

Kancha Over a year ago

Hi, actually no it will not work. For a single request my log trail looks like : START requestID1 xxxx INFO requestID1 CustomMessage END requestID1 xxxxx REPORT requestID1 xxxxx And then another log trail let's say : START requestID2 xxxx INFO requestID2 randomMessage END requestID2 xxxxx REPORT requestID2 xxxxx Your answer will still give me durations from reports of requestID1 which have this CustomMessage in @message.

Alex P Over a year ago

Why is this message random?

Collectives™ on Stack Overflow

Using nested query in AWS log insights

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related