Disable logging on AWS Lambda Python 3.9

Question

I have some code that is using logger package to log to a file on a Lambda function. The problem is that it also sends everything to CloudWatch, and since there are a lot of logs, is very expensive.

When I was using Python 3.7, this was working and logging only to the file:

import os
import sys
import logging

LOG_LEVEL = os.getenv('LOG_LEVEL', 'INFO')

root_logger = logging.getLogger()
root_logger.disabled = True

# create custom logger
logger = logging.getLogger('my_logger')
logger.removeHandler(sys.stdout)
logger.setLevel(logging.getLevelName(LOG_LEVEL))

for handler in logger.handlers:
    logger.removeHandler(handler)

handler = logging.FileHandler('file.log', encoding='utf-8')
handler.setLevel(logging.getLevelName(LOG_LEVEL))
logger.addHandler(handler)

sys.stdout = open(os.devnull, 'w')
run_code_that_logs_stuff()

But after upgrading to Python 3.9, the logs started to show in CloudWatch again.

I have tried changing the last 2 lines with:

with open(os.devnull, 'w') as f, contextlib.redirect_stdout(f):
    run_code_that_logs_stuff()

but same result:

START RequestId: 6ede89b6-26b0-4fac-872a-48ecf64c41d1 Version: $LATEST


2022-05-29T05:46:50.949+02:00   [INFO] 2022-05-29T03:46:50.949Z 6ede89b6-26b0-4fac-872a-48ecf64c41d1 Printing stuff I don't want to appear

@ForgottenUmbrella I'm sorry I don't really know what that means, but I am not talking about errors in particular even info/debug — Guillaume
– Guillaume, Commented May 27, 2022 at 14:54
Sorry I wasn't clear. I don't know the specifics, but I was wondering whether the logging is perhaps being output to stderr instead of stdout, in which case you would have to use contextlib.redirect_stderr instead of contextlib.redirect_stdout to suppress output. — ForgottenUmbrella
– ForgottenUmbrella, Commented May 28, 2022 at 10:18
@ForgottenUmbrella yeah could have, but no, doesn't change a thing — Guillaume
– Guillaume, Commented May 30, 2022 at 6:11

Parsifal · Accepted Answer · 2022-05-30 11:22:20Z

2

The easiest way to prevent Lambda from writing to CloudWatch Logs is to remove the IAM policy that allows it to do so.

If you look at your Lambda's configuration, you'll see an execution role. If you look at that role's configuration, you'll see that it references the managed policy AWSLambdaBasicExecutionRole (or, if it's running inside a VPC, AWSLambdaVPCAccessExecutionRole). If you look at those managed policies, you'll see that they grant permissions logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents.

Replace those managed policies with your own managed policies that grant everything except those three permissions.

Or, alternatively, create a managed policy that explicitly denies those three permissions (and nothing else), and attach it to the Lambda's execution role.

Update:

I noted in the comments that you can remove all handlers from the logging module at the start of your Lambda code. However, I think that's an incredibly bad idea, as it means that you will have no visibility into your function if things go wrong.

A much better approach would be to leverage the capabilities of the Python logging framework, and log your output at different levels. At the top of your function, use this code to set the level of the logger:

LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(os.environ.get('LOG_LEVEL', 'WARNING'))

Then, in your code, use the appropriate level when logging:

LOGGER.debug("this message won't normally appear")
LOGGER.warning("this message will", exc_info=True)

This should trim your log messages considerably, while still giving you the ability to see what's happening in your function when things go wrong.

Docs: https://docs.python.org/3/library/logging.html

Update 2:

If you're determined to disable logging entirely (imo a bad idea), then I recommend doing it within the logging framework, rather than hacking streams:

import logging

logging.root.handlers = [logging.NullHandler()]


def lambda_handler(event, context):
    logging.warn("this message won't appear")

edited May 30, 2022 at 11:22

answered May 26, 2022 at 14:08

Parsifal

4,5447 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Guillaume Over a year ago

But I don't want to completely disable it, as otherwise I won't see when the function runs, how much memory it used, etc. I just want to disable the logs from the code

Parsifal Over a year ago

You can see when the function runs using CloudWatch Metrics or the Monitoring tab on the Lambda Console. If you're determined to muck with it inside your program code, then just remove all existing handlers at the start of your function. That's covered in the Python logging module doc.

Guillaume Over a year ago

My issue is not really with logging level, but not being able to remove the logs from the stdout, which was working on Python 3.7 on Lambda but no longer with 3.9. I do want to log to a file. I will update the question to be more specific

Guillaume Over a year ago

Done. What I want is log everything to a file, and nothing in CloudWatch, whatever the logging level. But also be able to go to CloudWatch and see all the times the function was running. Again sys.stdout = open(os.devnull, 'w') was doing just that on 3.7

Parsifal Over a year ago

Hacking standard io channels is the wrong way to do this. The right way is to replace the log handler that Lambda installs, at the time your Lambda initializes. The docs I linked above describe how to do this.

|

Collectives™ on Stack Overflow

Disable logging on AWS Lambda Python 3.9

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related