I have tried to google up this issue and think that I am calling the handlers multiple times in the script and that is why my logger is printing the same info multiple times. As an example:
My main folder structure is:
├── config
├── config.py
├── logs
├── all my log.txt files
├── src
├── models.py
├── trainer.py
├── main.py
For simplicity, my folder structure is as above.
In my config.py, I defined the logging:
def init_logger(
log_file: str = Path(LOGS_DIR, "info.log"),
module_name: Optional[str] = None,
level=logging.INFO,
) -> logging.Logger:
"""Initialize logger and save to file.
Consider having more log_file paths to save, eg: debug.log, error.log, etc.
Args:
log_file (str, optional): [description]. Defaults to Path(LOGS_DIR, "info.log").
Returns:
logging.Logger: [description]
"""
if module_name is None:
logger = logging.getLogger(__name__)
else:
# get module name, useful for multi-module logging
logger = logging.getLogger(module_name)
logger.setLevel(level)
stream_handler = logging.StreamHandler(stream=sys.stdout)
stream_handler.setFormatter(
logging.Formatter("%(asctime)s: %(message)s", "%Y-%m-%d %H:%M:%S")
)
file_handler = logging.FileHandler(filename=log_file)
file_handler.setFormatter(
logging.Formatter("%(asctime)s: %(message)s", "%Y-%m-%d %H:%M:%S")
)
logger.addHandler(stream_handler)
logger.addHandler(file_handler)
return logger
The above code is a standard way of defining logging from what I've seen online. Now I need to log my messages into 3 separate log.txt files. So in each module models.py, trainer.py, main.py I did this:
models_logger = config.init_logger(
log_file=Path.joinpath(config.LOGS_DIR, "models.log"),
module_name="models",
)
trainer_logger = ...
main_logger = ...
It does create 3 different log files, but the messages are printed multiple times. In particular, my main module has a loop as below:
def train_loop(*args, **kwargs):
# calls the trainer n times
I am not sure how to debug this, but as the loop goes on, the logs get repeated more and more, like below:
loop 1: training on fold 1
loop 2: training on fold 2
loop 2: training on fold 2
loop 3: training on fold 3
loop 3: training on fold 3
loop 3: training on fold 3
I am really confused if it is my loop or, a multitude of issues concerning my logging function.
EDIT: A pseudo code:
# main.py, main script
```python
main_logger = ...
def train_loop(*args, **kwargs):
trainer.train(...) # calling trainer script's train
main_logger.info("I am in main script")
# in trainer.py
trainer_logger = ...
def train(fold, *args, **kwargs):
train_logger.info(f"Training on loop {fold}")
