3

My project structure is as follows:

.
├── my_script.Dockerfile
├── README.rst
├── log
│   └── my_script.log
├── pickles
├── requirements.txt
├── requirements_my_script.txt
├── src
│   ├── __init__.py
│   ├── __pycache__
│   ├── main.py
│   ├── modules
│   │   ├── __init__.py
│   │   └── module.py
│   ├── other_scripts
│   │   └── my_script.py
│   └── utils
│       ├── __init__.py
│       ├── __pycache__
│       └── logging_utils.py

my_script.Dockerfile is as follows:

# syntax=docker/dockerfile:1

FROM python:3.10-slim-buster
WORKDIR /src
COPY requirements_my_script.txt requirements_my_script.txt
RUN pip3 install -r requirements_my_script.txt
COPY src/. .
CMD ["python3", "other_scripts/my_script.py"]

my_script.py:

import logging

from src.utils.logging_utils import start_logger


def main(logger):
    # some code here
    logger.info('done')
    sleep(120)
       
    
if __name__=='__main__':
    logger = start_logger('my_script.log')
    try:
        logger.info('Starting..')
        main(logger)
    except Exception as e:
        logger.critical('Crashed with error: {}'.format(e))

And the logging_utils.py:

import os
import logging

from logging.handlers import RotatingFileHandler

def start_logger(filename, level='INFO'): # INFO or DEBUG
    # # LOGGING
    log_folder = os.path.join(os.path.expanduser('~'), 'project_folder_name', 'log')
    if not os.path.exists(log_folder):
            os.makedirs(log_folder)
    log_filename = filename

    logger = logging.getLogger()
    logFormatter = logging.Formatter(fmt='%(asctime)s :: %(levelname)s - %(message)s')
    if level == 'DEBUG':
        logger.setLevel(logging.DEBUG)
    else:
        logger.setLevel(logging.INFO)

    # add a rotating handler
    handler = RotatingFileHandler(os.path.join(log_folder, log_filename), maxBytes=1000000, backupCount=1)
    handler.setFormatter(logFormatter)
    logger.addHandler(handler)
    return logger

This runs flawlessly on my machine, but when I build the docker and try to run it I get the following error:

from src.utils.logging_utils import start_logger
ModuleNotFoundError: No module named 'src'

I tried several different COPY statements in the dockerfile but without success.

Could someone please tell me what I'm doing wrong?

EDIT I tried this dockerfile as well:

# syntax=docker/dockerfile:1

FROM python:3.10-slim-buster
WORKDIR /app
COPY requirements_my_script.txt requirements_my_script.txt
RUN pip3 install -r requirements_my_script.txt
COPY src/ src/
CMD ["python3", "src/other_scripts/my_script.py"]

But I get the exact same error.

4
  • Did you try from utils.logging_utils import start_logger in your my_script.py file? Commented Jul 21, 2022 at 11:08
  • Fails already in my local machine: from utils.logging_utils import start_logger ModuleNotFoundError: No module named 'utils' Commented Jul 21, 2022 at 11:14
  • I think It is because of your WORKDIR inside DockerFile. You should change that or Copy the src dir too, because you are copying only src contents. not that dir Commented Jul 21, 2022 at 11:18
  • Changed the dockerfile to: ``` FROM python:3.10-slim-buster WORKDIR /app COPY requirements_my_script.txt requirements_my_script.txt RUN pip3 install -r requirements_my_script.txt COPY src/ src/ CMD ["python3", "src/other_scripts/my_script.py"] ``` but I get the same error (ModuleNotFoundError: No module named 'src') Commented Jul 21, 2022 at 11:24

3 Answers 3

7

The way to solve this without refactoring the code is adding ENV as follows in the dockerfile:

# syntax=docker/dockerfile:1

FROM python:3.10-slim-buster
WORKDIR /app
COPY requirements_my_script.txt requirements_my_script.txt
RUN pip3 install -r requirements_my_script.txt
COPY src/ src/
ENV PYTHONPATH /app
CMD ["python3", "src/other_scripts/my_script.py"]
Sign up to request clarification or add additional context in comments.

Comments

1

Remove this line from my_script.py:

from src.utils.logging_utils import start_logger

Add these lines to your my_script.py:

from sys import path as syspath
from os import path as ospath
syspath.append(ospath.join(ospath.dirname(ospath.realpath(__file__)),'..','utils')) # To find out where to look, import the utils path into the script path
from logging_utils import start_logger

And do this in Dockerfile :

COPY src/ src/

1 Comment

This works as a workaround, thanks. Will mark as solved in the next days if there's no better answer. BTW the fact that Docker doesn't find modules in a parallel folder is a bug IMO
0

When your code does its import

from src.utils.logging_utils ...

Python looks for a ./src/utils/logging_utils.py, relative to the current directory and anything else in the Python search path. But in your Dockerfile, you strip out the src directory:

COPY src/. .

copies the contents of the src directory into the current directory. I'm guessing if you docker run --rm your-image ls, you'll see the utils directory immediately in the application directory and not in the src subdirectory.

If you need to copy a subdirectory from your host into a subdirectory in your image, you need to repeat the directory name on both sides of COPY, like

COPY src/ src/

1 Comment

Please see my other comment, I tried COPY src/ src/ and it didn't work (same error)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.