368

I have a Python (2.7) app which is started in my dockerfile:

CMD ["python","main.py"]

main.py prints some strings when it is started and goes into a loop afterwards:

print "App started"
while True:
    time.sleep(1)

As long as I start the container with the -it flag, everything works as expected:

$ docker run --name=myapp -it myappimage
> App started

And I can see the same output via logs later:

$ docker logs myapp
> App started

If I try to run the same container with the -d flag, the container seems to start normally, but I can't see any output:

$ docker run --name=myapp -d myappimage
> b82db1120fee5f92c80000f30f6bdc84e068bafa32738ab7adb47e641b19b4d1
$ docker logs myapp
$ (empty)

But the container still seems to run;

$ docker ps
Container Status ...
myapp     up 4 minutes ...

Attach does not display anything either:

$ docker attach --sig-proxy=false myapp
(working, no output)

Any ideas whats going wrong? Does "print" behave differently when ran in background?

Docker version:

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.2
Git commit (client): a8a31ef
OS/Arch (client): linux/arm
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.2
Git commit (server): a8a31ef
Improve this question

16 Answers 16

Reset to default
656

Finally I found a solution to see Python output when running daemonized in Docker, thanks to @ahmetalpbalkan over at GitHub. Answering it here myself for further reference :

Using unbuffered output with

CMD ["python","-u","main.py"]

instead of

CMD ["python","main.py"]

solves the problem; you can see the output now (both, stderr and stdout) via

docker logs myapp

why -u ref

- print is indeed buffered and docker logs will eventually give you that output, just after enough of it will have piled up
- executing the same script with python -u gives instant output as said above
- import logging + logging.warning("text") gives the expected result even without -u

what it means by python -u ref. > python --help | grep -- -u

-u     : force the stdout and stderr streams to be unbuffered;
Improve this answer
Sign up to request clarification or add additional context in comments.

10 Comments

-u seems to work for me, but is there some documentation somewhere with a description of what it actually does?
As suggested by other answers, you can try setting environment variable ENV PYTHONUNBUFFERED=0 in case the -u flag does not work.
This was my problem too. For a more detailed explanation, see stackoverflow.com/a/24183941/562883
Works like a dream on python3, while setting PYTHONUNBUFFERED=0 wasnt helping.
Thanks This helped us to start with debugging on prod
|
179

In my case, running Python with -u didn't change anything. What did the trick, however, was to set PYTHONUNBUFFERED=1 as environment variable:

docker run --name=myapp -e PYTHONUNBUFFERED=1 -d myappimage

[Edit]: Updated PYTHONUNBUFFERED=0 to PYTHONUNBUFFERED=1 after Lars's comment. This doesn't change the behavior and adds clarity.

Improve this answer

7 Comments

In my case, adding -e PYTHONUNBUFFERED=0 helps.
Thank you! I was banging my head off a wall for hours, and couldn't get logs to work even with -u. Your solution fixed it for me on Docker for Mac with Django
i think this is a better solution, that we don't have to rebuild the docker image to see the outputs
This is great thanks. Its worth mentioning that this just needs to be a non empty character to work according to the docs PYTHONUNBUFFERED
PYTHONUNBUFFERED=0 is misleading b/c it suggests that unbuffering is disabled. Instead it's enabled b/c python looks for a non-empty string. That said, better use PYTHONUNBUFFERED=1 which has the same effect but doesn't lead to wrong assumptions.
|
54

If you want to add your print output to your Flask output when running docker-compose up, add the following to your docker compose file.

web:
  environment:
    - PYTHONUNBUFFERED=1

https://docs.docker.com/compose/environment-variables/

Improve this answer

3 Comments

Someone able to elaborate why this variable not works when divined in the Dockerfile and not in the docker-compose file? I thought this shouldn't matter but works like a charm!
@0x78f1935 are you perhaps using an ARG directive instead of an ENV one? It should work in the dockerfile as well.
Doesn't work for me :/ Any Idea as to why this might not work?
44
+75

See this article which explain detail reason for the behavior:

There are typically three modes for buffering:

  • If a file descriptor is unbuffered then no buffering occurs whatsoever, and function calls that read or write data occur immediately (and will block).
  • If a file descriptor is fully-buffered then a fixed-size buffer is used, and read or write calls simply read or write from the buffer. The buffer isn’t flushed until it fills up.
  • If a file descriptor is line-buffered then the buffering waits until it sees a newline character. So data will buffer and buffer until a \n is seen, and then all of the data that buffered is flushed at that point in time. In reality there’s typically a maximum size on the buffer (just as in the fully-buffered case), so the rule is actually more like “buffer until a newline character is seen or 4096 bytes of data are encountered, whichever occurs first”.

And GNU libc (glibc) uses the following rules for buffering:

Stream               Type          Behavior
stdin                input         line-buffered
stdout (TTY)         output        line-buffered
stdout (not a TTY)   output        fully-buffered
stderr               output        unbuffered

So, if use -t, from docker document, it will allocate a pseudo-tty, then stdout becomes line-buffered, thus docker run --name=myapp -it myappimage could see the one-line output.

And, if just use -d, no tty was allocated, then, stdout is fully-buffered, one line App started surely not able to flush the buffer.

Then, use -dt to make stdout line buffered or add -u in python to flush the buffer is the way to fix it.

Improve this answer

Comments

22

Since I haven't seen this answer yet:

You can also flush stdout after you print to it:

import time

if __name__ == '__main__':
    while True:
        print('cleaner is up', flush=True)
        time.sleep(5)
Improve this answer

3 Comments

this worked perfectly for me, stupid that this needs to be there, but works great now.
This worked for me as well. None of the envar methods, nor the "-u" method worked for me.
PS: if you have several print and then you flush at the last one, you get to see all the previous print up to the one with flush=True.
20

Try to add these two environment variables to your solution PYTHONUNBUFFERED=1 and PYTHONIOENCODING=UTF-8

Improve this answer

2 Comments

why would you need PYTHONIOENCODING?
To get away with no ASCII chars.
9

You can see logs on detached image if you change print to logging.

main.py:

import time
import logging
print "App started"
logging.warning("Log app started")
while True:
    time.sleep(1)

Dockerfile:

FROM python:2.7-stretch
ADD . /app
WORKDIR /app
CMD ["python","main.py"]
Improve this answer

2 Comments

nice. tip: use Python 3.
question is in Python 2 (print statement without parenthesis) therefore am using 2 here. Although it is exactly the same behaviour on Python3.6 so thanks for a tip ;)
9

If anybody is running the python application with conda you should add --no-capture-output to the command since conda buffers to stdout by default.

ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "my-app", "python", "main.py"]
Improve this answer

1 Comment

Thanks for this. Now it logs to the interactive console as well as when using the docker log command!
5

I had to use PYTHONUNBUFFERED=1 in my docker-compose.yml file to see the output from django runserver.

Improve this answer

Comments

4

As a quick fix, try this:

from __future__ import print_function
# some code
print("App started", file=sys.stderr)

This works for me when I encounter the same problems. But, to be honest, I don't know why does this error happen.

Improve this answer

2 Comments

Thanks for the tip! Tried replacing all prints with your version, unfortunately it did not work for me, still can't get any output via docker logs (changing between sys.stderr / sys.stdout does has no visible result). Is this a docker bug?
See my answer, the reason is: stderr was unbuffered, so you can make it fix with your solution.
4

If you aren't using docker-compose and just normal docker instead, you can add this to your Dockerfile that is hosting a flask app

ARG FLASK_ENV="production"
ENV FLASK_ENV="${FLASK_ENV}" \
    PYTHONUNBUFFERED="true"

CMD [ "flask", "run" ]
Improve this answer

Comments

4

When using python manage.py runserver for a Django application, adding environment variable PYTHONUNBUFFERED=1 solve my problem. print('helloworld', flush=True) also works for me.

However, python -u doesn't work for me.

Improve this answer

Comments

3

I use visual studio code with docker extension to create Dockerfile in python automatically.

By default it have this line

# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1

it works and show logs when container run in detached mode, so I think that's the preferred way. I use python 3.11

Improve this answer

Comments

2

For me none of the answers here worked.

In order to see messages my Python program was printing to stdout within a Docker container I had to add:

tty: True

for my Python service in my docker compose.yaml

https://docs.docker.com/compose/compose-file/05-services/#tty

Improve this answer

1 Comment

This follows from @atline's answer: when you allocate a TTY, then python's print() uses line buffering by defaults. For me, this is the preferred fix, but the explanation of why allocationg a TTY worked is more useful.
1

Usually, we redirect it to a specific file (by mounting a volume from host and writing it to that file).

Adding a tty using -t is also fine. You need to pick it up in docker logs.

Using large log outputs, I did not have any issue with buffer storing all without putting it in dockers log.

Improve this answer

Comments

1

For anyone who ends up here, I tried a bunch of these solutions but none worked for me. In the end I needed to propertly configure my logger by adding

logging.basicConfig(level=logging.INFO)

before I instantiated my logger instance

logger = logging.getLogger()

This worked perfectly and I didn't have to make any other changes.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.