Docker-compose: bind Celery to Postgres database

Question

My Docker app runs with Flask as backend and Celery as an asyncronous task manager. Task results are then dumped in a SQLalchemy database owned by Postgres.

However, I am not being able to make Celery interact with Postgres.

Configuration:

docker-compose-dev.yml

  web:
    build:
      context: ./services/web
      dockerfile: Dockerfile-dev
    volumes:
      - './services/web:/usr/src/app' 
    ports:
      - 5001:5000
    environment:
      - FLASK_ENV=development
      - APP_SETTINGS=project.config.DevelopmentConfig
      - DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev 
      - DATABASE_TEST_URL=postgres://postgres:postgres@web-db:5432/web_test
      - SECRET_KEY=my_precious  
    depends_on:  
      - web-db
      - redis

  web-db:  
    build:
      context: ./services/web/project/db
      dockerfile: Dockerfile
    ports:
      - 5435:5432
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres

  celery:
    image: dev3_web
    restart: always
    volumes:
     - ./services/web:/usr/src/app
     - ./services/web/celery_logs:/usr/src/app/celery_logs

    command: celery worker -A celery_worker.celery --loglevel=DEBUG --logfile=celery_logs/celery.log -Q cache
    environment:
     - CELERY_BROKER=redis://redis:6379/0
     - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
     - web
     - redis
    links:
     - redis:redis
     - web-db

  redis:
    image: redis:5.0.3-alpine
    restart: always
    expose:
      - '6379'
    ports:
      - '6379:6379'

  monitor:
    image: dev3_web
    ports:
      - 5555:5555
    command:  flower -A celery_worker.celery --port=5555 --broker=redis://redis:6379/0
    depends_on:
      - web
      - redis

Log:

celery_1| Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".

How do I bind Celery tasks to my Postgres database?

Hi data_garden. I wonder if we can edit this again? I edited this because the usages of the quote block here are not quotes, and some of the usages of the inline code formatting are not code. These format types have a specific semantic meaning just like HTML, and it is preferable not to misuse them as general highlighters. This is particularly the case if you have a long tail of posts using the same approach. — halfer
– halfer, Commented Apr 21, 2019 at 0:04
(Things like Dockerfile and postgres are OK in code formatting, as file names and Unix users are frequently rendered to the console. It is also fine to use code formatting for console I/O. However Celery, Docker, Flask, etc are proper nouns, and so should either be rendered with an initial capital (or with special caps rules where appropriate, such as camel-case or all-caps, depending on how each name is normally written)). — halfer
– halfer, Commented Apr 21, 2019 at 0:07
The mis-spelling of "database" has also been re-added to the title. — halfer
– halfer, Commented Apr 21, 2019 at 0:11
Thank you, data_garden! If you could also abide by those conventions, volunteer editors would appreciate it - they may be disinclined to improve the readability of your earlier posts if new ones would be created in the future :-). — halfer
– halfer, Commented Apr 22, 2019 at 12:23

Allen · Accepted Answer · 2019-04-29 13:08:37Z

1

celery_1   | Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".

suggests that the error from your celery container, which missing the environment valuable setup, you might need DATABASE_URL setup for it which created based on dev3_web.

changes need apply to your docker-compose-dev.yml:

  celery:
    image: dev3_web
    restart: always
    volumes:
     - ./services/web:/usr/src/app
     - ./services/web/celery_logs:/usr/src/app/celery_logs

    command: celery worker -A celery_worker.celery --loglevel=DEBUG --logfile=celery_logs/celery.log -Q cache
    environment:
     - CELERY_BROKER=redis://redis:6379/0
     - CELERY_RESULT_BACKEND=redis://redis:6379/0
     - APP_SETTINGS=project.config.DevelopmentConfig
     - DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev 
    depends_on:
     - web
     - redis
    links:
     - redis:redis
     - web-db

edited Apr 29, 2019 at 13:08

answered Apr 28, 2019 at 19:29

Allen

6,52519 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Allen Over a year ago

add - DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev below your - CELERY_RESULT_BACKEND=redis://redis:6379/0 ?

8-Bit Borges Over a year ago

it worked only when I added DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev AND APP_SETTINGS=project.config.DevelopmentConfig below, as well. could you please add both to your answer for the sake of completeness?

Collectives™ on Stack Overflow

Docker-compose: bind Celery to Postgres database

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related