4

I have a Django-Celery-PostgreSQL project that should partially run in Docker. Having Redis and PostgreSQL servers running locally on the machine and using virtual environment for the project everything runs smoothly. But when I try to setup Docker instance Celery seems unable to connect to the Postgres database while Django can.

When running project in Docker I put only Django and Celery in it and set network_mode: "host". Redis and Postgres remains on the local machine. Django server itself works flawlessly reading and writing data into the Postgres database, but when I try to run Celery tasks - I get following exception (thrown by django_celery_results):

[2019-08-13 11:26:24,815: ERROR/MainProcess] Pool callback raised exception: OperationalError('could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?\n',)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/billiard/pool.py", line 1750, in safe_apply_callback
    fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/worker/request.py", line 371, in on_failure
    store_result=self.store_errors,
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 160, in mark_as_failure
    traceback=traceback, request=request)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 342, in store_result
    request=request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django_celery_results/backends/database.py", line 38, in _store_result
    using=using,
  File "/usr/local/lib/python3.6/site-packages/django_celery_results/managers.py", line 50, in _inner
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django_celery_results/managers.py", line 129, in store_result
    defaults=fields)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 402, in get
    num = len(clone)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1098, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 256, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 233, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

Docker-compose:

version: '3'

services:
  road_data_service:
    build:
      context: ./
      dockerfile: ./_docker/backend/local_dev/Dockerfile
    volumes:
      - ./:/server
    network_mode: "host"
    ports:
      - "8326:8000"

Dockerfile:

RUN apt-get update \
  && apt-get install -y binutils libproj-dev gdal-bin \
  && rm -rf /var/lib/apt/lists/*

WORKDIR ./
COPY ./ /server

COPY _docker/backend/local_dev/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh

ENTRYPOINT ["entrypoint.sh"]
WORKDIR /server/
CMD python3 manage.py runserver 0.0.0.0:8000 && celery -A worker_name \
worker -l info
1
  • 1
    I have exactly the same problem when my task tries to save data to postgresql. How did you solve it may I ask ? what did you change in DJANGO_SETTINGS_MODEULE ? Thanks in advance. Commented Apr 14, 2020 at 14:21

1 Answer 1

1

Turned out I messed up with the DJANGO_SETTINGS_MODULE in celery.py - Celery got common settings file and not the one intended for the local development.

Sign up to request clarification or add additional context in comments.

2 Comments

How did you fixed it? Can you please elaborate on your answer.
@ArchitKapoor I had several django settings files and messed up with the name of the file that had to be used. So when I wrote correct name in DJANGO_SETTINGS_MODULE everything worked.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.