Populating Postgres Docker image during building (not running)

Question

I want to prepare custom image (based on offical Postges image) with two tasks:

Download data (eg. get CSV file by wget),
Load data into database (creating tables, inserts).

I want to do both steps during building image, not during running container, because each of them takes a lot of time, and I want to build image once and run many containers quickly.

I know how to do step 1 (download data) during building image, but I don't know how to load data into database during building image instead of run container (step 2).

Example:

(download - during building image, load - during running container)

Dockerfile:

FROM postgres:10.7

RUN  apt-get update \
  && apt-get install -y wget \
  && rm -rf /var/lib/apt/lists/* 

COPY download.sh /download.sh
RUN /download.sh

download.sh:

#!/bin/bash

cd /docker-entrypoint-initdb.d/
wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/northwindextended/northwind.postgre.sql

To download data I run script myself. To load data I use "initialization scripts" utility from official Postgres image.

Building image:

docker build -t mydbimage .

Running image:

docker run --name mydbcontainer -p 5432:5432 -e POSTGRES_PASSWORD=postgres -d mydbimage

After running, you can see how much loading data takes time:

docker logs mydbcontainer

This example dataset is small, but with bigger, long time running container is awkward.

Possible duplicate of docker postgres with initial data is not persisted over commits — David Maze
– David Maze, Commented Mar 4, 2019 at 22:14
You can't really create images with prepopulated data from any of the standard database images. You can start the database with a mounted host directory and take a snapshot of that as a straightforward workaround, though. — David Maze
– David Maze, Commented Mar 4, 2019 at 22:14
I wanted to avoid create volume, backup database created at first run, and next restoring it. I thought there is a way to start database with "default" user/password, load data and save it as image layer. But how to do with standard image instead of creating empty image and installing Postgres by own? — mkczyk
– mkczyk, Commented Mar 4, 2019 at 22:25
you want your database to have its data on a volume or you will have performance issues. Container file system is slow, volume file system is fast — Thomasleveil
– Thomasleveil, Commented Mar 4, 2019 at 22:41

webwurst · Accepted Answer · 2019-03-04 23:38:03Z

You can dissect the upstream Dockerfile and its docker-entrypoint.sh and just pick the needed snippets to initialize your database:

FROM postgres:10.7

ENV PGDATA /var/lib/postgresql/datap-in-image
RUN mkdir -p "$PGDATA" && chown -R postgres:postgres "$PGDATA" && chmod 777 "$PGDATA" # this 777 will be replaced by 700 at runtime (allows semi-arbitrary "--user" values)

RUN set -x \
  && apt-get update && apt-get install -y --no-install-recommends ca-certificates wget && rm -rf /var/lib/apt/lists/* \
  && wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/northwindextended/northwind.postgre.sql \ 
    -O /docker-entrypoint-initdb.d/northwind.postgre.sql \
  && cp ./docker-entrypoint.sh ./docker-entrypoint-init-only.sh \
  && sed -ri '/exec "\$@"/d' ./docker-entrypoint-init-only.sh \
  && ./docker-entrypoint-init-only.sh postgres \
  && rm ./docker-entrypoint-initdb.d/northwind.postgre.sql ./docker-entrypoint-init-only.sh \
  && apt-get purge -y --auto-remove ca-certificates wget

Build, run and test:

docker build -t mydbimage .

# bring up the database
docker run --rm mydbimage --name pgtest

# run this in another terminal to check for the imported data 
docker exec -ti pgtest psql -v ON_ERROR_STOP=1 --username "postgres" --no-password --dbname postgres --command "\d"

Caveats:

With this setup there is no password set for the database. You can add it during the build, but then it would be persisted in the image. You would need to take precaution that no one gets access to your image. Depending on your setup this might be hard to achieve, maybe that is even impossible.
The second problem is that writes to your database are ephemeral. There is no volume during building to persist the imported data. That is why PGDATA is changed to a directory that is not declared as a volume.

Basically these are the reasons why importing is handled when starting the container instead of during building in the upstream repository. In case you have non secret data that is used as read only it might still make sense to import during the build to save time and for easier handling during the startup of the container.

Collectives™ on Stack Overflow

Populating Postgres Docker image during building (not running)

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related