Elasticsearch docker image with data persistence

I found a solution by investigating the ownership of the folders from inside and outside the docker container. It starting working by running the command sudo chown -R 1000:root $PWD/elasticsearch/data before launching the container so that, in the container, docker thought it owned the directory.

Why does the same docker run create two different results on two different machines? Isn't the point of docker to be one size fits all?

This will work for now but I would like a better solution because I'm not sure if mine will always work as I don't know if 1000 will be the UID for docker every time.

In you docker-compose.yml, you can specify the user with user: $USER, like this:

elasticsearch:
  user: $USER
  image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.5
  volumes:
    - /srv/graylog/es_data:/usr/share/elasticsearch/data

This is also solves the problem, and you don't need to run chown command.

This solved it for me:

sudo chown 1000:1000 -R $PWD/elasticsearch/data

The reason is that ES now runs with user 1000 and needs the directory to have permissions for user 1000

Why this error happens?

The Elasticsearch can't be started using root user, it's probably an extra security the developer team gave us. Therefore, the docker image for Elasticsearch checks if the current container user is root, if it is, it changes the user to elasticsearch:elasticsearch or 1000:1000.

Possible solution

You can change the directory owner to 1000:1000. It will work. But what if you can't change the owner? I had this problem recently, I was trying to map a NFS directory to docker and couldn't change the owner in NFS.

Final solution

the docker-entrypoint.sh located at /usr/local/bin/docker-entrypoint.sh is the responsible for checking and changing root user to elasticsearch. It can be overriden with a simple custom image like the following:

FROM docker.elastic.co/elasticsearch/elasticsearch:7.11.0
COPY ./docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh

Below there's an example of a modified docker-entrypoint.sh currently working for Elasticsearch 7.11.0. You can always get the docker-entrypoint.sh by starting a test container and using something like

docker cp <elasticsearch-container>:/usr/local/bin/docker-entrypoint.sh ./docker-entrypoint.sh

Modified docker-entrypoint.sh

#!/bin/bash
set -e

# Files created by Elasticsearch should always be group writable too
umask 0002

run_as_other_user_if_needed() {
  if [[ "$(id -u)" == "0" ]]; then
    # If running as root, drop to specified UID and run command
    exec chroot --userspec=<uid>:<gid> / "${@}"
  else
    # Either we are running in Openshift with random uid and are a member of the root group
    # or with a custom --user
    exec "${@}"
  fi
}

# Allow user specify custom CMD, maybe bin/elasticsearch itself
# for example to directly specify `-E` style parameters for elasticsearch on k8s
# or simply to run /bin/bash to check the image
if [[ "$1" != "eswrapper" ]]; then
  if [[ "$(id -u)" == "0" && $(basename "$1") == "elasticsearch" ]]; then
    # centos:7 chroot doesn't have the `--skip-chdir` option and
    # changes our CWD.
    # Rewrite CMD args to replace $1 with `elasticsearch` explicitly,
    # so that we are backwards compatible with the docs
    # from the previous Elasticsearch versions<6
    # and configuration option D:
    # https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html#_d_override_the_image_8217_s_default_ulink_url_https_docs_docker_com_engine_reference_run_cmd_default_command_or_options_cmd_ulink
    # Without this, user could specify `elasticsearch -E x.y=z` but
    # `bin/elasticsearch -E x.y=z` would not work.
    set -- "elasticsearch" "${@:2}"
    # Use chroot to switch to UID 1000 / GID 0
    exec chroot --userspec=<uid>:<gid> / "$@"
  else
    # User probably wants to run something else, like /bin/bash, with another uid forced (Openshift?)
    exec "$@"
  fi
fi

# Allow environment variables to be set by creating a file with the
# contents, and setting an environment variable with the suffix _FILE to
# point to it. This can be used to provide secrets to a container, without
# the values being specified explicitly when running the container.
#
# This is also sourced in elasticsearch-env, and is only needed here
# as well because we use ELASTIC_PASSWORD below. Sourcing this script
# is idempotent.
source /usr/share/elasticsearch/bin/elasticsearch-env-from-file

if [[ -f bin/elasticsearch-users ]]; then
  # Check for the ELASTIC_PASSWORD environment variable to set the
  # bootstrap password for Security.
  #
  # This is only required for the first node in a cluster with Security
  # enabled, but we have no way of knowing which node we are yet. We'll just
  # honor the variable if it's present.
  if [[ -n "$ELASTIC_PASSWORD" ]]; then
    [[ -f /usr/share/elasticsearch/config/elasticsearch.keystore ]] || (run_as_other_user_if_needed elasticsearch-keystore create)
    if ! (run_as_other_user_if_needed elasticsearch-keystore has-passwd --silent) ; then
      # keystore is unencrypted
      if ! (run_as_other_user_if_needed elasticsearch-keystore list | grep -q '^bootstrap.password$'); then
        (run_as_other_user_if_needed echo "$ELASTIC_PASSWORD" | elasticsearch-keystore add -x 'bootstrap.password')
      fi
    else
      # keystore requires password
      if ! (run_as_other_user_if_needed echo "$KEYSTORE_PASSWORD" \
          | elasticsearch-keystore list | grep -q '^bootstrap.password$') ; then
        COMMANDS="$(printf "%s\n%s" "$KEYSTORE_PASSWORD" "$ELASTIC_PASSWORD")"
        (run_as_other_user_if_needed echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password')
      fi
    fi
  fi
fi

if [[ "$(id -u)" == "0" ]]; then
  # If requested and running as root, mutate the ownership of bind-mounts
  if [[ -n "$TAKE_FILE_OWNERSHIP" ]]; then
    chown -R <uid>:<gid> /usr/share/elasticsearch/{data,logs}
  fi
fi

if [[ -n "$ES_LOG_STYLE" ]]; then
  case "$ES_LOG_STYLE" in
    console)
      # This is the default. Nothing to do.
      ;;
    file)
      # Overwrite the default config with the stack config
      mv /usr/share/elasticsearch/config/log4j2.file.properties /usr/share/elasticsearch/config/log4j2.properties
      ;;
    *)
      echo "ERROR: ES_LOG_STYLE set to [$ES_LOG_STYLE]. Expected [console] or [file]" >&2
      exit 1 ;;
  esac
fi

# Signal forwarding and child reaping is handled by `tini`, which is the
# actual entrypoint of the container
run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch <<<"$KEYSTORE_PASSWORD"

notice that the comments in the code are the original ones, it may be confusing they saying the code will change the user to elasticsearch.

In case somebody land here with the same problem as me.

java.lang.IllegalStateException: failed to obtain node locks, tried 
[[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations
 are not writable or multiple nodes were started without increasing 
[node.max_local_storage_nodes] (was [1])?

And created 2+ containers with docker-compose.yml, check that the volumes are different for each container. It was that in my case.