Run docker commands in AWS Lambda Function

Question

Goal

I'm curious to know if it's possible to run docker commands within AWS Lambda Function invocations. Specifically I'm running docker compose up -d to run one-off ECS tasks (see this aws article for more info). I know it's easily possible with AWS CodeBuild but for my use case where the workload duration is usually below 10 seconds, it would be more cost effective to use Lambda.

AFAIK Docker DooD is not available given Lambda Functions hosts can not be toggled to mount the host's docker daemon onto the Lambda Function's container.

Attempts

I've tried the following Docker DinD approach below with no luck:

Lambda custom container image:

ARG FUNCTION_DIR="/function"

FROM python:buster as build-image
ARG FUNCTION_DIR

# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
  apt-get install -y \
  g++ \
  make \
  cmake \
  unzip \
  libcurl4-openssl-dev


RUN mkdir -p ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}

COPY ./* ${FUNCTION_DIR}

RUN pip install --target ${FUNCTION_DIR} -r requirements.txt

FROM python:buster
ARG FUNCTION_DIR
WORKDIR ${FUNCTION_DIR}

COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}

ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/bin/aws-lambda-rie
RUN chmod 755 /usr/bin/aws-lambda-rie ./entrypoint.sh ./runner_install_docker.sh
RUN sh ./runner_install_docker.sh

ENTRYPOINT [ "./entrypoint.sh" ]
CMD [ "lambda_function.lambda_handler" ]

contents ofrunner_install_docker.sh (script that installs docker)

#!/bin/bash

apt-get -y update

apt-get install -y \
    software-properties-common build-essential \
    apt-transport-https ca-certificates gnupg lsb-release curl sudo

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

sudo chmod u+x /usr/bin/*
sudo chmod u+x /usr/local/bin/*
sudo apt-get clean
sudo rm -rf /var/lib/apt/lists/*
sudo rm -rf /tmp/*

When I run docker compose or other docker commands, I get the following error:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Mark B · Accepted Answer · 2022-08-27 14:22:38Z

1

Docker isn't available inside the AWS Lambda runtime. Even if you built it into the custom container, the Lambda function would need to run as a privileged docker container for docker-in-docker to work, which is not something supported by AWS Lambda.

Specifically I'm running docker compose up -d to run one-off ECS tasks

Instead of trying to do this with the docker-compose ECS functionality, you need to look at invoking an ECS RunTask command via one of the AWS SDKs.

answered Aug 27, 2022 at 14:22

Mark B

203k27 gold badges339 silver badges332 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Marshallm Over a year ago

Thanks for the clear answer and links to alternatives. The original workload does use the boto3 SDK to run the ECS RunTask command although I was trying to use docker compose to run the ECS task remotely or locally by changing the context without having to mock the ECS boto3 client directly within the Lambda Function during testing. If only ECS RunTask was available for free within Local Stack...

mfittko · Accepted Answer · 2024-05-01 09:11:31Z

0

It seems to be feasible to connect to a remote docker host via tls. It's pretty complex to set this up securely though, but I think it's worth it - considering the performance gains, especially in a (codebuild) CI context.

answered May 1, 2024 at 9:11

mfittko

4214 silver badges3 bronze badges

Collectives™ on Stack Overflow

Run docker commands in AWS Lambda Function

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related