I am literally pulling my hair out over this problem. Recently I have started working on a Remote Code Execution service.
The service sends the following payload to a Celery worker:
{
"language": "python",
"code": "print('Hello World!')"
}
The service expects to receive back the output of the code.
The Celery worker runs in a docker container, with the following Dockerfile:
FROM python:3.9-slim-buster
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONBUFFERED 1
RUN mkdir /worker
WORKDIR /worker
COPY requirements.txt /worker/
RUN /usr/local/bin/python -m pip install --upgrade pip
RUN pip install -r /worker/requirements.txt
COPY . /worker/
For the service and worker I am using ths docker-compose file:
version: "3"
services:
rabbit:
image: rabbitmq:latest
ports:
- "5672:5672"
api:
build: app/
command: uvicorn remote_coding_compilation_engine.main:app --host 0.0.0.0 --reload
volumes:
- ./app:/app
ports:
- "8000:8000"
worker:
build: worker/
command: watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery --app=worker.celery_app worker -c 2 --loglevel=INFO
#celery --app=worker.celery_app worker -c 2 --loglevel=INFO
volumes:
- ./worker:/worker
depends_on:
- api
- rabbit
flower:
image: mher/flower
command: ["flower", "--broker=pyamqp://guest@rabbit//", "--port=8888"]
ports:
- "8888:8888"
depends_on:
- api
- rabbit
- worker
The worker creates a .py file with the code given in the payload and then runs a bash script to execute the code, the code output should be automatically redirected to an out_file from python. However I am not able to get the output of the bash script.
This is the bash script:
#!/bin/bash
if [ $# -lt 1 ]
then
echo "No code running command provided!"
else
eval $@
fi
I am passing the follwoing arguments to this script: python path_to_file
This is the worker code which runs the bash script:
def execute_code(cls, command: str, out_file_path: str) -> None:
"""
Function which execudes the given command and outputs the result in the given out_file_path
@param command: command to be executed
@param out_file_path: path to the output file
@return None
@raise: FileNotFoundError if the command does not exist
@raise: IOError if the output file cannot be created
@raise: CalledProcessError if the command fails to be executed
"""
try:
logging.info(f"Creating output file: {out_file_path}")
with open(out_file_path, "w+") as out_file_desc:
logging.info(f"Creating process with command: {command} and output_file: {out_file_path}")
os.chmod(cls.EXECUTE_CODE_SCRIPT_PATH, 0o755)
rc = subprocess.call(command, shell=True, stdout=out_file_desc, stderr=out_file_desc)
logging.info(f"Commandmd: {command.split()}")
logging.info(f"Return Code: {rc}")
logging.info(f"OutFile Content: {out_file_desc.read()}")
if rc != 0:
logging.debug(f"Command execution was errored: {out_file_desc.read()}")
raise subprocess.CalledProcessError(rc, command)
except FileNotFoundError as fnf:
logging.error(
f"Command not found: {command}"
)
raise fnf
except IOError as ioe:
logging.error(
f"Could not create output file: {out_file_path}"
)
raise ioe
except subprocess.CalledProcessError as cpe:
logging.error(
f"Process exited with error: {cpe.output}"
)
raise cpe
Obs1. The command to be executed in subprocess.call is in the form /worker/execute_code/code_execute.sh python /worker/tmp/in_files/some_file_name.py
Obs2. I am sure the script runs the command since, for example if I alter the path to folder containing the code to be executed I am getting a "FileNotFound" error.
Obs3. If instead of "$@" I am using literally python <<hardcoded_path>>, I can see the output of the bash script
Obs4. The running entity has rw access on the output file. I cannot get the output of the script even if I use the default stdout
What I've tried:
- Firstly I didn't want to use a bash script, I wanted to use subprocess.run, but again I was unable to get the output
- If I call the script using the terminal, I get the code output
Any help would be appreciated, thank you!
