Easy way to automate running same python script across several EC2 instances?

Question

I have 4 linux EC2 instances created from the same AMI that I use to process files in S3. I run the same python script on each instance. It takes a directory of files in S3 to process and a number telling it which files it is supposed to process. Say mydir contains myfile1 ... myfile8.

On instance 0 I call: python process.py mydir 0 This causes it to process myfile1 and myfile5.

On instance 1 I call: python process.py mydir 1 This causes it to process myfile2 and myfile2.

And so on.

Inside the script I do: keys = keys[pid::4] where pid is the argument from the command line.

I redistribute changes to my python script by syncing from S3.

Is there a simple way to automate this further? I would like to press one button and say dir=yourdir and have it sync code from s3 and run on each instance.

Subhrajyoti Das · Accepted Answer · 2019-07-08 15:27:44Z

1

You can try using Fabric.
Example taken from Fabric documentation:

from fabric import Connection
result = Connection('web1.example.com').run('uname -s', hide=True)
msg = "Ran {0.command!r} on {0.connection.host}, got stdout:\n{0.stdout}"
print(msg.format(result))

# Output:
# Ran 'uname -s' on web1.example.com, got stdout:
# Linux

answered Jul 8, 2019 at 15:27

Subhrajyoti Das

2,7203 gold badges26 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Easy way to automate running same python script across several EC2 instances?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related