3

I'm building a test app on AWS. I have a 'master' machine that owns the application. I'd like, upon occasion, to be able to offload processing to some on-demand slave machines. Assuming an instantiated slave, with the bare minimum of required software (not including my main app), how can I 'send' functional units of Python code to one or more slaves for execution?

I know and accept that this architecture may not be optimal for many reasons - and I know it would be simpler if I could preinstall the relevant code on my instances at start-up. But let's assume we have a hard constraint that the slave instances cannot be preloaded beyond the OS and some basic libraries - for example, what if I wanted to squirt some generated code at another machine?

EDIT: StackOverflow is such a SHIT HOT resource! Thanks to everyone for their replies. Awesome. Give me a couple of days and I'll report back.

Improve this question
4
  • I disagree that this belongs on ServerFault. It is a real programming/design question. Commented Aug 11, 2011 at 16:11
  • In what context is your snippet of Python code going to execute? You say that your main app isn't going to be on the remote machine, but you don't say what is going to be there. The more context the snippet requires, the more code you are going to have to ship to the remote. Commented Aug 11, 2011 at 16:14
  • I'm actually trying to understand how to 'ship and execute' the code to the remote (... still a noob when it comes to exploring the possibilities of AWS). But a trivial, valid (in my case) example would be: imagine a function foo() on the master - which owns the source of foo() - how do I send foo() to the remote (... assuming that any libraries upon which foo() depends are already installed on the remote. Commented Aug 11, 2011 at 16:24
  • You can send the code a lot of ways. By default the AWS firewall rules only open port 22 so typically folks will use SSH (e.g.: the paramiko library on PyPI) to copy files. Alternatively, you could send just the string describing the function foo and then use the compile and exec builtin functions to turn that string into bytecode and execute it on the remote machine. Of course, I still think the best solution is dropping the requirement that you can't install the code on the slave machines. Commented Aug 11, 2011 at 16:41

4 Answers 4

Reset to default
4

I've run into this problem several times in various flavors.

Option 1 - eval()

When I want to just hack something out quickly, I use eval() or one of it's cousins in the stdlib. Transfer the source to your master, then compile and eval:

src = getSourceFromMaster()
obj = compile(src, "master.py", "exec")
exec(obj)

As long as your transport to get source from client to server is trustworthy and the actions the source needs to take are relatively straightforward, this works. A few times I've needed tighter integration between the master and slave machines, with lots of back and forth processing or complex data structures. In those cases, I use Pyro.

Option 2 - Pyro

Pyro is a full on, cross-platform remote method execution library for Python. I've used it in production environment to send processing from a linux machine to a windows machine and back, and it's been super stable.

Example from their docs:

Master:

# save this as greeting.py
class GreetingMaker(object):
    def get_fortune(self, name):
        return "Hello, {0}. Here is your fortune message:\n" \
               "Behold the warranty -- the bold print giveth and the fine print taketh away.".format(name)

Slave:

# save this as client.py
import greeting
name=raw_input("What is your name? ")
greeting_maker=greeting.GreetingMaker()
print greeting_maker.get_fortune(name)

output:

$ python client.py
What is your name? Irmen
Hello, Irmen. Here is your fortune message:
Behold the warranty -- the bold print giveth and the fine print taketh away.

The super-awesome thing about Pyro is the "import greeting" line in client.py -- that code comes from the server.

If you're starting from a bare OS install, you can push down a python script to host the client code for either with SSH immediately after you spin up the new instance. Then you'll have a nice infrastructure to work within.

I cannot speak in anymore detail to the application of either of these to AWS, nor how they compare to the utilities provided by the AWS infrastructure. Welcome ideas or discussion on it.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

Use SSH to scp the code to the slave. Setup the slaves with an EC2 keypair dedicated to this purpose. Put the EC2 keypair on the master. This is simple and quick to setup.

Another option is to use AWS Elastic MapReduce and dumbo or mrjob to manage the slave. This is more work to setup but will be more robust to failures, provide you task tracking features, etc.

Improve this answer

1 Comment

Thanks. Sounds like my first port of call.
0

You could investigate the XMLPRClib

It requires a little bit of setup, but once it is setup it is very much like you're calling local functions.

Improve this answer

2 Comments

Thanks. Not explored XMLRPClib yet ... looked too daunting for what I thought might be simpler.
XMLRPC isn't actually as hard to use as it first appears.
0

I think Pyro 3 will do this by copying python bytecode files from master to slave. However, I think this 'mobile code' approach is hacky because it's one more thing you have to test and debug. It makes what code is running (and when) unpredictable and can create a whole new set of problems that are more difficult to diagnose. Distributed computing can already be challenging to manage, I don't think you need the complexity of not knowing what code is running where.

IMO, You would be better off just ditching this (artificial?) restriction that you can't install the code on the worker machines. Once you drop that constraint and setup the code on your workers you will have a bunch of options. Just to name a handful:

For managing setting up code on the workers you might look at writing deployment scripts in Fabric.

Improve this answer

7 Comments

Thanks Mike - I'm already using Fabric and it's sensational. I'm just kinda tempted by the notion of having a 'master' machine that co-ordinates things - including deciding what code to run where. It's a single point of failure, of course, but what does that really mean in an AWS environment?
By all means, coordinate things on a master server. I'm just advocating a slightly less automagical approach. Also, remember that a single point of failure is still bad in an AWS environment. However, AWS gives you tools to allow you to cope. For example, you can have masters in multiple ec2 zones or regions for example. In any case you still need to deal with the problem of having your code in more than one place to actually be able to cope with AWS availability issues.
Point taken. Ironically (if that's the word) I was mid-flow moving my app from Slicehost to AWS in Ireland when AWS in Ireland had a major outage. Then today (post move) Slicehost had a major outage in Illinois. :-) And yeah, right now it's an 'artificial' restriction, but it represents a general pattern that I feel fitting for future work.
Sorry to hear about the outages, I've suffered through a few myself ;-) So why do you feel it is more likely that you'll have to programmatically / dynamically move code around versus installing the code at the same time you setup an ec2 machine? I'm thinking if you've got a system to dynamically provision an ec2 machine and installing python and the dependencies for a snippet of code to run that deploying your code to that machine cannot be that much additional effort or time.
Just have a prickly feeling about having to do the book-keeping for pre-installing x versions of y types of software on my slaves, all keeping in sync with whatever works on the master. It seems better to me to dry-run it on the master and then distribute it. I also have a design hope for a pattern that involves genuinely generated code that couldn't possibly be preinstalled.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.