Deploying Python analysis scripts to a server

Question

I'm the data guy at a small startup in Austin. All of the analysis that I do (thus far) is stored as a set of ad hoc scripts that I run from my laptop. This is a bad idea.

I'm going to sketch my plans here to deploy my analysis, and I'd like to know if there are any glaring holes that I've missed, or anything else that I should be considering. I think the outline I have keeps things atomic enough so that I can plug things in when and where I need to, but also allows me to run one script pretty easily. A secondary (longer-term) goal would be to put a simple web front-end up that allows a user (i.e., employee at my company) to invoke scripts one at a time, see here: Very simple web service: take inputs, email results .

I want to deploy the scripts to a server, and I'm thinking of organizing my scripts into a set of python modules. Then, I want my batch script to look like:

import analysis

batch_dict = analysis.build_batch_dict()
assert sorted(batch_dict.keys()) = ['Hourly', 'Monthly', 'Nightly', 'Weekly']

scripts_to_run = analysis.what_batch() # get from command line?
results_directory = analysis.make_results_directory()
failures = {}
for script in scripts_to_run:
    try:
        script.analyze()
        script.export_results(results_directory)
    except Exception as e:
        failures.update(script.failed(e))

analysis.completed(failures)

I'll rewrite my analyses so that they are handled by a class.

class AnalysisHandler(object):
    ...
    def analyze():
        pass
    def export_results(some_directory):
        pass
    def failed(exception):
        pass
    def run_with_non_default_args(*args, **kwargs):
        pass
    def something_else_im_missing_now():
        pass

All of the scripts will be handled by something that inherits from AnalysisHandler.

The directory structure on the server will look like:

datalytics/
    results/
        02-14-2013/
            script1/
                /log
                /error
                /data
            script2/
                /log
                /error
                /data
            .
            .
            .
            <etc>
    scripts/
        script1/
            bin/
            data/
            doc/
            script_1/
            tests/
            setup.py
        .
        .
        .
    analysis/
        __init__.py
        analysis.py
    batch.py (see above)
    new_script.py
    run_all_tests.py
    run_some_tests.py
    run_this_script.py
    run_everything.py

"I'm going to sketch my plans here to deploy my analysis, and I'd like to know if there are any glaring holes that I've missed, or anything else that I should be considering." Do I need a question mark? — BenDundee
– BenDundee, Commented Feb 14, 2013 at 17:12
Funky permissions issue may throw an exception on making a directory. From a testing background, I think you are better off wrapping your for loop in a 'with open(output) etc' and writing results as you go along, vs storing them. In the event that the server crashes you'd at least like to know what ran and whether there was a possibility one of your scripts got stuck in an infinite something and chewed up all the ram, etc. — pyInTheSky
– pyInTheSky, Commented Feb 14, 2013 at 17:23
not sure the entire scope or complexity of your tests, but leveraging the unittest framework from 'import unittest', may give you some added power in sorting which tests to run, provide logging, etc. — pyInTheSky
– pyInTheSky, Commented Feb 14, 2013 at 17:26
Would you recommend any threading/parallelization in the handler? Or is that just inviting trouble? — BenDundee
– BenDundee, Commented Feb 14, 2013 at 17:41

pyInTheSky · Accepted Answer · 2013-02-14 18:02:43Z

1

As a formal answer:

Watch out for and try to catch the exception when creating files (you may get permissions issues), especially when you add a front end and want to perhaps write in a directory an employee can't modify
With your current setup, I'd suggest wrapping the loop inside the 'with' statement, keeping an open file handle, and flushing results as they come in. This allows you to keep track of progress to some extent, and also lets you know if your server crashes, whether your tests were running, and possibly if one of them caused the crash
You seem to be developing quite a bit of framework. While the unittest module is meant to test python code, it can certainly be leveraged to replace a lot of your framework (ie sorting tests, specifying which tests to run, logging, etc). It will also provide you with an easy way of attaching an interface and then marking tests for expected failures and the like.
As important as output is to have, it's also important to make it useful. You may want it printed in clean text, but if it started in say, a python dictionary, and you flattened it, add a comma to your log file, and dump the dict as a string so you can scoop that thing right back into python and manipulate data if need be.
going off the last point, take a look at json.dumps and json.loads, especially for working w/ logs and a web UI. javascript is friendly w/ that format, and you can save yourself a whole lot of work by keeping everything in a 'python-happy' format
Threading the tests isn't hard to add if you need it. If you know you have one task that takes a long time to run, or is IO intensive, maybe let that go spin off on its own. If you do think you are going to go that route, plan on it from the beginning and be aware of variables you may need to protect, or push all your results onto a queue instead of a dictionary to head off the problem of race conditions
Be aware of the resolution of timestamps, especially if you are loading a dictionary that uses a timestamp as a key o_O ---> just don't do it
I noticed the config function and that it accepted args and kwargs. If you are going to allow configuring of tests, either via ui or file, especially if it's a file, use json friendly format, you can do kwargs = json.loads( open(configfile,'r').read() ) and be super happy that you didn't have to write a parser or regex, or modify your code when you add a parameter.

edited Feb 14, 2013 at 18:02

answered Feb 14, 2013 at 17:46

pyInTheSky

1,4891 gold badge10 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

BenDundee Over a year ago

points[-1]: Nice. I also like the idea of taking inputs to scripts as json. In regards to the amount of framework---I thought that was relatively LITTLE overhead. Leveraging unittest is a nice idea, though.

pyInTheSky Over a year ago

Indeed, it's not much, but I find with testing that it always starts out small and then grows rather quickly. If you are working on coding up a new test, you can keep it in the same directory and just mark as ignore. I've done both. I was very happy w/ my homebrew framework, especially since I did it before unittest added a lot of features in the python 2.7 release, but looking at the module now, I see the benefits of using a 'known' framework, especially if someone other than you will ever have to use it later on.

Collectives™ on Stack Overflow

Deploying Python analysis scripts to a server

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related