Submitting jobs with numeric arguments to QSUB using Python

Question

I want to run a python function,say my_fun(x1,x2), on different nodes of cluster with SGE (QSUB). I created a script my_script.py which takes in the numeric arguments from the command line, so when running locally, I'd call it as

python my_script.py x1 x2

Now I want to submit this script to the cluster in a loop with different values of x1 and x2. Also, for the node to have access to python & the installed modules, I need to run module load Python/2.7 on the node before calling the python script through QSUB.

This seems like a pretty simple and typical use-case, but I can't find any straightforward way to do this from Python. Going back and forth between BASH and Python seems a bit clunky.

please, have a look at array jobs

Dima Chubarov
– Dima Chubarov

2016-06-06 06:04:29 +00:00
Commented Jun 6, 2016 at 6:04 — Dima Chubarov
– Dima Chubarov, Commented Jun 6, 2016 at 6:04

Ben K. · Accepted Answer · 2016-06-03 16:29:55Z

1

I suggest you divide the job into multiple independent jobs depending on the number of nodes you have.

For each node/core, create a folder with a file containing the a list of the parameters this subjob should treat. Then in python, write a script which reads the file and calls your script (maybe using the multiprocessing module for multi-core support).

EDIT:

If you want to pass additional parameters through qsub, you can call qsub with arguments which can be passed to your script:

qsub -F "myarg1 myarg2 myarg3=myarg3value" myscript.sh

You can find this documentation here

edited Jun 3, 2016 at 16:29

answered Jun 3, 2016 at 16:04

Ben K.

1,1506 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

KartMan Over a year ago

That's horribly inefficient for the number of jobs I need to submit. I think there are ways to do this without requiring writing files to disk.

Ben K. Over a year ago

where do your x1 x2 values come from? because qsub is also inefficient for calling small scripts

Ben K. Over a year ago

Another suggestion if you are looking for a way to farm out your jobs would be to use ipython/jupyter which allows you to start up a bunch of workers, submit calculations and collect the results.

KartMan Over a year ago

That would have been my preferred way actually, but unfortunately, the ipengines kept crashing on importing one module (sklearn.svm). Doing the import through QSUB seems to be fine. I have no idea why. Here's my post on that issue : stackoverflow.com/questions/37531089/…

Ben K. Over a year ago

That is weird. Sorry I can't help you there.

KartMan · Accepted Answer · 2016-06-03 16:43:00Z

0

This more or less does what I'm looking for:

https://gist.github.com/timflutre/a9085660271bd059f71c

import sys
import subprocess

job_param1 = 12.5
job_param2 = 5.0
jobName = "python my_script.py %f %f" % (job_param1,job_param2)
cmd = "module load Python/2.7; sleep 0.2; %s" % jobName
echoArgs = ["echo", "-e", "'%s'" % cmd]
print(" ".join(echoArgs))
qsubArgs = ["qsub","-cwd"]
print(" ".join(qsubArgs))

wholeCmd = " ".join(echoArgs) + " | " + " ".join(qsubArgs)
out = subprocess.Popen(wholeCmd, shell=True, stdout=subprocess.PIPE)
out = out.communicate()[0]

jobId = out.split()[2]
print jobId

answered Jun 3, 2016 at 16:43

KartMan

4795 silver badges20 bronze badges

1 Comment

KartMan Over a year ago

Of course, I'll be running the body inside a (python) FOR loop.

Collectives™ on Stack Overflow

Submitting jobs with numeric arguments to QSUB using Python

2 Answers 2

5 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related