Submit Slurm Jobs using Python

Question

I have foo.sh as below

#!/bin/bash
#SBATCH -A research
#SBATCH -p long
#SBATCH --mem-per-cpu=1024
#SBATCH -N 1
#SBATCH -n 24
#SBATCH -t 2-00:00:00
#SBATCH --mail-type=END
#SBATCH --exclude=node37

module load Gaussian/09revC

export GAUSS_SCRDIR=/scratch/$USER.$SLURM_JOBID
/bin/mkdir -p $GAUSS_SCRDIR

g09 $1

/bin/rm -rf $GAUSS_SCRDIR

And I call foo.sh from another bash file called submit.sh

#!/bin/bash
sbatch foo.sh 00_molecule_name_some_method.com
sbatch foo.sh 01_molecule_name_some_method.com
sbatch foo.sh 02_molecule_name_some_method.com

This submits multiple jobs.

I want to change submit.sh to Python file which will call foo.sh provide it with argument which will go to $1 in foo.sh, Python should not wait for the jobs to get completed.

Also molecule_name and some_method are variables which I will provide in for loop

template Python script submit.py

for molecule_name in molecules:
    for some_method in methods:
        foo.sh molecule_name some_method

I get this error:

/bin/mkdir: cannot create directory ‘/scratch’: Permission denied
PGFIO/stdio: No such file or directory
PGFIO-F-/OPEN/unit=11/error code returned by host stdio - 2.
File name = /scratch/USERNAME./Gau-41212.inp
In source file ml0.f, at line number 181
PGFIO/stdio: No such file or directory
PGFIO-F-/OPEN/unit=11/error code returned by host stdio - 2.
File name = /scratch/USERNAME./Gau-41213.inp
In source file ml0.f, at line number 181

Vikas Mulaje · Accepted Answer · 2020-04-20 13:48:57Z

1

Use Subprocess

import subprocess
for molecule_name in molecules:
    for some_method in methods:
        command1 = subprocess.Popen(['foo.sh', molecule_name, some_method])

answered Apr 20, 2020 at 13:48

Vikas Mulaje

7656 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

WhySoSerious Over a year ago

So document says this will spawn child process in new process. This will not halt the python script right ?

Vikas Mulaje Over a year ago

No, it will execute the command provided which will start in background in new process.

Collectives™ on Stack Overflow

Submit Slurm Jobs using Python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related