parallel processing of python script

Question

I have a python script which takes one input at a time and process it . I want to run script parallel with different inputs at a same time . Like 50 or 100 times but with different inputs feed from txt file .

i execute script like:

python script.py -d url1

then

python script.py -d url2

then

python script.py -d url3

and instead of input one argument at a time , i want to feed those url from a text file and process them in parallel .

I tried this script running in bash shell using gnu-parallel but bash script not runs python shell and thus errors.

the code is as follows---

#!/usr/bin/env bash
doit() {
    host="$1"
    ~/script/python script1.py -d $host
      }
   export -f doit

   cat "$1" | parallel -j50 -k doit

contents of txt file---

url1.com
url2.com
url3.com
--------
url1000.com
url_any.com

i tried to run it with with gnu parallel but before , i run parallel in bash script like go language accepts but this needs shell in python and thus i can't able to run it . — Amarjit Singh
– Amarjit Singh, Commented Sep 4, 2018 at 15:00

Mark Setchell · Accepted Answer · 2018-09-04 15:01:25Z

6

With GNU Parallel, like this:

parallel --dry-run -a arguments.txt python script.py

which assumes your arguments are one per line in "arguments.txt".

Use parallel -k ... to keep outputs in order, if required.

Use parallel --bar ... to get a progress bar.

answered Sep 4, 2018 at 15:01

Mark Setchell

210k32 gold badges309 silver badges503 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Amarjit Singh Over a year ago

python script itself takes argument like "python script1.py -d abc"

Mark Setchell Over a year ago

Not sure I understand. Just add any parameters you need at the end. Please click edit under your question and show 3 lines of your "arguments.txt" file and then show the 3 corresponding commands that should be executed.

Amarjit Singh Over a year ago

i tried your code and it just output the text required command rather then executing that paricular command like " python script.py - d any.com "

Mark Setchell Over a year ago

If the output looks correct, run it again but without --dry-run which is for debugging.

Amarjit Singh Over a year ago

yeah its working , can it take like -j50 or -j100 like argument to control input.

|

Victor Wong · Accepted Answer · 2018-09-04 15:10:22Z

0

An alternative to GNU Parallel is to use Python subprocess to execute the command repeatedly.

This is an example using Python 2.7.

First your code will need to read the text file to assign all the arguments to a list.

with open('<Arguments text file location>') as f:
    arguments = f.read().splitlines()

Then you use a loop to run the command once for each argument using subprocess.

import subprocess
procs = []
for argument in arguments:
    cmd = 'python script.py %(argument)' % {'argument': argument}
    procs.append(
        subprocess.Popen(cmd, shell=True)
    )
exit_codes = [p.wait() for p in procs]

edited Sep 4, 2018 at 15:10

answered Sep 4, 2018 at 14:44

Victor Wong

3,8511 gold badge23 silver badges33 bronze badges

2 Comments

greenlantern Over a year ago

if no. of arguments are say 1000, this would create 1000 shell processes right? is that a thing to worry about? Is there any limit to no of sub processes we can spawn without crashing anything?

Victor Wong Over a year ago

AFAIK the limit is your CPU and memory. You need to do some benchmarking for that.

Collectives™ on Stack Overflow

parallel processing of python script

2 Answers 2

7 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related