Python Multiprocessing pass commands to processes/pool/queue?

Question

I am, unfortunately, attempting to communicate with about 14 raspberry pi's all at once (art project, burning man, don't ask...) and am using Paramiko to connect to the RPi's over SSH, then issue various other commands: sync files, start server, ect...

To do this I have been using the multiprocessing module in python, but I am running into errors. After I connect to the various RPis, I would like to have the python script hang and wait for an input such as: start server(passing the server name, location, ect), which would send the ssh command via Paramiko to start running the python script on each of the RPIs.

My question is: how do I ensure that the appropriate command gets send to the correct process/pool? For instance, if I instantiate the class which connects to the various RPis and then issue the start server command, I would like: the server on RPi_A to be initialized with the name A, the server on RPi_B to be initialized with the name B, and not RPi_A named B, ect...

Do I need to use the process command for this? Or will pools work? If so is it apply, apply_async, map, map_async. The documentation is a little vague unfortunately.

Sample Code:

import sys
import time
import paramiko
import multiprocessing as mp

login = 'pi'
password = 'raspberry'
serverIp = '192.168.1.143'

config = [
    {'hostname': 'pi1.local', 'name': 'carousel'},
    {'hostname': 'pi2.local', 'name': 'bench'}
]



class Dreamlandia():
    def __init__(self):
        pool = mp.Pool(processes=12)
        results = [pool.apply_async(self.connectToServer, args=(dreamlandObject,)) for dreamlandObject in config]
        output = [p.get() for p in results]

    def connectToServer(self, dreamlandObject):
        host = dreamlandObject['hostname']
        structureName = dreamlandObject['name']
        i = 1
        while True:
            try:
                ssh = paramiko.SSHClient()
                ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
                ssh.connect(host, username=login, password=password)
                print ("Connected to " + structureName)
                break
            except paramiko.AuthenticationException:
                print ("Authentication failed when connecting to " + structureName)
                sys.exit(1)
            except:
                print ("Could not SSH to " + structureName + ", waiting for it to start")
                i += 1
                time.sleep(1)

            # If we could not connect within time limit
            if i == 30:
                print ("Could not connect to " + structureName + ". Giving up")
                sys.exit(1)

If I understood correctly, what you need to do is: connect 14 raspPI by ssh and be able to send them commands ? I would implement it with a main thread and every ssh connections in a subthread for each. For communication look how Queue worked in python. — Cyrbil
– Cyrbil, Commented Jun 6, 2015 at 19:17
I do have the main class instantiated in the main loop, the thing is that I have to send with different arguments to various raspPIs. Would the queue class keep track of which thread is responsible for the raspPI and issue the relevant command to that one only? — Random Name
– Random Name, Commented Jun 6, 2015 at 19:35

Cyrbil · Accepted Answer · 2015-06-06 20:00:16Z

1

Your main process will be responsible for creating, managing and killing your sub process.

from threading import Thread
from Queue import Queue


def worker(name, queue):
    # connect() #connect to raspPI & stuff ...
    # while True: # in real example this loop ...
        cmd = queue.get()
        print "MSG: thread_%s %s\n" % (name, cmd)
        # execute(cmd) # send command to raspPI
        queue.task_done()

# spawn threads
queues = []
num_threads=4
for i in range(num_threads):
    q = Queue()
    queues.append(q)
    t = Thread(target=worker, args=(i,q))
    t.start()

# send message to every threads
for q in queues:
    q.put("hello", False)

Since my workers end, i did not kill my subprocess. You might want to add a condition for your process to die or else you will have to kill it manually.

answered Jun 6, 2015 at 20:00

Cyrbil

6,5081 gold badge27 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Multiprocessing pass commands to processes/pool/queue?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related