Use Bash ssh to run a Java program on multiple remote machines

I need to run a Java program on many remote machines. I'm using ssh in a loop and calling a remote script that runs the Java program.

As you can imagine, this is used for testing a distributed system on a cluster.

Problem is, the script hangs right after I input the password for the first ssh session. It's probably a bash error, as the Java program runs fine on local.

The exact structure is this, a local bash script running many remote bash scripts. Each remote script compiles and runs a Java program. This Java program starts a separate thread to do some work. When a SIGINT signal is received, the Java thread is informed so it can exit cleanly.

I made a simplified working example.

EDIT: the code below now works (fixed for posterity)

Please, if you want to answer, don't change the structure of the code too much, or it won't resemble the original one and I won't be able to understand what's wrong.

Bash script that's run by hand

#!/bin/bash

function startBatch()
{
    #the problem was using -n
    ssh -f "$1" "cd $projectDir;./startBatch.sh $2"
}

function stopBatch()
{
    #the problem was using -n
    ssh -f "$1" "pkill -f jnode_.*"
}

projectDir=NetBeansProjects/Runner

#start nodes
nodeNumber=0
while read node; do
    startBatch "$node" "$nodeNumber"
    nodeNumber=$(($nodeNumber + 1))
done < ./nodes.txt

sleep 3

#stop nodes
while read node; do
    stopBatch "$node"
done < ./nodes.txt

Bash script that is run by the other script

#!/bin/bash

#this is a simplified working example
myNumber=$1
$(exec -a jnode_"$myNumber" java -cp build/classes runner.Runner "$myNumber.txt")

Here's a less simplified version of the above script. Check the second part of the accepted answer if you want proper logging.

#!/bin/bash

batchNumber=$1
procNumber=0
batchSize=3
while [ "$procNumber" -lt "$batchSize" ]; do
    procName="$batchNumber"_"$procNumber"
    #this line was no good
    #$(exec -a jnode_"$procName" java -cp build/classes runner.Runner "$procName.txt" &)
    #this line works fine
    exec -a jnode_"$procName" java -cp build/classes runner.Runner "$procName.txt" 1>/dev/null 2>/dev/null &
    procNumber=$(($procNumber + 1))
done

Java Runner (the thing that starts the thread)

import java.io.File;
import java.io.FileNotFoundException;
import java.io.PrintStream;

public class Runner {

    public static void main(String[] args) throws FileNotFoundException, InterruptedException {
        //redirect all outputs to a given file
        PrintStream output = new PrintStream(new File(args[0]));
        System.setOut(output);
        System.setErr(output);

        //controlled object
        final MyRunnable myRunnable = new MyRunnable();

        //shutdown the controlled process on command
        Runtime.getRuntime().addShutdownHook(new Thread() {
            @Override
            public void run() {
                myRunnable.stop = true;
            }
        });

        //run the process
        new Thread(myRunnable).start();
    }
}

Java MyRunnable (the running thread)

public class MyRunnable implements Runnable {

    public boolean stop = false;

    @Override
    public void run() {
        while (!stop) {
            try {
                System.out.println("running");
                Thread.sleep(1000);
            } catch (InterruptedException ex) {
                System.out.println("interrupted");
            }
        }
        System.out.println("stopping");
    }
}

Do not to use System.exit() in your Java program, or the shutdown hook will not be properly called (or completely executed). Send a SIGINT message from outside.

As it was mentioned int the comments, inputting passwords can be boring. Password-less RSA keys are an option, but we can do better. Let's add some security features.

Create the public/private key pair

ssh-keygen -t rsa
Enter file in which to save the key (home/your_user/.ssh/id_rsa): [input ~/.ssh/nameOfKey]
Enter passphrase (empty for no passphrase): [input a passphrase not weaker than your ssh password]

Add the public key to the authorized_keys file of the remote hosts, so it can be authenticated.

#first option (use proper command)
ssh-copy-id [email protected]

#second option (append the key at the end of the file)
cat ~/.ssh/nameOfKey.pub | ssh [email protected] "cat >> ~/.ssh/authorized_keys"

Now, if we use ssh-agent, we can make it so that the passphrase(s) will be asked only once (when executing the first command). Notice, it will ask for the passphrases (the ones inputted when creating the keys), not for the actual ssh passwords.

#activate the agent
eval `ssh-agent`

#add the key, its passphrase will be asked
ssh-add ~/.ssh/keyName1

#add more keys, if needed
ssh-add ~/.ssh/keyName2

You now have a very simple yet functional testing framework for your distributed system. Have fun.