In Windows: Get unbuffered stdout from C program into Java program InputStream [Now with minimal example!]

Question

SOLVED:

HovecraftFullOfEels found my mistake. See the comments on his answer below. I was calling run() on a Thread object when I should have been calling start(). In my original program, that meant it blocked on stderr until the program quit and then got stdout all at once.

UPDATE:

As requested, I've produced a minimal example. Here's the producer:

package producer;

public class Producer {

    public static void main(String[] args) throws InterruptedException {
        for (int i=0; i<10; i++) {
            System.out.println(i);
            System.out.flush();
            Thread.sleep(1000);
        }
    }    
}

Here's the consumer:

package consumer;

import java.io.IOException;
import java.io.InputStream;

public class Consumer {
    static class Dumper extends Thread {
        InputStream r;
        boolean print;

        Dumper(InputStream r, boolean print) {
            this.r = r;
            this.print = print;
        }

        @Override
        public void run() {
            int c;
            try {
                while ((c = r.read()) != -1) {
                    if (print) System.out.print((char)c);
                }
                r.close();
            } catch (IOException ex) {}
        }
    }

    public static void main(String[] args) throws Exception {
        Process p = Runtime.getRuntime().exec("java -jar C:\\Users\\millerti\\Documents\\NetBeansProjects\\Producer\\dist\\Producer.jar");
        new Dumper(p.getErrorStream(), false).run();
        new Dumper(p.getInputStream(), true).run();
        p.waitFor();
    }
}

Expected behavior: Since the producer flushes its output and the consumer is completely unbuffered, the consumer should get the producer's output in real time.

Observed behavior: The consumer receives the producer's output all at once only when the producer finished.

Original post:

I'm working with another developer on a Windows application. She has written a C program (which we can modify) that performs a media conversion and prints progress messages (in percentage) to stdout. I am writing a graphical front-end in Java, and I would like my program to capture those messages and update a GUI element.

The relevant C program code looks like this:

printf ("progress_msg: %d%%\n", currentProgress);
fflush(stdout);

In the Java program, I'm using Runtime.getRuntime().exec() to run the C program and grabbing the raw InputStream objects for C program's stdout and stderr. (The stderr is just being thrown away by another thread.) I'm parsing the stdout stream, looking for those messages (well, I was, using a BufferedReader, but right now, I'm trying to debug this, so I'm talking directly to the low-level InputStream byte source.)

The problem is that although there is no buffering on the Java side (that I know of) and there's an fflush in the C program, the stdout text appears all at once only when the C program finishes. This is unlike when I run it from the command prompt, where the messages come out gradually as expected.

So clearly, there's some buffering going on. Is the Process facility in Java doing some buffering that I don't know about? If so, is there a way to turn it off? Does fflush(stdout) in Windows not work in the expected way? What is the proper Windows equivalent?

Thanks.

Note: I have found two related stackoverflows, but they do not answer this question. In particular, we CAN modify the C program, there's no line buffering, and we ARE doing a proper flush (as far as we know). See I want realtime output of my Runtime.getRuntime().exec() and Unbuffered subprocess stdout on windows.

Community · Accepted Answer · 2017-05-23 11:49:24Z

1

A wild guess from my part, but you state:

... I am writing a graphical front-end in Java, and I would like my program to capture those messages and update a GUI element.
...

In the Java program, I'm using Runtime.getRuntime().exec() to run the C program and grabbing the raw InputStream objects for C program's stdout and stderr. ... I'm parsing the stdout stream, looking for those messages ...

The problem is that although there is no buffering on the Java side (that I know of) and there's an fflush in the C program, the stdout text appears all at once only when the C program finishes.

I again have to guess as you've left out important details, all code, but if your front end is a Swing GUI, then the symptoms you've described suggest that you're trying to get all information on the Swing event thread. If this is a Swing application, a solution is to make sure to read the output of the Stream in a background thread such as that provided by a SwingWorker, to make sure that you buffer this data that you're reading in, and to then display it in the GUI in a thread-safe way (again using a SwingWorker, especially its publish/process method pair).

For better help, please give us better information and code, preferably a small, in fact minimal, example program that we can run and test and improve.

Edit
My test program:

import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;

public class Consumer {

   static class StreamGobbler implements Runnable {
      private String name;
      private boolean print;
      private Scanner scanner;

      public StreamGobbler(String name, InputStream inputStream, boolean print) {
         this.name = name;
         this.print = print;

         scanner = new Scanner(inputStream);
      }

      @Override
      public void run() {
         System.out.printf("in %s run method%n", name);
         while (scanner.hasNextLine()) {
            String line = scanner.nextLine();
            if (print) {
               String output = String.format("From %s: %s%n", name, line);
               System.out.printf(output);
            }
         }
         if (scanner != null) {
            scanner.close();
         }
      }
   }

   private static final String PRODUCER_PATH = "C:/Users/Pete/Documents/Fubar/Java/producer.jar";

   public static void main(String[] args) {
      List<String> command = new ArrayList<>();
      command.add("java.exe");
      command.add("-jar");
      command.add(PRODUCER_PATH);

      try {
         ProcessBuilder pBuilder = new ProcessBuilder(command);
         Process process = pBuilder.start(); 
         StreamGobbler errorGobbler = new StreamGobbler("Error Gobbler", process.getErrorStream(), true);
         StreamGobbler inputGobbler = new StreamGobbler("Input Gobbler", process.getInputStream(), true);

         new Thread(errorGobbler).start();
         new Thread(inputGobbler).start();
         process.waitFor();
      } catch (IOException | InterruptedException e) {
         e.printStackTrace();
      }
   }
}

edited May 23, 2017 at 11:49

CommunityBot

11 silver badge

answered Jun 26, 2014 at 21:22

Hovercraft Full Of Eels

286k25 gold badges268 silver badges391 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Timothy Miller Over a year ago

Your concern is reasonable. Manipulating the UI off of the EDT is a common pitfall. However, in my debugging, I'm actually bypassing the GUI. The InputStream is in its own thread, and the progress is being printed via System.out.print(). I was hoping that someone might have some knowledge about this problem. But it seems as though I'll have to follow your advice and produce a minimal example. I'll update my post when I've completed that.

Hovercraft Full Of Eels Over a year ago

@TimothyMiller: Are you buffering your input correctly? I imagine that you're using an InputStreamReader, are you then wrapping this in a BufferedReader? Please have a look at this question for instance.

Hovercraft Full Of Eels Over a year ago

@TimothyMiller: hmm,... let me test your code, and thanks for posting it! That deserves a 1+ at least.

Hovercraft Full Of Eels Over a year ago

@TimothyMiller: um,... your code,... you're calling run() on a Thread, and you know what that does right? You should call start() always if you want it to actually function as a background thread, right?

Hovercraft Full Of Eels Over a year ago

@TimothyMiller: I posted the code that I used to test your concept.

|

Collectives™ on Stack Overflow

In Windows: Get unbuffered stdout from C program into Java program InputStream [Now with minimal example!]

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related