How to read a python binary string from Standard Input in Java

Question

I have an python application using protocol buffers, and a Java application using protocol buffers too. What i want to do is just to be able to print the message (binary string after serializing) to the standard output. For this purpose i do the following in the Python app:

def printMessage(self, protobuf_msg):

data = protobuf_msg.SerializeToString()

sys.stdout.write(data)

sys.stdout.flush()

def main():

protobuf_msg = create_message()

controller.printMessage(protobuf_msg)

And after that, i want to pipe this output (python pytonApp | java javaApp) and get this data with the javaApp and parse it. I've tried two options, doing this with the Protobuf API:

protected ProtobufMsg receiveMsg() throws Exception{

ProtobufMsg message = null;

message = protobuf_msg.parseFrom(System.in);

return message;

}

I have also try to do this with the BufferedInputStream in the following way:

protected ProtobufMsg receiveMsg() throws Exception{

ProtobufMsg message = null;

byte[] data = receiveFromStd();

message = protobuf_msg.parseFrom(data);

return message;

}

public byte[] receiveFromStd() throws Exception{
BufferedInputStream input = new BufferedInputStream(System.in);

byte[] out = new byte[1024];
int i=0;

System.out.println("Entering While");

while((out[i] = (byte)input.read())!= -1){

    i++;
System.out.println("One byte readed");

}
byte[] data_out = new byte[i];

for(int l=0; l<data_out.length; l++){

    data_out[l]=out[l];

}

return data_out;
}

So it's obvious i'm doing something wrong but i'm not able to realized what i'm doing wrong, because it stays inside input.read()...

EDIT: I have decided to change strategy and now i get first the size of packet, and afterwards the packet, as i'm using input.read(byte []) function... The script i'm using is the following:

FIFO_FILE=/tmp/named_$$   # unique name ($$ is the PID of the bash process running this script)
mkfifo $FIFO_FILE   
export FIFO_FILE    # export the env variable
ant run &    # start a background process that reads the env variable and reads the fifo
cat > $FIFO_FILE #  reads the standard input and writes to the fifo 
rm $FIFO_FILE

And I call this as: python pythonApp.py | ./script .

readLine() requires an end of line character to return or "unblock". so maybe you also need to also add sys.stdout.write('\n') — lalli
– lalli, Commented Nov 14, 2013 at 15:39
Hint: The {} button in the Stackoverflow editor turns the selected lines into properly indented, highlighted code. — Aaron Digulla
– Aaron Digulla, Commented Nov 15, 2013 at 10:30

Aaron Digulla · Accepted Answer · 2013-11-15 10:11:46Z

1

You can't use readLine() since you have binary data.

Don't use the Reader API, you have binary data. Just use BufferedInputStream.
protobuf surely has an API to read directly from a stream. Use that. Don't forget to flush the output of the child or the data will sit forever in the 4K pipe buffer:
```
sys.stdout.write(data)
sys.stdout.flush()
```

edited Nov 15, 2013 at 10:11

answered Nov 14, 2013 at 15:43

Aaron Digulla

330k111 gold badges626 silver badges840 bronze badges

Sign up to request clarification or add additional context in comments.

13 Comments

Aaron Digulla Over a year ago

The second one doesn't work because you only get an EOF when the child terminates. You really need to know in advance how many bytes to read. It was my impression that protobuf can do that, so the first call should work unless you forget to flush in the child.

user2992697 Over a year ago

Thank you, I've tried with the API of protobuf: protobufMsg.parseFrom(System.in); and also, i tried with BufferedInputStream: BufferedInputStream input = new BufferedInputStream(System.in); while((out[i] = (byte)input.read())!= -1){ i++; }. but in this case it stays at input.read() once and never reads any byte. Thank you for your help @AaronDigulla.

user2992697 Over a year ago

I've flushed the output of python app, and i've tried with protobuf API and BufferedInputStream and still doesn't work... But i'm not sure if i'm using properly the BufferedInputStream...

Aaron Digulla Over a year ago

Edit the question and show us both the Python code you use to send and the Java code you use to read.

Aaron Digulla Over a year ago

Add sys.stdout.flush() after sys.stdout.write(data)

|

Bruce Martin · Accepted Answer · 2013-11-14 20:53:15Z

0

I can not comment on Python side, but if it is a binary message you can do

    FileInputStream in = new FileInputStream(fileName);

    message = Extension01.Message.parseFrom(in);

or if it is a delimited message:

    FileInputStream in = new FileInputStream(fileName);

    message = Extension01.Message.parseDelimitedFrom(in);

There is no need to read it in as bytes

answered Nov 14, 2013 at 20:53

Bruce Martin

10.6k1 gold badge29 silver badges38 bronze badges

Collectives™ on Stack Overflow

How to read a python binary string from Standard Input in Java

2 Answers 2

13 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

13 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related