Python not getting raw binary from subprocess.check_call

Question

How can I get subprocess.check_call to give me the raw binary output of a command, it seems to be encoding it incorrectly somewhere.

Details:

I have a command that returns text like this:

some output text “quote” ...

(Those quotes are unicode e2809d)

Here's how I'm calling the command:

f_output = SpooledTemporaryFile()
subprocess.check_call(cmd, shell=True, stdout=f_output)
f_output.seek(0)
output = f_output.read()

The problem is I get this:

>>> repr(output)
some output text ?quote? ...
>>> type(output)
<str>

(And if I call 'ord' the '?' I get 63.) I'm on Python 2.7 on Linux.

Note: Running the same code on OSX works correctly to me. The problem is when I run it on a Linux server.

Its possible the called program adjusts its output depending on what stdout is. How about opening a regular file and see what bytes are actually written. BTW, SpooledTemporaryFile is over kill. The "spooled" part only works for stuff written from python. When you got the file descriptor it changed it to a regular temporary file. The extra StringIO buffer wasn't used. — tdelaney
– tdelaney, Commented May 16, 2016 at 2:29
I wrote a quick python program that spits out the utf-8 string and your program worked for me. — tdelaney
– tdelaney, Commented May 16, 2016 at 2:29
Try running the command in a shell and redirect to a file. If you have vim installed you should also have xxd, which can display a file hex dump. In your example text, the utf-8 output should look like: 0000000: 736f 6d65 206f 7574 7075 7420 7465 7874 some output text 0000010: 20e2 809c 7175 6f74 65e2 809d 202e 2e2e ...quote... ... The left quote is e2 80 9c and the right quote is e2 80 9d — Lex Scarisbrick
– Lex Scarisbrick, Commented May 16, 2016 at 2:54
Another way to think of it is that since the python 2.7 file read an ascii ?, it was in the file being read. So, the program didn't write the string you think it did. — tdelaney
– tdelaney, Commented May 16, 2016 at 3:00
@tdelaney, did you try it on osx? It actually works correctly for me on OSX. I'll update my question. It prints what I wrote about to the console when I run the command line directly. I can try having it redirect to a file but I don't know what that would show. — Greg
– Greg, Commented May 16, 2016 at 3:06

Community · Accepted Answer · 2017-05-23 11:44:09Z

1

Wow, this was the weirdest issue ever but I've fixed it!

It turns out that the program it was calling (a java program) was returning different encoding depending on where it was called from!

Dev osx machine, returns the characters fine, Linux server from command line, returns them fine, called from a Django app, nope turns into "?"s.

To fix this I ended up adding this argument to the command:

-Dfile.encoding=utf-8

I got that idea here, and it seems to work. There's also a way to modify the Java program internally to do that.

Sorry I blamed Python! You guys had the right idea.

edited May 23, 2017 at 11:44

CommunityBot

11 silver badge

answered May 18, 2016 at 0:36

Greg

47.5k91 gold badges239 silver badges298 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jfs Over a year ago

have you tried to fix your locale settings (locale.getpreferredencoding()) as I've suggested in my answer (check them in the same context as the code that you want to run)?

Community · Accepted Answer · 2017-05-23 11:52:46Z

The redirection (stdout=file) happens at the file descriptor level. Python has nothing to do with what is written to the file if you see ? instead of “ in the file itself (not in a REPL).

If it work on OS X and it "doesn't work" on Linux server then the likely reason is the difference in the environment, check LC_ALL, LC_CTYPE, LANG envvars—python, /bin/sh (due to shell=True), and the cmd may use your locale encoding that is ASCII if the environment is not set (C, POSIX locale).

To get "raw binary" from a subprocess:

#!/usr/bin/env python
import subprocess

raw_binary = subprocess.check_output(['cmd', 'arg 1', 'arg 2'])
print(repr(raw_binary))

Note:

no shell=True—don't use it unless it is necessary
many programs may change their behavior if they detect that the output is not a tty, example.

Collectives™ on Stack Overflow

Python not getting raw binary from subprocess.check_call

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related