As the embedded C is sending quite a lot of numbers and I want to cut down on the transfer time,
To improve time performance you could send and/or receive more than one number at a time. In my tests, array.fromfile() is 10 - 100 times faster than struct.unpack(). It comes at the cost of calling array.byteswap() sometimes to take into account endianness explicitly.
If the C program and Python script were running on the same machine (same size, same endianess); then you could use fwrite to write short ints as platform values on C side and array.fromfile() on Python side to read them back in a native format.
For example, print short ints as binary:
#include <stdio.h>
#include <stdlib.h>
int main(void) {
short a[] = {31415, 9265, 3589, 793};
size_t n = sizeof a / sizeof *a;
if (fwrite(&n, sizeof n, 1, stdout) != 1) exit(EXIT_FAILURE); /* send size */
return (fwrite(a, sizeof *a, n, stdout) < n) ? EXIT_FAILURE : EXIT_SUCCESS;
}
Read it in Python:
#!/usr/bin/env python3
import sys
import array
import struct
# make stdin binary
file = sys.stdin.detach()
# read size
size_format = 'N' # size_t
n, = struct.unpack(size_format, file.read(struct.calcsize(size_format)))
print(n)
a = array.array('h') # native short int
a.fromfile(file, n)
print(a.tolist()) # -> [31415, 9265, 3589, 793]
array.fromfile should be efficient both time and memory-wise. If you don't know the size then call a.fromfile until EOFError is raised.
If C program and Python script are on different machines then you could send integers in the network byte order:
#include <stdio.h>
#include <stdlib.h>
#include <netinet/in.h> /* htons() */
int main(void) {
short a[] = {31415, 9265, 3589, 793};
/* print one integer at a time */
short *p = a, *end = &a[sizeof a / sizeof *a];
for ( ; p != end; ++p) {
uint16_t s = htons(*p); /* convert from native to network byte order */
if (fwrite(&s, sizeof s, 1, stdout) != 1) exit(EXIT_FAILURE);
}
return 0;
}
And swap byte order if necessary on the Python side:
#!/usr/bin/env python
import array
import sys
a = array.array('h') # short int in native byte order, byte swap might be needed
for i in range(15, 128):
try: # double size to avoid O(n**2) behaviour
a.fromfile(sys.stdin, 2 << i)
except EOFError:
break
if sys.byteorder != 'big': # if not network order
a.byteswap() # swap byte order
print(a.tolist()) # -> [31415, 9265, 3589, 793]
To avoid converting to network order, you could send a magic number instead. It allows to send numbers in a native byte order on C side (as in the 1st code example) and check it in Python to swap bytes if necessary:
MAGIC = 1
if a[0] != MAGIC:
a.byteswap()
if a[0] != MAGIC:
raise ValueError("Unexpected %d" % a[0])
