Encoding and Decoding Bytes in Python

Question

I'm dealing with a communication channel that:

Converts \n characters into \r\n
Removes 0xfd, 0xfe, and 0xff characters entirely

Not ideal for transferring binary data. I can effectively use base64 to transfer binary data through it, but this makes the data being transferred 33% larger, which, when dealing with large amounts of binary data like I am, kinda sucks.

A simple, not entirely efficient way to use Python to create my own encoding is this:

escape_char = b'='

# These are the bytes we don't want in our stream
bytes_to_escape = b'\n\xfd\xfe\xff'

# This is a mapping of the bytes being replaced, with the bytes we're
# replacing them with
replacements = {
    escape_char: escape_char,
    **{
        byte: i.to_bytes(1, "big")
        for i, byte in enumerate([bytes([b]) for b in bytes_to_escape])
    }
}

# Reverse mapping, for decoding
reverse_replacements = {v: k for k, v in replacements.items()}

# Encoder
def encode(data: bytes) -> bytes:
    result = b''

    for byte in [bytes([i]) for i in data]:
        replacement = replacements.get(byte)
        if replacement:
            result += escape_char + replacement
        else:
            result += byte

    return result

# Decoder
def decode(data: bytes) -> bytes:
    result = b''
    i = 0

    while i < len(data):
        current_byte = bytes([data[i]])
        next_byte = bytes([data[i+1]]) if i+1 < len(data) else None

        if current_byte == escape_char:
            if not next_byte:
                result += current_byte
            else:
                replacement = reverse_replacements[next_byte]
                result += replacement
                i += 1
        else:
            result += current_byte

        i += 1

    return result

This will essentially:

Escape the = character with ==, because it's used as our escape character
Convert \n to 0x00
Convert 0xfd to 0x01
Convert 0xfe to 0x02
Convert 0xff to 0x03

Is there a way to make this more efficient, short of writing it in a low-level language as a python module? Maybe something that uses regex might work faster?

"This works, until ..." Does your code work as expected or not? — user51621
– user51621, Commented Nov 23, 2022 at 11:33
@RichardNeumann it does what it's told to do, but doesn't handle some types of input data — John
– John, Commented Nov 23, 2022 at 13:51
I would suggest that you think of and implement a working encoding first and come back with that working encoding later, which we then can review. — user51621
– user51621, Commented Nov 23, 2022 at 14:12
Welcome to the Code Review Community. We only review code that is working as expected, there are other sites that will help you debug your code. Please read Where can I get help? and How do I ask a good question?. — pacmaninbw
– pacmaninbw ♦, Commented Nov 23, 2022 at 14:43

stefan · Accepted Answer · 2022-11-24 11:03:28Z

bytes

You are working with bytes which is very cumbersome.

You have to prefix all literals
single bytes have type int and need to be converted to to bytes again At least I cannot do this without debugging some errors.

It is much more convenient to work with strings. So I suggest to decode the bytes to a string with an 8-bit codec like 'latin-1'. No more bytes([x]) or x.to_bytes(). You can use plain comprehension, plain ''.join(), etc. If required you encode the string to bytes again with the very same codec.

dict.get()

For conditional translations there is a nice trick - dict.get() - which allows to pass a default value for non-existing keys.

translated = replacements.get(char, char)

This returns replacements[char] if char is in replacements.keys(). Otherwise it returns the default char. To use this functionality you include the escape character in the translation table (however you fill it).

replacements = {escape_char: escape_char + escape_char,
       '\n':   escape_char + '\x00',
       '\xfd': escape_char + '\x01',
       '\xfe': escape_char + '\x02',
       '\xff': escape_char + '\x03',
       }

The encoding function now reads

def encode(data: bytes) -> bytes:
    str_data = data.decode(encoding='latin-1')
    str_enc = ''.join(replacements.get(x, x) for x in str_data)
    return str_enc.encode(encoding='latin-1')

loop like a pro

In your decoding function you loop over len(data) which is an anti-pattern. You should always loop over the elements. To make it worse you use a while loop and increment the loop counter manually. You also access elements with index i+1 which again is error prone. Loop over elements only and keep a little state like a previous_byte. You cannot be out of bounds.

def decode(data: bytes) -> bytes:
    d_data = data.decode(encoding='latin-1')
    l = []
    esc = ''
    for x in d_data:
        if not esc and x == escape_char:
            esc = x
            continue
        l.append(reverse_replacements.get(esc + x, esc + x))
        esc = ''
    assert esc == ''
    return ''.join(l).encode(encoding='latin-1)')

testing

For your nicely testable functions you should provide some tests. There are unit test frameworks available. You can also do the most primitive assertions, of course in a main guard.

if __name__ == '__main__':
    assert encode(b"asdf") == b"asdf"
    assert decode(b"asdf") == b"asdf"

    assert encode(b"as\ndf") == b"as=\x00df"
    assert decode(b"as=\x00df") == b"as\ndf"
    assert encode(b"\ndf") == b"=\x00df"
    assert decode(b"=\x00df") == b"\ndf"
    assert encode(b"as\n") == b"as=\x00"
    assert decode(b"as=\x00") == b"as\n"

    assert encode(b"as\xfddf") == b"as=\x01df"
    assert decode(b"as=\x01df") == b"as\xfddf"

    assert encode(b"as\xfedf") == b"as=\x02df"
    assert decode(b"as=\x02df") == b"as\xfedf"

    assert encode(b"as\xffdf") == b"as=\x03df"
    assert decode(b"as=\x03df") == b"as\xffdf"

Stack Exchange Network

Encoding and Decoding Bytes in Python

1 Answer 1

bytes

dict.get()

loop like a pro

testing

You must log in to answer this question.

Hot Network Questions

Encoding and Decoding Bytes in Python

1 Answer 1

bytes

dict.get()

loop like a pro

testing

You must log in to answer this question.

Related

Hot Network Questions