Python string-bytes to bytes

Question

I have string data that look like bytes reprs of JSON in Python

>>> data = """b'{"a": 1, "b": 2}\n'"""

So on the inside of that, we have valid JSON that looks like it's been byte-encoded. I want to decode the bytes and loads the JSON on the inside, but since its a string I cannot.

>>> data.decode() # nope
AttributeError: 'str' object has no attribute 'decode'

Encoding the string doesn't seem to help either:

>>> data.encode() # wrong
b'b\'{"a": 1, "b": 2}\n\''

There are oodles of string-to-bytes questions on stackoverflow, but for the life of me I cannot find anything about this particular issue. Does anyone know how this can be accomplished?

Things that I do not want to do and/or will not work:

eval the data into a bytes object
strip the b and \n (inside of my JSON there's all sorts of other escaped data).

This is the only working solution I have found, and there is a lot not to like about it:

from ast import literal_eval

data = """b'{"a": 1, "b": 2}\n'"""
print(literal_eval(data[:-2] + data[-1:]).decode('utf-8'))

"I have string data that look like bytes reprs of JSON in Python" - that sounds like a bug you should fix on the producing end. — user2357112
– user2357112, Commented Dec 16, 2020 at 19:14
I wish i could solve it on that end! These are actually airflow logs vis structlog that i must analyze — Nolan Conaway
– Nolan Conaway, Commented Dec 16, 2020 at 19:18
Anyway, ast.literal_eval. There's probably a good dupe target somewhere around here. — user2357112
– user2357112, Commented Dec 16, 2020 at 19:27
The weird slicing you had to do in your literal_eval attempt is almost certainly due to a bug you introduced while attempting to write a string literal for data - you've got an actual newline in the middle of your bytes literal, which is invalid syntax for a bytes literal. You probably meant for that to be an actual backslash and n - that, or the newline was supposed to be outside the bytes literal. — user2357112
– user2357112, Commented Dec 16, 2020 at 19:39
data = r"""b'{"a": 1, "b": 2}\n'""" is likely more representative of the actual kinds of values you're working with. If it's not, then that's going to be an issue. — user2357112
– user2357112, Commented Dec 16, 2020 at 19:41

Ryan · Accepted Answer · 2020-12-17 21:11:51Z

2

I know you said you didn't want to strip the b inside the string due to other escaped data, but can't we assume that whatever generated this only output ascii (hence the b), and we can re-encode that. So I was thinking you can use a simple regexp (https://regex101.com/r/M0ratk/1) which you then encode as bytes.

import json
import re

match = re.match(r"\Ab'(.*)'\Z", data, re.DOTALL)
data = json.loads(bytes(match[1], 'ascii'))

Will this work? I am not sure how it compares to the literal_eval solution.

answered Dec 17, 2020 at 21:11

Ryan

2,2231 gold badge22 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Nolan Conaway Over a year ago

Never in my wildest dreams did i picture a regex solution wow wow

Ryan Over a year ago

What can I say, when you spend a good portion of every work day writing regex, everything starts looking like a regex problem 🤪

Mark Ransom Over a year ago

@NolanConaway and now you have two problems. (It's a famous quote, look it up)

Collectives™ on Stack Overflow

Python string-bytes to bytes

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related