Strange characters in (invalid) json string from post request (encoding issues)

Question

I am trying to get data from a post request using the following line :

$data = file_get_contents('php://input');

The json string might be like: {"test" : "test one \xe0 "}

The problem is when I try to do a json_decode($data), I get null. By var_dump()ing $data, I see some characters like \xe0 \xe7a.

The data sent is in utf-8. I use utf8_decode($data) as well, but with no luck. Could someone explain what I am missing or how to solve this issue?

I need to convert the invalid json from:

$data = '{"test" : "test one \xe0 "}';

to:

$data = '{"test" : "test one à "}';

If the JSON error is fairly consistent when json decode fails, fallback to string manipulation to replace /xe0 for example and then attempt to decode again. But doesn't make sense to send JSON if you are going to send invalid JSON. — Jacob
– Jacob, Commented Dec 19, 2023 at 20:18
You might be able to use a third party library to parse the incoming JSON differently than json_decode — Jacob
– Jacob, Commented Dec 19, 2023 at 20:23

Olivier · Accepted Answer · 2023-12-19 21:05:47Z

2

A way to fix your JSON is to replace the invalid \xNN sequences with valid \u00NN sequences:

$data = '{"test" : "test one \xe0 "}';
$val = json_decode(str_replace('\x', '\u00', $data));
echo $val->test;

Output:

test one à

answered Dec 19, 2023 at 21:05

Olivier

19.6k1 gold badge12 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

KIKO Software Over a year ago

This does work, but be careful, it won't work when there's a two or three byte hexadecimal code in the original JSON representing a multi-byte character.

mickmackusa · Accepted Answer · 2023-12-19 23:10:58Z

1

Mutating a json string with string functions will always be something to be done with apprehension because it is generally easy for a false positive replacement to damage the payload. That said, here is a script to attempt to correct your invalid json string.

Code: (Demo)

$json = '{"test" : "test one \xe0, \x270B"}';
    
$json = preg_replace_callback(
           '/\\\\x([[:xdigit:]]+)/',
           fn($m) => sprintf('\u%04s', $m[1]),
           $json
     );
     
echo "\n" . var_export(json_validate($json), true);
echo "\n$json\n";
var_export(json_decode($json));

Output:

true
{"test" : "test one \u00e0, \u270B"}
(object) array(
   'test' => 'test one à, ✋',
)

If this has known flaws, please leave a comment below and I'll endeavor to overcome the issue when I have time.

A related answer of mine: Replace all hex sequences with ascii characters

answered Dec 19, 2023 at 23:10

mickmackusa♦

49.2k13 gold badges98 silver badges165 bronze badges

2 Comments

Álvaro González Over a year ago

This works fine as long as the escaped entity was meant to be the Unicode code-point. If it's meant to be some random undisclosed impossible-to-autodetect single-byte encoding, you'll need to add some mb_convert_encoding() to the mix and do a lot of guessing. Of course, the main point is clear: why use a standard in the first place if you aren't going to comply with it?

mickmackusa Over a year ago

Is there a reason that this answer received a no-comment dv and the other answer didn't? Is this just a user popularity contest?

Collectives™ on Stack Overflow

Strange characters in (invalid) json string from post request (encoding issues)

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related