JSON empty array

Question

I am trying to parse some JSON that is returned from a REST web service. The return from the get() call is a TStringStream. I'm using dbxjson to work with the data. To make things easier to demonstrate here, I've created a test project that reproduces the error without calling the web service (uses a text file for the web service output instead). Here's the code:

var SL : TStringStream;
  LJsonObj : TJSONObject;
begin
  SL := TStringStream.Create;
  try
    SL.LoadFromFile('output.txt');
    LJsonObj := TJSONObject.ParseJSONValue(TEncoding.ASCII.GetBytes(SL.DataString), 0) as TJSONObject;
  finally
    SL.Free;
  end;
end;

Sometimes the phone_numbers array in this JSON data is empty. In the stream object coming from the web service call, it looks like this:

{
    "Contact Information Service": {
        "response": {
            "phone_numbers": [

]
        }
    }
}

This causes ParseJSONValue to return a nil value.

However, if I change the empty phone_numbers array to this in my test txt file:

{
    "Contact Information Service": {
        "response": {
            "phone_numbers": []
        }
    }
}

it works fine (i.e. returns a TJSONObject). The difference being the whitespace in the empty array. For some reason the first JSON response with whitespace in the empty array causes ParseJSONValue to return nil. It works fine with no whitespace between the square braces.

What am I doing wrong with my JSON parsing? Is there some sort of pre-parsing I need to do before calling ParseJSONValue?

Looks like it's most probably a bug in the implementation of TJSONByteReader, but frankly trying to make sense of the parsing code makes it impossible to tell at a glance. Empirically the evidence is quite clear. Fortunately my own TJSONObject reader handles such cases perfectly well. Time to publish maybe ? :) — Deltics
– Deltics, Commented Aug 29, 2012 at 0:24
@Deltics: Really? I traced through it while investigating this question, and I didn't find it that difficult to understand the parsing code. I think the parser is badly written--this issue would have been avoided completely if it had a proper lexer instead of mixing the lexing in with the parsing--but it's not too hard to understand what's going on... — Mason Wheeler
– Mason Wheeler, Commented Aug 29, 2012 at 2:54
If you found PeekByte() easy to understand then you must dream in hex. :) When I publish my JSON code you'll see the difference between what I consider readable (dare I say maintainable) code and, um, dbxJSON. — Deltics
– Deltics, Commented Aug 29, 2012 at 20:47
Since I wasn't solving a problem that I was experiencing, I gave up when I came across PeekByte(), as a clear "Here be dragons" red flag. So I never actually got as far as ParseArray() which is, as you mention below, quite clear and obviously flawed. (what are the Parse..() methods even doing in TJSONObject anyway?) ;) — Deltics
– Deltics, Commented Aug 29, 2012 at 20:55
@Deltics: You mean you don't dream in hex? I've just gotten used to it, I guess. I don't even see the code anymore. All I see is blonde, brunette, redhead... ;) — Mason Wheeler
– Mason Wheeler, Commented Aug 29, 2012 at 21:19

RRUZ · Accepted Answer · 2012-08-29 01:51:03Z

This issue is not exclusive of the Delphi JSON implementation (DBXJSON), I worked with some JSON PHP parsers with the same limitation.

Now because all the blanks outside a double quoted strings literals are (and must be) ignored by the JSON parsers, you can remove these white spaces safely, So A possible workaround is Minify your Json string, before to parse it.

Try this sample, which uses regular expressions to remove the extra whitespaces from a string.

{$APPTYPE CONSOLE}

{$R *.res}


uses
  System.RegularExpressions,
  System.Classes,
  System.SysUtils,
  Data.DBXJSON;

const
JsonString=
'{'+
'    "Contact Information Service": {'+
'        "response": {'+
'            "phone_numbers": [        ]'+
'        }'+
'    }'+
'}';

function JsonMinify(const S: string): string;
begin
 Result:=TRegEx.Replace(S,'("(?:[^"\\]|\\.)*")|\s+', '$1');
end;

procedure TestJSon;
var
  s : string;
  SL : TStringStream;
  LJsonObj : TJSONObject;
begin
  SL := TStringStream.Create;
  try
    s:=JsonMinify(JsonString);
    SL.WriteString(s);
    LJsonObj := TJSONObject.ParseJSONValue(TEncoding.ASCII.GetBytes(SL.DataString), 0) as TJSONObject;
    Writeln(LJsonObj.Size);
  finally
    SL.Free;
  end;
end;

begin
 try
    TestJSon;
 except
    on E:Exception do
        Writeln(E.Classname, ':', E.Message);
 end;
 Writeln('Press Enter to exit');
 Readln;
end.

Mason Wheeler · Accepted Answer · 2012-08-29 00:05:25Z

7

Have a look at TJsonObject.ParseArray. You'll find this:

while ValueExpected or (Br.PeekByte <> Ord(']')) do
begin
  ConsumeWhitespaces(Br);
  Pos := ParseValue(Br, JsonArray);
  if Pos <= 0 then
    Exit(Pos);

So at the top of the array (immediately after it reads the open bracket), if the next character is not a close bracket, eat whitespace and then try to read a valid JSON value. A close bracket is not a valid JSON value, so it bails out at this point.

This does appear to be valid JSON, (I can get my browser to accept it as a valid JavaScript object), so this ought to be considered a bug in the DBXJSON library. You might need to pre-parse this, use a different JSON library (there are a handful for Delphi) or find a way to ensure that the information being sent to you does not contain this pattern.

Either way, you ought to report this to QC as a bug.

answered Aug 29, 2012 at 0:05

Mason Wheeler

85k54 gold badges288 silver badges510 bronze badges

1 Comment

Deltics Over a year ago

But you won't get it fixed in XE2 even though it is a clear and simple failure to correctly implement the JSON specification - a feature you already paid for in XE2. You forgot to mention that part.

Collectives™ on Stack Overflow

JSON empty array

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related