JSON string (encoded from large array) got truncated when output to HTTP client

Question

I met an issue that the JSON string output to HTTP client was truncated near to close tag.

An example,

a. expect to see - ... ... "last_update":"2014-06-10 20:46:38","garden_id":"1"}],"message":null}

b. actually it is - ... ..."last_update":"2014-06-10 20:46:38","garden_id":"1"}],"message":nul

The last two characaters were truncated for some reason!!!

I tried both Postman on Chrome and curl on a console, all the same output. So looks not a browser specific issue. The JSON string in my application comes from a PHP json_encode on an associative array. The PHP code is using CodeIgniter framework running on Apache. I tried to write the json string into a file before http output, the file content is 100% correct. So it is not a json encoding issue in PHP.

The PHP is pretty straightforward. I have build below array (namely $finaldata) from database query

{
    "success": true,
    "data": [
        {
            "file_id": "1",
            "title": "xxx",
            "create_date": "2014-05-18 21:30:19",
            "auditor": "1",
            "status": "1",
            "last_updater": null,
            "last_update": "2014-06-10 20:43:14",
            "garden_id": "1"
        },
        {
            "file_id": "2",
            "title": "yyy",
            "create_date": "2014-05-18 21:30:19",
            "auditor": "1",
            "status": "1",
            "last_updater": null,
            "last_update": "2014-06-10 20:43:14",
            "garden_id": "1"
        }
    ],
    "message": null
}

the "data" has a sub array and it could be a long array depends on database records. Then the variable $finaldata is passed to a output function that has below general logic:

header('Content-Type: '.$this->_supported_formats[$this->response->format]);
$output = $this->format->factory($finaldata)->{'to_'.$this->response->format}();
header('Content-Length: ' . strlen($output));

The output function is built by a RESTful library from https://github.com/philsturgeon/codeigniter-restserver. In this context, it equals to

header('Content-Type: Application/json');
$output = json_encode($finaldata);

But I found an interesting thing that the issue only happened if the string length exceeds 8K. And when I append a random string like "ZZZ" to the JSON string before HTTP response, the issue was gone. I don't know the reason behind. It is not a correct hack because I could not prove 8K is a real threshold though.

Has anyone met such issue before? Any suggestion or comments are appreciated.

TiMESPLiNTER · Accepted Answer · 2014-06-20 09:52:07Z

1

I think you send the data in UTF-8 charset so try to change this lines

header('Content-Type: '.$this->_supported_formats[$this->response->format]);
header('Content-Length: ' . strlen($output));

into this one

header('Content-Type: '.$this->_supported_formats[$this->response->format] . '; charset=utf-8');
header('Content-Length: ' . mb_strlen($output));

If this doesn't work just don't send the Content-Length header at all. This header is "only useful" if you send a file to the browser (as a download).

edited Jun 20, 2014 at 9:52

answered Jun 19, 2014 at 13:34

TiMESPLiNTER

5,9193 gold badges31 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

navykoo Over a year ago

just tried, doesn't work. My hack now is, if(strlen($output)>8000){ $output = $output."ZZZ"; }

TiMESPLiNTER Over a year ago

Try to delete the Content-Length header statement and remove your workaround and have a look if the string still gets truncated.

navykoo Over a year ago

wow, it seems work. It tells me the strlen() function is returning an incorrect size, right?

navykoo Over a year ago

Is it possible due to my mbstring extension is not configured well? It is now configured in php.ini like this - [mbstring] mbstring.language = all mbstring.internal_encoding = UTF-8 mbstring.http_input = auto mbstring.http_output = UTF-8 mbstring.encoding_translation = On mbstring.detect_order = UTF-8 mbstring.substitute_character = none; mbstring.func_overload = 0 mbstring.strict_encoding = Off

TiMESPLiNTER Over a year ago

@navykoo Yep there seems to be a problem with strlen() and mb_strlen(). Can say if your mbstring now is configured right. Does it work with this settings you posted in the comment above? But if it works just don't send the Content-Length header at all.

|

Danh · Accepted Answer · 2016-12-02 03:39:35Z

0

In my issue, I enable gzip in nginx,and append application/json to the field of gzip_types. Then, I set a header when make a request with Guzzle. It looks like

    if (! isset($parameters['headers'])) {
        $parameters['headers']['Accept-Encoding'] = 'gzip';
    }

edited Dec 2, 2016 at 3:39

Danh

6,0217 gold badges34 silver badges45 bronze badges

answered Dec 2, 2016 at 3:20

user7239338

1

Collectives™ on Stack Overflow

JSON string (encoded from large array) got truncated when output to HTTP client

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related