PHP memory exhausted ( json_decode)

Question

When my application tries to decodes large (15K~ rows) JSON string (comes from CURL), it fails with:

Allowed memory size of 134217728 bytes exhausted (tried to allocate 91 bytes)

I know I can expand the memory limit or unlimit it, however I'd rather avoid that. I have been wondering whether there is a different approach to address that kind of issue - such splitting the JSON string into small chunks (like array_chunk).

UPDATE

Just to make sure that the issue is not caused by any other function / loop in the app, I've extract the JSON string into a file and tried to decode it directly from the file (file size = 11.8MB). still fails.

$y = json_decode( file_get_contents('/var/tmp/test.txt') );

UPDATE 2 The script runs on Mac OS X environment. I've tested it also on Ubunto env (also 128M memory limit) - and there it works perfectly. should i be concern?

Just how big is this JSON string? Because 128Mb is a LOT of memory... — Niet the Dark Absol
– Niet the Dark Absol, Commented Jul 17, 2014 at 13:21
those 15k rows how much data is in each row (about how much)? would help there to decide how to best help you there. Also is there anything else taking place like the part where the allowed memory is passed being part of a loop,....? (it could be that even though the json is where the error happens that it is not the reason if a memory leak is happening at an ealier instance in a loop for example) — Thomas
– Thomas, Commented Jul 17, 2014 at 13:22
@MarcinOrlowski Whoops, indeed. Then again, 128Mb (16MB) is still a LOT of RAM to be using in a single PHP script. — Niet the Dark Absol
– Niet the Dark Absol, Commented Jul 17, 2014 at 13:25

humbads · Accepted Answer · 2017-03-24 14:20:47Z

4

To permanently avoid this, use an event-based JSON parser like https://github.com/salsify/jsonstreamingparser.

That way, the whole thing doesn't have to be in memory at once. Instead, you process the events which give you one piece of the object/array at a time.

answered Mar 24, 2017 at 14:20

humbads

3,4521 gold badge29 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Marcin Nabiałek · Accepted Answer · 2014-07-17 13:28:37Z

2

There are no other PHP functions that let you decode JSON string. You can try on your own or find library to split JSON into parts.

However you should make sure that it's the only problem. For example before decoding JSON data you maybe created big arrays or create many objects.

If I were you, I would save this json string to file and write extra script just to get it from file and decode to make sure that using json_decode makes the only problem.

answered Jul 17, 2014 at 13:28

Marcin Nabiałek

112k43 gold badges271 silver badges299 bronze badges

Comments

Filip Halaxa · Accepted Answer · 2020-02-26 14:19:54Z

1

One of the simplest ways to iterate big json file in php is to use halaxa/json-machine. You only write one foreach. It will never hit memory limit, because it parses one item a at a time behind the scenes, so the memory consumption is constant no matter the file size.

answered Feb 26, 2020 at 14:19

Filip Halaxa

7786 silver badges14 bronze badges

Collectives™ on Stack Overflow

PHP memory exhausted ( json_decode)

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related