PHP | json_decode huge json file

One alternative here is to use the salsify/jsonstreamingparser

You need to create your own Listener.

$testfile = '/path/to/file.json';
$listener = new MyListener();
$stream = fopen($testfile, 'r');
try {
    $parser = new \JsonStreamingParser\Parser($stream, $listener);
    $parser->parse();
    fclose($stream);
} catch (Exception $e) {
    fclose($stream);
    throw $e;
}

To make things simply to understand, I"m using this json for example:

JSON Input

{
    "objects": [
    {
        "propertyInt": 1,
        "propertyString": "string",
        "propertyObject": { "key": "value" }            
    },
    {
        "propertyInt": 2,
        "propertyString": "string2",
        "propertyObject": { "key": "value2" }
    }]
}

You need to implement your own listener. In this case, I just want to get the objects inside array.

PHP

class MyListener extends \JsonStreamingParser\Listener\InMemoryListener
{
    //control variable that allow us to know if is a child or parent object
    protected $level = 0;

    protected function startComplexValue($type)
    {
        //start complex value, increment our level
        $this->level++;
        parent::startComplexValue($type);
    }
    protected function endComplexValue()
    {
        //end complex value, decrement our level
        $this->level--;
        $obj = array_pop($this->stack);
        // If the value stack is now empty, we're done parsing the document, so we can
        // move the result into place so that getJson() can return it. Otherwise, we
        // associate the value
        if (empty($this->stack)) {
            $this->result = $obj['value'];
        } else {
            if($obj['type'] == 'object') {
                //insert value to top object, author listener way
                $this->insertValue($obj['value']);
                //HERE I call the custom function to do what I want
                $this->insertObj($obj);
            }
        }
    }

    //custom function to do whatever
    protected function insertObj($obj)
    {
        //parent object
        if($this->level <= 2) {
          echo "<pre>";
          var_dump($obj);
          echo "</pre>";
        }
    }
}

Output

array(2) {
  ["type"]=>
  string(6) "object"
  ["value"]=>
  array(3) {
    ["propertyInt"]=>
    int(1)
    ["propertyString"]=>
    string(6) "string"
    ["propertyObject"]=>
    array(1) {
      ["key"]=>
      string(5) "value"
    }
  }
}
array(2) {
  ["type"]=>
  string(6) "object"
  ["value"]=>
  array(3) {
    ["propertyInt"]=>
    int(2)
    ["propertyString"]=>
    string(7) "string2"
    ["propertyObject"]=>
    array(1) {
      ["key"]=>
      string(6) "value2"
    }
  }
}

I tested it against a JSON file with 166MB and it works. Maybe you need to adapt the listener to your needs.

Not a reply intended for the op but as an alternative method to anyone else looking into this topic...

You CAN use json_decode() on ANY size file with next to no memory use. Yepp, the best of both worlds. I tried several solutions such as jsonmachine and json_decode as they were designed where some methods were fast digesting the entire file at once with a memory crash while others completed but were painfully slow.

My solution is to break apart the json file into smaller sections and process each with json_decode(). I did this by setting the head and the end of the json file to variables (or constants), then concatenating head + body excerpt + end and processing each batch separately, where the body excerpt was 200-400 records but can be anything the system can handle. I am sure some people will have something negative to say about this but in essence it would be the same as manually making many small json files and processing them individually. This method simply does it for you, relatively fast and can handle a file of literally any size.

My sample file had 1,177,437 records (3.8GB) that involved several operations to prepare the data such as many coordinate conversions, string manipulations, sql queries to retrieve additional data to be included and gz_deflate(). It created sql statements that were queried and completed in 37 min with no errors averaging 530 sql records created per second. The table ended up being 5.2 GB when said and done. If you know that the file(s) will be formatted 100% correctly this can be sped up by reading an entire line opposed to 1 character at a time. I opted for 1 character at a time because on occasion I get geojson files with no line breaks and I designed it for maximum compatibility first, speed second.

Tips: I found that preg_match() worked well to extract the head of the file while simply looking for an equal quantity of opening and closing curly brackets within a string indicated a complete record. The end of the file was a simple "\n]\n}\n" that I hard coded because it is common to all files.