Analysis of a large json log file in node.js

So you have that big fat sensorslog of 16GB wrapped in json.

To start, an entire json file of 16GB isn't realistic, simply because the opening and closing brackets breaks the regularity and turns into just annoying characters in an array. We know the file has a beginning and an end, and moreover, without them your program can work on chunks of the file or even on a stream directly plugged to the device. So let's assume what we will be processing is this :

{"UTCTime":10000001,"s1":22,"s2":32,"s3":42,"s4":12},
{"UTCTime":10000002,"s1":23,"s2":33,"s4":13},
{"UTCTime":10000003,"s1":24,"s2":34,"s3":43,"s4":14},
...
{"UTCTime":12345678,"s1":57,"s2":35,"s3":77,"s4":99},

and even adding or detecting that missing comma at the end shouldn't be too difficult.

Now every line is formatted the same way, and can be interpreted as json. The problem is : do the sensors outputted data when expected to ? If we are sure they speak at the right time and at the right frequency (case 1) but that sometimes they could miss a writing, all is well. However if they start to make slight slip over the time frame (case 2), then some kind of heuristic to recover the proper line-line frequency is needed and the analysis will be longer.

If we are not processing this realtime, a first and easy validation check for the file is to tell if on every freq line is found the expected sensors data, right ?

In any case, since it's a very big file it has to be processed line by line whenever possible.

In the following program, I considered only case 1, and that we could be processing a continuous stream.

#!/usr/bin/python
import json

sensors={}
sensors['s1']=[1] # frequencies
sensors['s2']=[1]
sensors['s3']=[2]
sensors['s4']=[1]

# append data array and error counter at sensors[i]
# it holds [freq,err,data]
for k,v in sensors.iteritems(): sensors[k].extend([0,[]])
FRQ=0;ERR=1;DAT=2
print list(sorted(sensors.items()))
S=list(sorted(sensors.keys()))

with open('./sensors.json', "r") as stream:
    i=0
    for line in stream:
      if not line.rstrip(): continue # skip blank lines
      j=json.loads(line[:-2]) # skip comma and \n
      t=j["UTCTime"]
      for k in S:
          sensor=sensors[k]
          if i%sensor[FRQ]==0 : # every Nth iteration
            v=j.get(k)
            if v is None:
                sensor[ERR]+=1
                print k,"has",sensor[ERR],"missing entries"
            sensor[DAT].append([t,v]) # append that sensor data
            # filling up the memory...
      i+=1

for k,v in sorted(sensors.iteritems()): print k,sensors[k][DAT]
for k,v in sorted(sensors.iteritems()): print k,'had',sensors[k][ERR],"missing entries"

To handle case 2 we would invert the None check with the modulus check, verify if a sensor wrote something when it wasn't supposed to and then try to detect shifts.

Last note : your program could get short on memory, so perhaps keeping the entire data in memory isn't a good idea. If it's intended to use separate arrays for each sensors to do further processing, it might be wiser to write them to files.

Edited again to consider your edits:

var fs = require('fs');
var stream = fs.createReadStream('sensorlogs.json', {flags: 'r', encoding: 'utf-8'});
var buffer = '';
var sensor = process.argv[2];
var readings = [];
var missingCont = 0;
console.log('Analizying ' + sensor + ':');

stream.on('data', function(d) {
    buffer += d.toString();
    processBuffer();
    console.log(readings);
    console.log(sensor + ' had ' + missingCont + ' missing entries');
});

function processBuffer() {
  buffer = buffer.slice(buffer.indexOf('[{'));
  while(buffer.indexOf('{') != -1) {
    buffer = buffer.slice(buffer.indexOf('{"'));
    processLine(buffer.slice(0, buffer.indexOf('}') + 1));
    buffer = buffer.slice(buffer.indexOf('}') + 2);
  }
};

function processLine(line) {
  if(line != ""){
    var obj = JSON.parse(line);
    if(!obj[sensor]){
      missingCont++;
    }else{
      var pos;
      for(pos = 0; pos < readings.length; pos++){
        if(obj.UTCTime < readings[pos][0]){
          var reading = [obj.UTCTime, obj[sensor]]
          readings.splice(pos, 0, reading);
          break;
        }
      }
      if(pos == readings.length){
        readings.push([obj.UTCTime, obj[sensor]]);
      }
    }
  }
};

You would have to call it with a param of which sensor you want to analyze:

node.exe scripts\processJson.js <param>

To test it out I took this sample:

{"arr":[{"UTCTime":10000001,"s1":22,"s2":32,"s3":42,"s4":12},
{"UTCTime":10000005,"s1":20,"s2":30,"s3":40,"s4":10},
{"UTCTime":10000002,"s1":23,"s2":33,"s4":13},
{"UTCTime":10000003,"s1":24,"s2":34,"s3":43,"s4":14},
{"UTCTime":12345678,"s1":57,"s2":35,"s3":77,"s4":99}
]}

And the output was:

> node.exe scripts\processJson.js s1
Analizying s1:
[[10000001, 22], [10000002, 23], [10000003, 24], [10000005, 20], [12345678, 57]]
s1 had 0 missing entries

> node.exe scripts\processJson.js s2
Analizying s2:
[[10000001, 32], [10000002, 33], [10000003, 34], [10000005, 30], [12345678, 35]]
s2 had 0 missing entries

> node.exe scripts\processJson.js s3
Analizying s3:
[[10000001, 42], [10000003, 43], [10000005, 40], [12345678, 77]]
s3 had 1 missing entries

> node.exe scripts\processJson.js s4
Analizying s4:
[[10000001, 12], [10000002, 13], [10000003, 14], [10000005, 10], [12345678, 99]]
s4 had 0 missing entries