MongoDB mapreduce missing data with 'null' in return

Question

So this is strange. I'm trying to use mapreduce to group datetime/metrics under a unique port:

Document layout:

{
        "_id" : ObjectId("5069d68700a2934015000000"),
        "port_name" : "CL1-A",
        "metric" : "340.0",
        "port_number" : "0",
        "datetime" : ISODate("2012-09-30T13:44:00Z"),
        "array_serial" : "12345"
}

and mapreduce functions:

var query = {
        'array_serial' : array,
        'port_name' : { $in : ports },
        'datetime' : { $gte : from, $lte : to}

    }

    var map = function() {
        emit( { portname : this.port_name } , { datetime : this.datetime,
                                metric : this.metric });
    }

    var reduce = function(key, values) {
        var res = { dates : [], metrics : [], count : 0}

        values.forEach(function(value){
            res.dates.push(value.datetime);
            res.metrics.push(value.metric);
            res.count++;
        })

        return res;
    }

    var command = {
        mapreduce : collection,
        map : map.toString(),
        reduce : reduce.toString(),
        query : query,
        out : { inline : 1 }

    }

    mongoose.connection.db.executeDbCommand(command, function(err, dbres){
        if(err) throw err;
        console.log(dbres.documents);
        res.json(dbres.documents[0].results);
    })

If a small number of records is requested, say 5 or 10, or even 60 I get all the data back I'm expecting. Larger queries return truncated values....

I just did some more testing and it seems like it's limiting the record output to 100? This is minutely data and when I run a query for a 24 hour period I would expect 1440 records back... I just ran it a received 80. :\

Is this expected? I'm not specifying a limit anywhere I can tell...

More data:

Query for records from 2012-10-01T23:00 - 2012-10-02T00:39 (100 minutes) returns correctly:

[
  {
    "_id": {
      "portname": "CL1-A"
    },
    "value": {
      "dates": [
        "2012-10-01T23:00:00.000Z",
        "2012-10-01T23:01:00.000Z",
        "2012-10-01T23:02:00.000Z",
         ...cut...
        "2012-10-02T00:37:00.000Z",
        "2012-10-02T00:38:00.000Z",
        "2012-10-02T00:39:00.000Z"
      ],
      "metrics": [
        "1596.0",
        "1562.0",
        "1445.0",
        ...cut...
        "774.0",
        "493.0",
        "342.0"
      ],
      "count": 100
    }
  }
]

...add one more minute to the query 2012-10-01T23:00 - 2012-10-02T00:39 (101 minutes) :

[
  {
    "_id": {
      "portname": "CL1-A"
    },
    "value": {
      "dates": [
        null,
        "2012-10-02T00:40:00.000Z"
      ],
      "metrics": [
        null,
        "487.0"
      ],
      "count": 2
    }
  }
]

the dbres.documents object shows the correct expected emitted records:

[ { results: [ [Object] ],
    timeMillis: 8,
    counts: { input: 101, emit: 101, reduce: 2, output: 1 },
    ok: 1 } ]

...so is the data getting lost somewhere?

Asya Kamsky · Accepted Answer · 2012-10-06 05:52:46Z

Rule number one of MapReduce:

Thou shall return from Reduce the exact same format that you emit with your key in Map.

Rule number two of MapReduce:

Thou shall reduce the array of values passed to reduce as many times as necessary. Reduce function may be called many times.

You've broken both of those rules in your implementation of reduce.

Your Map function is emitting key, value pairs.

key: port name (you should simply emit the name as the key, not a document)
value: a document representing three things you need to accumulate (date, metric, count)

Try this instead:

map = function() {  // if you want to reduce to an array you have to emit arrays
    emit ( this.port_name, { dates : [this.datetime], metrics : [this.metric], count: 1 });
}

reduce = function(key, values) {        // for each key you get an array of values
   var res = { dates: [], metrics: [], count: 0 };  // you must reduce them to one

   values.forEach(function(value) {
            res.dates = value.dates.concat(res.dates);
            res.metrics = value.metrics.concat(res.metrics);
            res.count += value.count;   // VERY IMPORTANT reduce result may be re-reduced
        }) 

        return res;
    }

vikas · Accepted Answer · 2012-10-06 04:12:27Z

1

Try to output the map reduce data in a temp collection instead of in memory. May that is the reason. From Mongo Docs:

{ inline : 1} - With this option, no collection will be created, and the whole map-reduce operation will happen in RAM. Also, the results of the map-reduce will be returned within the result object. Note that this option is possible only when the result set fits within the 16MB limit of a single document. In v2.0, this is your only available option on a replica set secondary.

Also, It may not be the reason but MongoDB has data size limitations (2GB) on a 32bit machine.

answered Oct 6, 2012 at 4:12

vikas

1,5581 gold badge14 silver badges22 bronze badges

4 Comments

Chris Matta Over a year ago

Good suggestion, I just tried it, and am still getting truncated data. The truncated collections have "null" as the first value of the metric and dates arrays, I'm wondering if that's a clue.

Chris Matta Over a year ago

Oh, and the output of a query with 4 ports is only 6k, so I'm not really getting close to the 16mb limit.

vikas Over a year ago

oh, then can you please paste the mongodump of data on pastebin or something like that.

Asya Kamsky Over a year ago

it's definitely a clue - see my answer - your map and reduce formats are inconsistent. Data limits don't have anything to do with your problem.

Collectives™ on Stack Overflow

MongoDB mapreduce missing data with 'null' in return

2 Answers 2

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related