MongoDB: Aggregate Data Using Map/Reduce

Question

I still don't fully understand how map/reduce works, so I thought I'd give an example of a problem I need to solve, and hopefully the answer will help me understand the concept.

I'm tracking page views using a document structure similar to this:

{
    "timestamp" : 1299990045,
    "visitor" : {
        "region" : {
            "country_code" : "US",
        },
        "browser" : {
            "name" : "IE",
            "version" : "8.0",
        }
    },
    "referer" : {
        "host" : "www.google.com",
        "path" : "/",
        "query" : "q=map%2Freduce"
    }
}

I store a single document for each page view. Because I get about 15 million page views a day, I'd like to aggregate these results each night, save the aggregate results for that day, and then drop the collection to begin storing page views again. I want the output of the map/reduce to look like this:

{
    "day" : "Sun Mar 13 2011 00:00:00 GMT-0400 (EDT)",
    "regions" : {
        "US" : 235,
        "CA" : 212,
        "JP" : 121
    },
    "browsers" : {
        "IE" : 145,
        "Firefox" : 245,
        "Chrome" : 95,
        "Other" : 120
    },
    "referers" : {
        "www.google.com" : 24,
        "yahoo.com" 56
    }
}

I really don't know where to begin doing this kind of thing. Any help would be appreciated.

jared · Accepted Answer · 2011-03-13 19:34:00Z

10

The typical process for writing a map-reduce job is to start with the data format you want as the output of your reduce, build a map function that outputs it, then a reduce function that adds them up. In your example, you'd do something like this:

function map() { 
    var date = new Date( this.timestamp.getFullYear(), 
                         this.timestamp.getMonth(),
                         this.timestamp.getDay() );
    var out  = { regions: {}, browsers: {}, referers: {} };
    out.regions[ this.visitor.region.country_code ] = 1; 
    out.browsers[ this.visitor.browser.name ] = 1;
    out.referers[ this.referer.host ] = 1; 
    emit( date, out);
}

function reduce( key, values ) { 
    var out = { regions: {}, browsers: {}, referers: {} };
    values.forEach(function(value) { 
      for( var region in value.regions ) { 
        if( out.regions[region] ) { 
          out.regions[ region ] += value[region];
        } else { 
          out.regions[ region ] = value[region];
        }
      };
      for( var browser in value.browsers ) { 
        if( out.browsers[browser] ) { 
          out.browsers[ browser ] += value[browser]; 
        } else { 
          out.browsers[ browser ] = value[browser]; 
        }
      };
      for( var referer in value.referers ) { 
        if( out.referers[ referer] ) { 
          out.referers[ referer ] += value[referer];
        } else {  
          out.referers[ referer ] = value[referer];
        } 
      }
    });
    return out; 
}

At the end of this, you should have an output collection that looks something like this:

{ 
  _id: "Sun Mar 13 2011 12:23:58 GMT-0700 (PDT)", 
  value: { 
    regions:  {
      "US" : 235,
      "CA" : 212,
      "JP" : 121
    },
    browsers: { 
      "IE" : 145,
      "Firefox" : 245,
      "Chrome" : 95,
      "Other" : 120 
    }, 
    referers: { 
      "www.google.com" : 24,
      "yahoo.com" 56
    } 
  } 
}

Note that there's another way you can do this.. Rather than doing a map reduce job, you can also keep all of this data in real time using atomic increments and upserts.

For example, each time you generate one of your page view docs, you could also do an update like this:

db.pageviews.summaries.update( { _id: new Date( this.timestamp.getFullYear(), 
                                                this.timestamp.getMonth(),
                                                this.timestamp.getDay() ) },
                               { $inc : {
                                     'visitor.region.US' : 1,
                                     'visitor.browser.IE' : 1,
                                     'referer.www.google.com' : 1 
                                  }
                               },
                               true // upsert
                             );

This would mean that you always have your summary document up to date and don't need any map reduce jobs.

Note, you might want to escape the '.' in your domain names as Mongo will interpret that as a hierarchy of documents rather than an attribute name.

answered Mar 13, 2011 at 19:34

jared

1,64714 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

mellowsoon Over a year ago

You're almost certainly right about upserts as I go along. Doing a map/reduce operation on 15 million documents would take FOREVER. Thank you for the map/reduce example though. The concept makes more sense now. Map = build key value pair for each document, reduce = aggregate all those key value pairs.

mellowsoon Over a year ago

Extra note, I think I'll need to look into a finalize function. After all, I wouldn't need every country that visited on a particular day. Only the top 20. Is that where and how a finalize function is used?

jared Over a year ago

Well, you can do this in a finalize(), but it would be somewhat inefficient because you'd be doing a bunch of aggregations that you were just going to throw away when it's done. If there's a way to skip processing the unwanted countries in the first place, that would be preferable. But in general, yes you can do this kind of filtering in a finalize.

mellowsoon Over a year ago

Because I'd be saving the top 20 countries, there's no way to know which ones are unwanted until after the aggregation.

Collectives™ on Stack Overflow

MongoDB: Aggregate Data Using Map/Reduce

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related