mongo db aggregation/mapreduce

Question

i am trying to extract uniq keys + all uniq values from an object of my mongo-documents.

lets say i have document structure like:

{
    "userId": "1234",
    "formFields": {
        "field1": "value1",
        "field2": "value2"
    }
},
{
    "userId": "1234",
    "formFields": {
        "field3": "value3",
        "field1": "value1-edited"
    }
},
{
    "userId": "1234",
    "formFields": {
        "field3": "value3",
        "field1": "value1-edited"
    }
}

i want to aggregate all documents from user "1234" to get the distinct values of "formFields"

result should look something like this:

{
    "_id": "1234",
    "formFields": {
        "field1": [
            "value1",
            "value1-edited"
        ],
        "field2": [
            "value2"
        ],
        "field3": [
            "field3"
        ],
        "field4": [
            "field4"
        ]
    }
}

they keys in the formFields are dynamic. i tried arround with aggregate, and mapreduce but haven't found any working sample that can be used as a basis.

anyone can answer this?

THANKS

regards helmut

i just tried to find a good sample to start off one was some of them are here: gist.github.com/hjanuschka/6fe70ca46c8b681a48d8 actually i am on point zero - just started using mongodb - and i have no clue where to go with this requirement. — Helmut Januschka
– Helmut Januschka, Commented Oct 12, 2015 at 14:18

chridam · Accepted Answer · 2015-10-12 15:10:35Z

1

Since you have dynamic keys that you don't know beforehand, you would need to get a list of those fields so that you can use them in your aggregation pipeline. One way to get that list is through mapReduce. The following demonstrates this approach. In the Map-Reduce operation, an array of keys in the formFields subdocument is generated to an output collection "uploads_keys" and then used to produce the aggregation pipeline expressions:

var mr = db.runCommand({
    "mapreduce" : "uploads",
    "map" : function() {
        for (var key in this.formFields) { emit(key, null); }
    },
    "reduce" : function(key, stuff) { 
        return null  
    }, 
    "out": "uploads" + "_keys"
});

var groupKeys = {},
    projectKeys = { "formFields": {} };
db[mr.result].distinct("_id").forEach(function (key){    
    groupKeys[key] = { "$push": "$formFields." + key };
    projectKeys["formFields"][key] = "$" + key
});

groupKeys["_id"] = "$userId";

> printjson(groupKeys);
{
        "field1" : {
                "$push" : "$formFields.field1"
        },
        "field2" : {
                "$push" : "$formFields.field2"
        },
        "field3" : {
                "$push" : "$formFields.field3"
        },
        "_id" : "$userId"
}
> printjson(projectKeys);
{
        "formFields" : {
                "field1" : "$field1",
                "field2" : "$field2",
                "field3" : "$field3"
        }
}

You can then use those variables in your aggregation pipeline:

db.uploads.aggregate([
    { "$group": groupKeys },
    { "$project": projectKeys }
])

edited Oct 12, 2015 at 15:10

answered Oct 12, 2015 at 14:23

chridam

104k26 gold badges246 silver badges243 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Helmut Januschka Over a year ago

ok this is actually a good starting point - does what i want it to do! BUT: they keys are hardcoded - is there a variation of this piece with dynamic keys? - and $push does not seem to be uniqe

chridam Over a year ago

I've updated my answer with an approach that you could take to generate the list of dynamic keys.

Helmut Januschka Over a year ago

this is like 99.9% the solution i am looking for - many thx - any chance to kick out duplicate values?

chridam Over a year ago

Yes, it's quite possible indeed. Just use $addToSet instead of the $push operator within the $group pipeline. Also, it would be very helpful if you could read What should I do when someone answers my question?

Helmut Januschka Over a year ago

THX - addToSet did it! now its a 100% perfect - many thx, and i already confirmed your question, just wanted to wait if i get a reply for the uniqe question.

Collectives™ on Stack Overflow

mongo db aggregation/mapreduce

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related