Mongoose - Search inside nested array with no limits

Question

My mongodb collection is stores records of groups which can have subgroups, which can further have sub-sub groups and there will not be any limits of nesting of this sub-groups. Is there anyway in mongoose to find a group name that resides in nth level?

My collection structure looks like the following, so suppose if i want to find if "My Group" has a subgroup named "My Group 1001", how am I supposed to find it?

{
      "id": "60c18d4ce973fc139f23bf93",
      "name": "My Group",
      "subgroups": [
        {
              "id": "70c18d4ce973fc139f23bf93",
              "name": "My Group 2",
              "subgroups": [
                {
                      "id": "80c18d4ce973fc139f23bf93",
                      "name": "My Group 3",
                      "subgroups": [
                          {
                                "id": "90c18d4ce973fc139f23bf93",
                                "name": "My Group 4",
                                "subgroups": [ ... ],
                          }

                      ]
                }

              ]
        }

      ]
}

The other solutions that I found on stackoverflow suggested that it can be achieved using the dot notation for eg,

$or: [
  {"name": "My Group 1001"},
  {"subgroups.name": "My Group 1001"},
  {"subgroups.subgroups.name": "My Group 1001"},
  ...
]

But since in my case the level is not known hence I cannot use the above pattern.

You supposed to design your schema to support your queries. This one does not. Do what the other suggested - build your $or array programmatically with reasonable level of nested documents. There is a hard limit of 16MB per document so there is no truly "unlimited" nested arrays. — Alex Blex
– Alex Blex, Commented Jun 14, 2021 at 10:22
16mb is a huge number to store texual data. When I say "unlimited", i mean there will be an unknown number of nesting, perhaps 10, 20, 50, 500. And even if there are 500 level of nesting, it will still be under 16 mb because the content is all text. — Phantom007
– Phantom007, Commented Jun 15, 2021 at 3:59
Fair enough. 500 $or conditions would be quite extreme query. I have put some recommendations in my answer. — Alex Blex
– Alex Blex, Commented Jun 15, 2021 at 8:31
If you were to design a schema that has such a requirement, how would you design it? I am not sure if my schema design is efficient hence need your inputs. — Phantom007
– Phantom007, Commented Jun 16, 2021 at 4:15
I already gave my recommendations regarding schema changes in my answer starting from words "If you have luxury of changing data format". A more radical change could be one of described in docs.mongodb.com/manual/applications/… Please read the small print, there are limitations to keep in mind. And of course there is a whole world of specialised graph databases which were purposeful designed to store tree-like data structures if you fill adventurous. — Alex Blex
– Alex Blex, Commented Jun 16, 2021 at 7:41

Alex Blex · Accepted Answer · 2021-06-15 08:30:17Z

Mongodb is not an ideal tool for recursive operations.

Not recommended, but you can use $function to do the job with javascript:

db.collection.find({
  $expr: {
    $function: {
      body: "function(subgroups, topName) { let recursiveSearch = function(sg){ if(!sg) {return false}; return sg.some(function(subdoc) {return subdoc.name == 'My Group 1001' || recursiveSearch(subdoc.subgroups)})}; return topName == 'My Group 1001' || recursiveSearch(subgroups);}",
      args: [
        "$subgroups",
        "$name"
      ],
      lang: "js"
    }
  }
})

The function alone formatted for readability:

function(subgroups, topName) {
    let recursiveSearch = function (sg) {
        if (!sg) { return false };
        return sg.some(function (subdoc) {
            return subdoc.name == 'My Group 1001' || recursiveSearch(subdoc.subgroups)
        })
    }; 
    return topName == 'My Group 1001' || recursiveSearch(subgroups);
}

There is no tail recursion so I imagine it may blow up if it exhaust call stack. Didn't test it myself though.

If you have luxury of changing data format, consider adding a top level "names" flat array of all names from the nested documents. It will add some overhead to the application logic to keep it up to date on each update. The search will be as simple as

db.collection.find({"names": "My Group 1001"})

As an additional bonus you will be able to benefit from multikey indexes

Murat Colyaran · Accepted Answer · 2021-06-14 11:35:00Z

0

Maybe finding keys with values can help you:

const flattenObj = (obj, parent, res = {}) => {
  for (let key in obj) {
    let propName = parent ? parent + '.' + key : key;
    if (typeof obj[key] == 'object') {
      flattenObj(obj[key], propName, res);
    } else {
      res[propName] = obj[key];
    }
  }
  return res;
}


const obj = {
  "id": "60c18d4ce973fc139f23bf93",
  "name": "My Group",
  "subgroups": [
    {
      "id": "70c18d4ce973fc139f23bf93",
      "name": "My Group 2",
      "subgroups": [
        {
          "id": "80c18d4ce973fc139f23bf93",
          "name": "My Group 3",
          "subgroups": [
            {
              "id": "90c18d4ce973fc139f23bf93",
              "name": "My Group 4"
            }

          ]
        }

      ]
    }
  ]
}

const dotNotation = flattenObj(obj);

// Finding keys from object by a specific value
const key1 = Object.keys(dotNotation).find(key => dotNotation[key] === 'My Group 2');
const key2 = Object.keys(dotNotation).find(key => dotNotation[key] === 'My Group 3');
const key3 = Object.keys(dotNotation).find(key => dotNotation[key] === 'My Group 4');

console.log(`Key of 'My Group 2': ` + key1);
console.log(`Key of 'My Group 3': ` + key2);
console.log(`Key of 'My Group 4': ` + key3);

answered Jun 14, 2021 at 11:35

Murat Colyaran

2,2072 gold badges13 silver badges34 bronze badges

3 Comments

Phantom007 Over a year ago

Does mongodb/mongoose put a limit how many data is populated per page or does it impose a limit on depth of nesting?

Murat Colyaran Over a year ago

You can use $project to limit returning data. Use my answer to get key names and unselect the fields you don't want to get.

Phantom007 Over a year ago

I understand the $project part, but my original question was does mongodb, by default, put any limit to nesting depth and number of pages shown at once? Meaning, if i have 100000 documents in my user collection, and if i run User.findAll({}), will it return 100000 documents at once?

Collectives™ on Stack Overflow

Mongoose - Search inside nested array with no limits

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related