How does Mongo DB handle a large array field?

Question

I'm trying to store a list of ObjectIds in a document as an array field.

I understand Mongo DB has a 4MB size limit for single documents. So considering the length of ObjectId is 12 bytes, a document should be able to handle more than 300,000 entries in one array field. (Let me know if the calculation is off).

If the number of entries in the array gets close to that limit, what kind of performance can I expect? Especially when the field is indexed? Any memory issues?

Typical queries would look like below:

Query by a single value

db.myCollection.find(
  {
    myObjectIds: ObjectId('47cc67093475061e3d95369d')
  }
);

Query by multiple values

db.myCollection.find(
  {
    myObjectIds: {$in: [ObjectId('47cc67093475061e3d95369d'), ...]}
  }
);

Add a new value to multiple documents

db.myCollection.update(
  {
    _id: {$in: [ObjectId('56cc67093475061e3d95369d'), ...]}
  },
  {
    $addToSet: {myObjectIds: ObjectId('69cc67093475061e3d95369d')}
  }
);

FYI: In 1.8, the max doc size has been increased to 16MB, and this limit is a soft limit imposed by 10gen. Someone on SO put it best: (paraphrasing) The limit was put in place to force us to think a little harder about our schema design. — Bryan Migliorisi
– Bryan Migliorisi, Commented Mar 15, 2011 at 13:59
@Bryan Migliorisi Thanks, I have come across that post and agree with being smarter about schema design. — Jaepil
– Jaepil, Commented Mar 15, 2011 at 23:47

AdaTheDev · Accepted Answer · 2011-03-15 10:53:22Z

5

TBH, I think the best thing you can do is to benchmark it. Create some dummy data, and test the performance as you increase the number of items in the array. It may be quicker to knock up a test in your environment - than wait for an answer here

It is one thing on my TODO list to investigate and blog about, but I haven't got round to it yet. If you do, I'd definitely be interested to see what your findings are! Likewise, if I get round to it soon I will post the results here too.

answered Mar 15, 2011 at 10:53

AdaTheDev

148k28 gold badges215 silver badges204 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jaepil Over a year ago

Wasn't planning on doing a benchmark. But if I do so, I will share my finding here. Thanks.

Josh Liptzin · Accepted Answer · 2013-05-05 21:02:23Z

5

With the release of mongo 2.4 you can use capped arrays. On insert, you can tell mongo to $sort and $slice the array to keep it to a fixed length based on your criteria (if you don't care about throwing data away). For example, you could use this to save the most recent N entries in a data log.

answered May 5, 2013 at 21:02

Josh Liptzin

7945 silver badges13 bronze badges

Comments

Theo · Accepted Answer · 2011-03-15 14:04:57Z

2

You won't notice when you hit the document size limit unless you use getLastError after each update. The update will fail, and a message is logged to the database log. I have anecdotal evidence from my local ops guy that Mongo seems to be working harder when there are a lot of updates that fail because of the document size being reached.

I know of no easy way of avoiding it, other than designing around it. As far as I know there is no way to conditionally push to a list. I've seen other questions here on SO where people have been trying to build fixed size lists and such, but no good solutions have been found.

answered Mar 15, 2011 at 14:04

Theo

133k22 gold badges170 silver badges213 bronze badges

1 Comment

Jaepil Over a year ago

Thanks, I didn't really think about what would happen when the limit is reached. Clearly I need to pay more attention to it.

Collectives™ on Stack Overflow

How does Mongo DB handle a large array field?

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related