Bulk insert into mongodb in javascript mvc project

Question

I need to contribute to a project (which is related to my master's degree) developed by someone else, and I couldn't reach the actual developer. I am new to MongoDB. In the project, he inserted data into MongoDB by insertmany function. However, now the data I am trying to insert is larger and gave me the error below. I specified related code scopes. How can I bulkinsert? I tried to splice data into smaller arrays, but then I got callback already called error. Should I bulkinsert or splice the data, and which way should I do?

Thanks.

Error:

2020-05-02T19:50:20+0200 <info> admin.js:521 () Error occured..on bulk pool saving... { MongoError: BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"
    at Function.MongoError.create (C:\TopicBinder2-master\node_modules\mongodb-core\lib\error.js:31:11)
    at C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\pool.js:497:72
    at authenticateStragglers (C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\pool.js:443:16)
    at Connection.messageHandler (C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\pool.js:477:5)
    at Socket.<anonymous> (C:\TopicBinder2-master\node_modules\mongodb-core\lib\connection\connection.js:331:22)
    at Socket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
  name: 'MongoError',
  message:
   'BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"',
  operationTime:
   Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1588441812 },
  ok: 0,
  errmsg:
   'BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"',
  code: 10334,
  codeName: 'Location10334',
  '$clusterTime':
   { clusterTime:
      Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1588441812 },
     signature: { hash: [Binary], keyId: [Long] } } }
2020-05-02T19:50:20+0200 <error> admin.js:400 () Error creating pool { message: 'Error occured on bulk saving!' }



var Pools = (module.exports = mongoose.model("pools", poolSchema));
module.exports.createPoolItems = function (poolItems, callback) {
  populateUniqueId(poolItems);

  Pools.collection.insertMany(poolItems, { ordered: false }, callback);
};

function populateUniqueId(poolItems) {
  if (poolItems.length > 0) {
    poolItems.forEach((element) => {
      element.unique_id =
        element.project + "_" + element.topic_id + "_" + element.document_id;
    });
  }
}

Mohammad Oftadeh · Accepted Answer · 2020-05-02 19:29:17Z

0

This error is related to the fact that your document is more than 16 MB. And that's a limit on the MongoDB. You see more details: MongoDB Limits and Thresholds

...
 errmsg:
   'BSONObj size: 16972498 (0x102FAD2) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "pools"',
...

You can find data model examples and patterns from link below:

MongoDB Data Modeling

edited May 2, 2020 at 19:29

answered May 2, 2020 at 19:17

Mohammad Oftadeh

1,4451 gold badge7 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user8737957 Over a year ago

And this would be the doc docs.mongodb.com/manual/reference/limits

Collectives™ on Stack Overflow

Bulk insert into mongodb in javascript mvc project

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related