Store huge Array of JSON data in MongoDB via Mongoose

Question

Let's say I have an JavaScript Array companyBills and inside of it I have m objects stored in such manner; with m > 100.000 elements.

{key_0:value_0, key_1:value_0_1, key_n:value_0_n},
{key_0:value_1, key_1:value_1_1, key_n:value_1_n},
{key_0:value_m, key_1:value_m_1, key_n:value_m_n}

Now I want to store this in MongoDB via Mongoose, in such a manner:

CompanyBills.create (companyBills, function(err, jellybean, snickers) {
        if(!err) {          
          console.log('Success');
        } else {
          console.log('Error');
        }

However the array companyBills is too huge. Thus I want to store in chunks of 1/100 m. How to split the array companyBills Time-efficiently and store in MongoDB?

I am creating the array with https://github.com/SheetJS/js-xlsx, maybe there is an opportunity to hook-in early.

Can you post the code of array creation? If you don't mind the order of inserted data you can divide array into chunks and use async library for put each chunk into the db in parallel — Vsevolod Goloviznin
– Vsevolod Goloviznin, Commented Feb 24, 2015 at 15:33
There is indeed a solution to this particular scenario: github.com/SheetJS/js-xlsx/issues/138 However I would be very interested in the async library for parallel access. — Cloudscape Germany
– Cloudscape Germany, Commented Feb 24, 2015 at 15:47

sctskw · Accepted Answer · 2015-02-24 17:44:04Z

2

Have you tried the bulk insert method?

CompanyBills.collection.insert([huge array], function (err, results) {
    Console.log(results)
})

http://docs.mongodb.org/manual/tutorial/insert-documents/

edited Feb 24, 2015 at 17:44

answered Feb 24, 2015 at 16:00

sctskw

1,6081 gold badge13 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Cloudscape Germany Over a year ago

Looks like that is the cuprit, there is no bulk write method in Mongoose as of now; see github.com/LearnBoost/mongoose/issues/2366, so the create method I used is very slow, inserting each element on its own. Thus I have to use the underlying MongoDB driver.

sctskw Over a year ago

Its very easy to access to driver directly by using CompanyBill.collection.insert(). Edited above

Cloudscape Germany Over a year ago

Confirmed this would be a solution, if not for a new bug rendering the native driver bulk insert option useless - stackoverflow.com/questions/24466366/… Maybe you can confirm?

Collectives™ on Stack Overflow

Store huge Array of JSON data in MongoDB via Mongoose

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related