How to save a large array to MongoDB using mongoose in Node.js

Question

I'm wanting to save a very large array (over 40k strings) in MongoDB.

const allWords = require("an-array-of-english-words");
const { patterns } = require("./models/pattern");
const mongoose = require("mongoose");

// Create a model for the Words object
const Words = mongoose.model(
  "Words",
  new mongoose.Schema({
    words: {
      type: Array,
      required: true,
    },
  })
);

/*
  Filters the list of words to use only words greater than 4 and less than 6
*/
const array = allWords.filter(function (text) {
  return text.length >= 4 && text.length <= 6;
});

let words = [...array];
for (let i = 0; i < array.length; i++) {
  for (let j = 0; j < patterns.length; j++) {
    let result = patterns[j].test(array[i]);
    if (result) {
      let index = array.indexOf(array[i]);
      array.splice(index, 1);
    }
  }
}

async function saveWords(words) {
  console.log("start");
  const array = new Words({ words });
  console.log("mid");
  console.log(array);

  //it's successfully making the array object but it's having trouble saving it

  await array.save();
  console.log("done");
}

saveWords(words);

console.log("array length: " + array.length + " " + allWords.length);

Everything works up until the call to save the array, then a time out error is logged on the console. This is my first project working on my with Node.js and I'm sure I'm making a small easily fixable mistake but I'm just not sure what it is.

Why do you want to keep so much data in one array? You should design it better as there will be performance issues. — Apoorva Chikara
– Apoorva Chikara, Commented Dec 20, 2021 at 16:16
I'm open to any feedback you can give me to improve the design and flow of my program. What would you suggest? — palesc
– palesc, Commented Dec 20, 2021 at 16:24
What do you want to achieve? What makes you save such large data? How will you consume it or update it if needed? — Apoorva Chikara
– Apoorva Chikara, Commented Dec 20, 2021 at 16:26
So, I'm making a password generator but I'm wanting to use whole words rather than a random string (no particular reason just thought it would be fun to do it that way). The reason there's so much data is because I'm using an npm package called 'an-array-of-english-words' which contains over 275k words and 40k is what I have after filtering the list. I'm still new to coding lingo but I'm assuming "consume" means how I will use the data. I will basically be choosing a word at random in the array and concatenating it to a string adding a special character and number in between each word. — palesc
– palesc, Commented Dec 20, 2021 at 16:36

Apoorva Chikara · Accepted Answer · 2021-12-21 05:08:56Z

1

I would say you don't need to save these words in the DB instead create an algorithm that creates a random string first, then you can add randomness to the created string if you want like adding numbers or special characters.

You could use a third-party npm package. You can check this.

Also, it would be good if you look at how others achieve it. This will be a good practice to improve on how you can implement it efficiently. This can be a good package.

answered Dec 21, 2021 at 5:08

Apoorva Chikara

8,8033 gold badges24 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to save a large array to MongoDB using mongoose in Node.js

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related