Import text files (CSV format) into mongodb using mongoose schema

Question

I am new to moogoose/mongodb and I'm trying to import data. I have a REGIONS.model.js file ...

var mongoose = require('mongoose');
var Schema = mongoose.Schema;

var REGIONS = new Schema({
    CODE: String,
    DESCRIPTION_FRANCAISE: String,
    DESCRIPTION_ANGLAISE: String
});

module.exports = mongoose.model('REGIONS', REGIONS);

and a REGIONS.TXT file ...

"01","Bas-Saint-Laurent","Bas-Saint-Laurent"
"02","Saguenay/Lac-Saint-Jean","Saguenay/Lac-Saint-Jean"
"03","Capitale-Nationale","Capitale-Nationale"
"04","Mauricie","Mauricie"
"05","Estrie","Estrie"
"06","Montréal","Montréal"
"07","Outaouais","Outaouais"
"08","Abitibi-Témiscamingue","Abitibi-Témiscamingue"
"09","Côte-Nord","Côte-Nord"
"10","Nord-du-Québec","Nord-du-Québec"
"11","Gaspésie/Iles-de-la-Madeleine","Gaspésie/Iles-de-la-Madeleine"
"12","Chaudière-Appalaches","Chaudière-Appalaches"
"13","Laval","Laval"
"14","Lanaudière","Lanaudière"
"15","Laurentides","Laurentides"
"16","Montérégie","Montérégie"
"17","Centre-du-Québec","Centre-du-Québec"
"50","Autres provinces","Other provinces"
"70","Etats-Unis","USA"
"80","Autres pays","Other countries"

and I would like to import the REGIONS.TXT's data into a mongodb db document that respects the REGIONS Schema.

In fact I will have multiple TXT files that I will need to import daily and that will need to match the corresponding Schema. (There are no headers in the *.TXT files)

*UPDATED to explain why it's not a duplicate,

I do not wish to hardcode the list of the Schema's keys and use them in a hardcoded function that aggregates these with their values since I have lot of Schemas and corresponding data files (*.TXT).

I guess I would need to "scan" the Schema (which has the same name as the TXT file) and dynamically "build" the function. Or maybe there's something built in mongoose?

Many thanks for your time and help.

Brian Shamblen · Accepted Answer · 2015-05-27 02:20:07Z

Unfortunately, since the order of keys within a schema (object) are not guaranteed to be in the same order as the file you'll need to keep a list of fields in an array somewhere. I'm assuming you know the format of each file (the fields and their order) and the corresponding model that each file should be imported into.

Create a new module with the following code. I'm naming it csv.js:

var mongoose = require('mongoose')
  , csv = require('fast-csv');

module.exports.importFile = function(filePath, fileHeaders, modelName) {
    csv
        .fromPath(filePath, {headers: fileHeaders})
        .on('data', function(data) {

            var Obj = mongoose.model(modelName);

            var obj = new Obj();

            Object.keys(data).forEach(function(key) {
                var val = data[key];

                if (val !== '')
                    obj.set(key, val);
            });

            obj.save(function (err) {
                if (err)
                    console.log(err);
            });
        })
        .on('end', function() {
            console.log("done");
        });
}

You'll need to create an object somewhere with the properties of your import files (name, headers, and model name).

var csv = require('./csv');

var csvHeaders = {
    REGIONS: {
        headers: ['CODE', 'DESCRIPTION_FRANCAISE', 'DESCRIPTION_ANGLAISE']
    },
    STATES: {
        headers: ['something']
    }
}

//adjust this path to the correct location
csv.importFile(__dirname + '/REGION.CSV', csvHeaders.REGIONS.headers, 'REGIONS');

Collectives™ on Stack Overflow

Import text files (CSV format) into mongodb using mongoose schema

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related