NodeJS handling file upload with streaming and asynchronous validation

Question

I am writing an expressJS web app that handles file upload in chunks. The receivable file contains data that I need to process and validate on the fly.

I am handling it like below:

module.exports = function(req, res, next) {
  logger.info("file converter executing...");

  req.on('data',function(chunk){

    // Read a line from the chunk
    var line = readALine(req, chunk);
    while ( (line = readALine(req, chunk)) !== null) {

      // ignore comment lines and blank lines...
      if (line.indexOf("#") === 0 || line.length === 0) continue;  

      result = processLine(req, line);
    }
  });

  /** EOF: request stream -- move processing to next layer.  **/
  req.on('end', function() {
    console.log("handing control to next middle ware...");
    return next();
  });

};

In the processLine() function, I have to validate each line of incoming data. But the problem is that validation can be asynchronous, because I am using trying to check against some values which are originally in DB, but lazy-loaded to Redis Cache or program's internal cache. So, the validation is asynchronous.

So, while I am processing a chunk of data, I do not want to receive any further events from the stream, until I have finished the validation of current chunk.

My questions:

Am I right in assuming that the pause(), resume() API will fit here? So, I can pause the stream, until the promise of async validation finishes, and then resume it? Is it safe?
Assuming that the first query is right, if my validation fails, plan to return res.send(some error). Will it properly clear all the remaining chunks in the request stream, or should I do anything more?

Thanks for any help...

Sahil · Accepted Answer · 2016-07-16 10:06:41Z

1

module.exports = function(req, res, next) {
  logger.info("file converter executing...");
  req.pause();
  req.on('data',function(chunk){

    // Read a line from the chunk
    var line = readALine(req, chunk);
    while ( (line = readALine(req, chunk)) !== null) {

      // ignore comment lines and blank lines...
      if (line.indexOf("#") === 0 || line.length === 0) continue;  
      //take a callback to know when validation is complete
      result = processLine(req, line, function(err, resp){
        if(err) return next(err);
        req.resume();     
});
    }
  });

  /** EOF: request stream -- move processing to next layer.  **/
  req.on('end', function() {
    console.log("handing control to next middle ware...");
    return next();
  });

};

answered Jul 16, 2016 at 10:06

Sahil

755 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Mopparthy Ravindranath Over a year ago

Good answer. After exploring more, I think I should settle for a pull based read, instead of 'data' event. This way, I can consume only what I want, instead of having data "pushed" at me.

Collectives™ on Stack Overflow

NodeJS handling file upload with streaming and asynchronous validation

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related