Uploading large file to NodeJS webserver

Question

I am serving up my web application using NodeJS server (ExpressJS) currently. One of the new requirements is for the users to be able to upload large videos (potentially in gigs) to the server. They will later then be able to download them again. Could I achieve this with the current stack?

I came across tutorials using Multer JS with "multipart/form-data" specified by the front-end. If I use this, would my Express be able to serve up other HTTP requests while writing a giant single video file to the server's disk?

major-mann · Accepted Answer · 2016-11-21 15:43:17Z

5

Having done this myself (~20GB files, without multer) I can recommend the following (Probably most of which you have considered, but for completeness):

Choose an appropriate upload control (or write your own, but basically something that chunks up the data is best. I used plupload I think)
On the server make an API to handle the received chunk data, and write it out to a temp location during upload (You may want to hash and check individual pieces as well).
Once upload complete, check file (I used a hash generated by the client, and checked it against the uploaded data)

answered Nov 21, 2016 at 15:43

major-mann

2,6621 gold badge26 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Randy Over a year ago

Just curious, is there a reason you decided not to use multer?

major-mann Over a year ago

I did it 4 / 5 years ago... don't think the project even existed (When I did it we were still doing a lot of HTML5 shimming with flash :O)

Randy · Accepted Answer · 2016-11-21 15:36:13Z

4

Multer should work fine. It'll stream the data as it comes in over HTTP to a file on disk. It won't lock up your server. Then in your endpoint you have access to that file location and can do whatever you need with it.

answered Nov 21, 2016 at 15:36

Randy

4,4112 gold badges27 silver badges47 bronze badges

8 Comments

Randy Over a year ago

On a side note: Are you then just uploading that file to S3? If you are, you can actually have the client upload the file directly to S3. This will save your Node server from having to process GB's of data. And S3 can likely serve your files back to whomever faster than your own servers can.

jiminssy Over a year ago

Uploading to S3 seems like a good solution. Seeing as you are recommending it, it is safe to do this in terms of security I guess? Surely you wouldn't want anybody to start uploading files to your S3 so is there some kind of authentication scheme?

Randy Over a year ago

The client asks your API for an access token to S3. Your API basically creates this token using the S3 libraries Amazon provides. This is a temporary token that your client will then use to connect to S3 to upload a file. Once uploaded you can call back to your API with either the path or the identifier. Then your API can use the S3 file however it pleases.

Hosar Over a year ago

There is a multer s3 plugin github.com/badunk/multer-s3 maybe can be useful

pagep Over a year ago

I am working with multer but it always loads the file into the memory first. Does not stream the data to the disk, any ideas?

|

Arun · Accepted Answer · 2017-12-12 09:24:35Z

0

    var express =   require("express");

    var app =   express();
    var async = require('async');
    var fs = require('fs');
    var client = redis.createClient();
    var multiparty = require('multiparty');
    var util = require('util');

    app.post('/',function(req,res){

        var form = new multiparty.Form();

        form.parse(req, function(err, fields, files) {

             client.get(fields.otp, function(err, reply) {
    var oldpath = files.file[0].path;
                          var newpath = path + files.file[0].originalFilename;
                          fs.rename(oldpath, newpath, function (err) {
                            if (err) throw err;
                            res.write('File uploaded and moved!');
                            res.end();
                          });
    }
    }



app.listen(2001,function(){
    console.log("Server is running on port 2001");
});

answered Dec 12, 2017 at 9:24

Arun

1,12511 silver badges13 bronze badges

1 Comment

zipzit Over a year ago

I appreciate the code, but its not clear to me on how multiparty differs from multer. Its also not clear at all why you are using the redis client stuff. No client in the multiparty example. Hint: it's okay to add comments to make this stuff helpful and clear to others. Why is this recommendation better (or equal) to the checked answer above?

Collectives™ on Stack Overflow

Uploading large file to NodeJS webserver

3 Answers 3

2 Comments

8 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

8 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related