Concurrent request handling in Node.js and MongoDB

Question

I have a Node.js server which has to handle 6 concurrent requests. Each request reads a document from MonogDB database and then modifies it.

So, when we send 6 requests concurrently they all read the same data item and modify it but I want that value modified by the first request should be read by the second request and then it should be able to modify it further and so on.

For example, I have a data item called, x with initial value 'a'. The first request modifies it to 'ab'. So, the second request now should read 'ab' and not 'a'. But it always reads 'a' because they are coming concurrently.

I have tried to apply semaphore, mutex, async.series, async.waterfall, sleep and promises but nothing worked. Probably I am not using them correctly. So, how should they be used to handle these concurrent requests in a synchronous manner.

Or is there any other way of doing it? Can we do something with timestamp ordering protocol? If yes, then how can I apply it in Node.js?

This is the code that I want to run for these concurrent requests-

app.get('/urltest', function (req, res) {
    var q=require('url').parse(req.url,true).query;
    var s=q.s;

    MongoClient.connect(url, function(err, db) {
        if(err)
            throw err;
         db.collection( 'Comp' ).find({no:2}).snapshot().forEach(
            function (elem) {

                db.collection( 'Comp' ).findAndModify (
                    { no : elem.no },
                    [['no', 1]],
                    { $set : { age:elem.age+s } }, {new:true},
                    function( err, result ) {
                        if ( err ) console.log( err);

                        console.log("after update: "+result.value.age);
                        res.end("success");
                    }
                ); 
            });
    });
});

Here I am sending requests like - http://192.168.197.140:8081/urltest?s=b, http://192.168.197.140:8081/urltest?s=c, http://192.168.197.140:8081/urltest?s=d, etc. concurrently.

Now, they can run in any order. But I want all 'b', 'c', 'd' etc to be appended. And thus the output for each should look somewhat like - 'ab', 'abc', 'abcd' and so on.

But the output I get is- 'ab', 'ac', 'ad'.

How exactly should I use promise to make sure these requests execute in a synchronous manner?

If 6 requests arrive at about the same time, how would you know which one was supposed to go first, second, third, etc...? If you want help with some specific code solutions you've tried, then you have to show us the code you tried and explain what happened when you tried it. It's not hard to make one operation wait for the completion of another, but we'd have to see what exactly you're trying to do (see actual code and understand actual algorithm) to know how to code it for you. — jfriend00
– jfriend00, Commented Mar 21, 2017 at 6:27
You could also create an access queue. If nobody is in the queue for access to a particular data item, then it just gets that item. If an operation is already underway, then the next request gets in the queue and will be serviced when the prior one is done. Promises can be used to do this very easily. But, to know what code to suggest, we'd need to see specific code for what you're trying to do and understand how you know which requests need to serialize like this. — jfriend00
– jfriend00, Commented Mar 21, 2017 at 6:31
@jfriend00 I have updated the question and included the basic code that I want to run synchronously. I have tried using Promises but they are also not working. For access queue I tried storing these incoming requests in an array and then accessing it one by one but that also doesn't work. So, can you please elaborate on how should I exactly code to achieve my result? — sowmyaa guptaa
– sowmyaa guptaa, Commented Mar 21, 2017 at 9:13

twg · Accepted Answer · 2017-03-21 04:47:46Z

2

It sounds like a complex system. My suggestion would be to take a look at the new Mongo Node driver. Then take a look at the ECMAScript6 documentation. This involves installing the co module it works with. Use co as you would use promises (take a look at the module). It supplies you with yield which returns a promise.

It also sounds like you will need to aggregate (if I am not mistaken.) For a then ab then abc and so on.

The concurrency problem should be handled obviously by waiting for the yield to return. The examples in the new documentation are not great but decent enough for you to get a working example out of them.

answered Mar 21, 2017 at 4:47

twg

1,1058 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

sowmyaa guptaa · Accepted Answer · 2017-03-21 11:32:20Z

So, all I had to do was to implement semaphore correctly. This is the final version of the code that worked-

var sem = require('semaphore')(1);      
app.get('/urltest', function (req, res) {

    sem.take(function() {

        var q=require('url').parse(req.url,true).query;
        var s=q.s;                                  

        MongoClient.connect(url, function(err, db) {
            if(err)
                throw err;
            db.collection( 'Comp' ).find({no:2}).snapshot().forEach(
                function (elem) {                           

                    db.collection( 'Comp' ).findAndModify (
                        { no : elem.no },
                        [['no', 1]],
                        { $set : { x:elem.x+s } }, {new:true},          
                        function( err, result ) {
                            if ( err ) console.log( err);

                            console.log("after update: "+result.value.x);
                            res.end("success");

                            sem.leave();    

                        }
                    ); 
                });
        });
    })
});

bluehipy · Accepted Answer · 2017-03-21 09:32:20Z

1

You need to implement a semaphore.

Read about it here

In your case you have to serve one request at a time though they come in "all together". So

var sem = require('semaphore')(1);
var server = require('http').createServer(req, res) {
    sem.take(function() {
      // find, update and commit data before leaving the semaphore
      expensive_database_operation(function(err, res) {
        sem.leave();

        if (err) return res.end("Error");

        return res.end(res);
      });
    });
});

edited Mar 21, 2017 at 9:32

answered Mar 21, 2017 at 9:15

bluehipy

2,2941 gold badge15 silver badges19 bronze badges

13 Comments

sowmyaa guptaa Over a year ago

Done that. Node.js provides a semaphore module using npm install semaphore but that didn't work either. I reffered this link- npmjs.com/package/semaphore. Could you please elaborate on how else semaphore can be implemented in node.js?

bluehipy Over a year ago

Show me how you implemented the semaphore, please.

sowmyaa guptaa Over a year ago

var sem = require('semaphore')(1); 	sem.take(function() { 		var q=require('url').parse(req.url,true).query; 		var s=q.s; 		MongoClient.connect(url, function(err, db) { 			if(err) throw err; 			db.collection( 'Comp' ).find({no:2}).snapshot().forEach( 				function (elem) { 					db.collection( 'Comp' ).findAndModify ( 						{ no : elem.no }, [['no', 1]], { $set : { age:elem.age+s } }, {new:true}, function( err, result ) { 							if ( err ) console.log( err); 							console.log("after update: "+result.value.age);  res.end("success"); 								sem.leave();});  	}); 			}); 		})

sowmyaa guptaa Over a year ago

Done it in a similar manner only. Just added my MongoDB query in place of "expensive_database_operation" and did sem.leave() in its callback.

bluehipy Over a year ago

Did you check in your logs after update if the record is correctly updated? Because that is what semaphore is doing. It makes all the other requests wait until the semaphore was left.

|

Collectives™ on Stack Overflow

Concurrent request handling in Node.js and MongoDB

3 Answers 3

Comments

Comments

13 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

13 Comments

Your Answer

Sign up or log in

Post as a guest

Related