MongoDB oplog stream Node.js

Question

I have a local instance of MongoDB, started with --master, and I can see in the start log that I have a replica set oplog (file: /data/db/local.1) created. And it appears local.1 is updated every time I perform and insert().

I am trying to tail / stream the oplog with the following code...

MongoDB.connect('mongodb://localhost:27017/test',(e,db) => {
          if ( e ) { return console.log('Connect MongoDB Error',e); }
          console.log('Connected MongoDB'); 
          Server.DB = db;  
              db.collection('oplog.rs').find({}).toArray((e,d) => {
                console.log('oplog.rs');
                console.log(e);
                console.log(d);
              })  

          var updates = db.collection('oplog.rs').find({},{ 
                              tailable: true, 
                              awaitData: true, 
                              noCursorTimeout: true, 
                              oplogReplay: true, 
                              numberOfRetries: Number.MAX_VALUE 
                              }).stream();
                              
              updates.on('data',(data) => { console.log('data',data); });
              updates.on('error',(e) =>   { console.log('error',e); });
              updates.on('end',(end) =>   { console.log('end',end); });
        });

The Code logs the following on start...

oplog.rs
null
[]

However there is no output from stream, I set numberOfRetries to a lower value I get the error..

error { MongoError: No more documents in tailed cursor
    at Function.MongoError.create (/Users/peter/node_modules/mongodb-core/lib/error.js:31:11)
    at nextFunction (/Users/peter/node_modules/mongodb-core/lib/cursor.js:637:50)
    at /Users/peter/node_modules/mongodb-core/lib/cursor.js:588:7
    at queryCallback (/Users/peter/node_modules/mongodb-core/lib/cursor.js:211:18)
    at Callbacks.emit (/Users/peter/node_modules/mongodb-core/lib/topologies/server.js:116:3)
    at Connection.messageHandler (/Users/peter/node_modules/mongodb-core/lib/topologies/server.js:282:23)
    at Socket.<anonymous> (/Users/peter/node_modules/mongodb-core/lib/connection/connection.js:273:22)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:189:7)
    at readableAddChunk (_stream_readable.js:176:18)
  name: 'MongoError',
  message: 'No more documents in tailed cursor',
  tailable: true,
  awaitData: true }
end undefined

Alex Blex · Accepted Answer · 2017-04-18 10:47:05Z

1

oplog.rs is used in replica sets. Starting mongodb with --master you are using long-deprecated master-slave replication, which uses local.oplog.$main database, so there is nothings in oplog.rs.

You need to start mongod with --replSet to benefit from replica set oplog. Read more how to deploy and configure replica sets.

answered Apr 18, 2017 at 10:47

Alex Blex

37.3k7 gold badges53 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

user3094755 Over a year ago

I have now connected to an Atlas cluster, and still oplog.rs is empty and tail() results in the same message. (tried local.oplog.rs / oplog.rs / oplog).

Alex Blex Over a year ago

Are you sure you use local database? How exactly you tried local.oplog.rs? Just to make it crystal clear, it suppose to be MongoDB.connect('mongodb://host:port/local',(e,db). What do you have when you run rs.status() and rs.config() from the shell?

user3094755 Over a year ago

OK, I see now. I would need to use db/local which would be a global db on my cluster, so I don't think it will work for my application, since I wanted to capture transaction for a given db, not all db's. So I can either filter the global db/local, or create new cappedCollection for each db. Thanks for the answer.

Alex Blex Over a year ago

You just need to filter oplog for the namespace that matches your database name

user3094755 Over a year ago

I'm multi-tenanting the Atlas instances, so wanted to avoid having a shared 'manager' process.

|

Collectives™ on Stack Overflow

MongoDB oplog stream Node.js

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related