change stream in NodeJs for elasticsearch

Question

The aim is to synchronize fields from certain collections on elasticsearch. With every change on mongodb, this should also be implemented on elasticsearch. I've seen the different packages. For example River. Unfortunately it didn't work out for me so I try without it. Is that the right approach with change streams?

How could you solve that more beautifully? The data must be synchronized with every change (insert, update, delete) on Elasticsearch. For several collections but different for each one (only certain fields per collection). Unfortunately, I don't have the experience to solve this in such a way that it doesn't take much effort if a collection or fields are added or removed

    const res = await client.connect();
    const changeStream = res.watch();
  
    changeStream.on('change', (data) => {
     // check the change (is the chance in the right database / collection)
     // parse 
     // push it to elastic server
    });

I hope you can help me, thanks in advance :)

Akash Patel · Accepted Answer · 2021-02-25 13:51:47Z

1

Yes. it will work but you have to handle following scenarios

when your node js process goes down while mongodb updates are ongoing.

you can use resume token and keep track of that token so once your process comes up it can resume from there.
inserting single document on each change.

it will be overwhelimg for elasticsearch and might result in slow inserts, which will eventually result in sync lag between mongo and elastic. so better collect multiple document in change stream and insert with bulk API operation.

answered Feb 25, 2021 at 13:51

Akash Patel

2001 silver badge10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

change stream in NodeJs for elasticsearch

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related