How kafka connector works for postgresql and mysql database

Question

I'm following the quick start tutorials from here quick-start-kafka-connect

This tutorial shows how to stream the mysql database table changes into kafka topic.

The only part is download everything and just add /tmp/kafka-connect-jdbc-source.json file with some config properties and start

How does this work in background ?

1 : Does it create connection with database and monitor tables for specific intervals of time? OR

2 : Does it uses replication log? (i don't know how this works)

3 : Is this same mechanism for mysql and postgresql?

OneCricketeer · Accepted Answer · 2019-01-05 23:10:04Z

2

Debezium monitors the OpLog.

Kafka Connect JDBC by Confluent (which you've linked to) can use a time-interval, and that configuration is shared by all JDBC-compliant connections, MySQL and Postgres included.

For incremental query modes that use timestamps, the source connector uses a configuration timestamp.delay.interval.ms ...

replication log? (i don't know how this works)

You can find the Debezium guide here, but this mechanism differs for Mongo, Postgres, MySQL, MSSQL, Oracle, etc.

answered Jan 5, 2019 at 23:10

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How kafka connector works for postgresql and mysql database

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related