0

I would like to implement the outbox pattern with the following concepts:

  • event generating service (let's call it Generator) will save them to its db in one transaction with the data change (the principle of the outbox pattern)
  • there would be a separate event dispatcher service (let's call it Dispatcher) that would pull the generated events from the db periodically and sent them (synchronously, HTTP) to the external service managing Kafka (black box for me), then mark them as sent in the db
  • there would be a REST API implemented that would allow manual creation and modification of events in case such need occurs (a simple CRUD)
  • the expected traffic is rather small (max 10k requests a day)

As I understand, this can be realised in two ways.

One approach is for both services to share a db. This should require less work to implement and is generally simpler. The problem with this is as follows:

  • I don't really have experience in such situations and I don't entirely know what problems to expect (apart from the fact that I should make sure that enough connections are available)
  • sharing a db between microservices is an anti-pattern in my head (although I'm probably biased)
  • I'm not sure which service should have the REST API for the events as both seem equally right (or equally wrong) for that

The second approach is to leave the Dispatcher service without the db access and communicate with the Generator service through an API. This approach seems much clearer for me not only because the db is not shared, but now the Dispatcher service has a clear responsibility of just communicating with the external service, while the Generator service becomes the single owner of the events.

However what I don't like about this idea is adding the additional network layer with all its potential problems (latency, errors, etc.).

What would be the recommended approach in this situation?

Improve this question

1 Answer 1

Reset to default
2

One approach is for both services to share a db

The thing here is that those are not strictly speaking different services. Quite often it does not make sense to separate this functionality into two different services, you can even have as a single application (which will contain both "Generator" and "Dispatcher"). Or you can split them into two applications and (I will allow myself to borrow a bit of k8s terminology here) which will be in a single deployment unit (which will allow to scale them differently depended on the infrastructure used) and/or service+sidecar.

It does not make sense to separate the "Dispatcher" as "full-blown microservice" since it does not actually represent subdomain/business capability, it is basically a technical/helper functionality (see decompose by subdomain and decompose by business capability)

sharing a db between microservices is an anti-pattern in my head

In most cases - yes, but as in most cases - patterns are not laws but some general guidelines.

The second approach is to leave the Dispatcher service without the db access and communicate with the Generator service through an API.

This basically will break the outbox pattern, you will have the same problems like with writing directly to the message queue/external service (in theory you can somewhat compensate for this but it is not worth the effort).

I'm not sure which service should have the REST API for the events as both seem equally right (or equally wrong) for that

That one is quite opinionated and hugely depends on concrete details, but I've seen both approaches in action doing quite fine.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks, you basically confirmed my thinking. Regarding separate service the main reason is that I need only instance of the pulling job to avoid conflicts and this is the simplest way to handle that (separate service with one instance).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.