Questions tagged [message-queue]
A message queue is a service that receives events or messages from producers and provides them to consumers.
259 questions
3
votes
3
answers
555
views
Design question for exactly-once processing in a message-driven system using a unique ID
To achieve exactly-once processing where messages are consumed from a queue with at-least-once delivery, many sources (e.g. here and here and here) suggest attaching a unique ID to messages in the ...
- 139
0
votes
0
answers
195
views
How does each application replica read a unique message from the message broker
In RabbitMQ streams or Kafka, messages are not deleted after being consumed. If you have a consumer application replicated across multiple Kubernetes pods, how can you ensure that each pod picks up a ...
- 9
0
votes
5
answers
170
views
How can I measure which messages are causing queue backups?
Consider the following multi-producer, multi-consumer system:
For simplicity, assume that we only have one queue
Messages are dequeued on a first-in-first-out basis
Oldest-message-pickup-latency has ...
- 119
1
vote
1
answer
201
views
Background thread processing vs queue based processing for relatively short tasks (max 30-40 seconds)
I need suggestion / recommendation on the design approaches mentioned below.
UseCase: I have a usecase where a client uses my system to generate some recommendations. Now, these recommendations are ...
- 19
3
votes
3
answers
259
views
Handling rate limits / delays in consumers without affecting performance of other operations
I have a producer that generates a batch job that consists of multiple operations (approx. 100 - 10000). These operations can be processed in any order, ideally as fast as possible.
The processing of ...
- 141
0
votes
1
answer
180
views
How to rebalance data across nodes?
I am implementing a message queue where messages are distributed across nodes in a cluster. The goal is to design a system to be able to auto-scale without needing to keep a global map of each message ...
- 115
7
votes
5
answers
445
views
What can I do to get a message processor to slow down the rate of writes that it is making to a database?
We have this architecture:
queue -> message processor (horizontal scaling) -> RDBMS
Sometimes external systems dump 10k messages onto the queue and the message processor of course dutifully ...
- 229
0
votes
3
answers
2k
views
Backend to client communication: WebSocket or Message Queue?
I have a hobby project, for simplicity imagine a poker game, consisting of a backend (Spring) and a client application (Flutter) where the client can join rooms in which he can take a seat. The client ...
- 9
2
votes
4
answers
310
views
Ensure concurrent processing based on a message key
I've been wondering about the following: Say you have an async message/task and you want that message only processed one at the time. for example: Only process one order for each customer at the time. ...
- 174
0
votes
2
answers
342
views
How to implement event payload isolation in an event driven architecture?
I have an event, let's say order.placed The payload consists of some general information about the order and 3 confidential fields. For example part of the message should only be visible to consumer A,...
- 129
0
votes
1
answer
94
views
Design a sequential processing of records
About 10 to 15 records are processed per day with the time interval of 5 minutes between each record.
System A inserts a record in DB and sends id of that record to active mq.
System B Listener ...
- 119
1
vote
0
answers
134
views
golang: pattern for handling message queues? Are named functions anti-idiomatic somehow?
Had a discussion today in how to implement services that work with messages coming in from event queues. We call these services processors. One of us argues for using several functions, while the ...
2
votes
0
answers
139
views
parent correlation id in message with rabbit queue and open telemetry
I have a rabbit queue that I use to communicate between applications.
I have a chain of three message.
App 1 generate and push a message in rabbit(message 1)
App 2 catch the message and generate ...
- 307
3
votes
2
answers
497
views
Can multiple producers cause events with the same key to be enqueued out of order within a specific partition of a topic?
I'm aiming to create a simple event-driven system where each microservice operates with its own database. The idea is to share database changes across microservices through events. To ensure proper ...
0
votes
1
answer
152
views
Is RabbitMQ suitable for allowing SaaS customers to subscribe to events?
I work on a SaaS system which is generally a single-tenant environment. Customers (whom the SaaS instances are for) would like a way to subscribe to events, such as updates of certain data entities. ...
- 101
0
votes
1
answer
185
views
Generic message broker based on WCF
I apologize in advance for a long-read question, but i'm hardly stuck here and don`t how to ask for help using fewer words.
I need to implement a middleware service which should be placed between two ...
0
votes
0
answers
113
views
Event based update of In memory Hashmap in multiple pods of kubernetes
I have a service (S1) that is deployed on kubernetes having multiple replicas (pods).
S1 requires some data from database (Cosmos Cassandra Api) for calculation.
This data is constantly updated by ...
- 101
1
vote
3
answers
352
views
Long-polling with message queues in a clustered environment
I have a system design question that I'm looking for some guidance on. I have two different systems that need to have a basic level of communication. This is abstracted via message queues.
For example,...
- 111
1
vote
1
answer
104
views
Handling a single event as broadcast and round-robin depending on service
I have Service A that publishes an event to RabbitMQ. I have two instances of Service B that will use the event to write to its database. I have two instances of Service C that will send the event ...
- 136
0
votes
0
answers
126
views
How to use an external event queue with inversion of control (IoC)
I'm working on an AspNetCore application, with a requirement to raise and handle certain events asynchronously using an external event queue. I'll use AWS services as examples here because that's what ...
- 135
0
votes
1
answer
651
views
Is a message queue the right pattern to use for intra-module communication in a .net core application
Background Information
We have a .Net core application that uses Hangfire (pretty version of cron) jobs to batch process different types of data.
As a bogus example, we have one job that will run ...
- 581
-1
votes
3
answers
632
views
Avoid race condition using RDBMS transaction
I am working on design of a distributed system which will process some events. For simplicity lets say, multiple instance of same service will be consuming from same queue, every message will have a ...
0
votes
2
answers
284
views
Complicated task scheduling architecture
So I want to make an application in which a user will hit an endpoint to save a job model to storage that includes some metadata to perform a long computation against which will be offloaded to a ...
- 101
3
votes
4
answers
498
views
Organize and maintain a lot of cron jobs
I am working on a fairly large project written in PHP (Yii2) in which we are increasingly using cron for background tasks, such as generating caches, reports, etc.
We have started to move the hourly ...
- 39
2
votes
2
answers
9k
views
Is Kafka an event bus or a publisher subscriber (pubsub)pattern?
Building on this answer here, and its comments it entails that subscribers need to know and locate the publishers in a traditional pub/sub system. It also entails that publishers need to live at least ...
- 131
0
votes
2
answers
300
views
Is it a good/bad practice for a service to send events to itself via an external queue?
Imagine there is an Order entity that has a lot of states and conditional transitions between them. At the moment it is stored in a relational DB.
There are multiple consumers of Orders that perform a ...
- 307
0
votes
2
answers
338
views
How can I track the status of multiple messages in asynchronous processes?
Let's say I've a sendMessage endpoint. This endpoint looks at configured message destinations and then puts 1..N messages into a queue (e.g. {userId: 3, type: SMS, trackingID: X} and {userId: 3, type: ...
user374484
0
votes
3
answers
152
views
Queue for tasks and notifications
I've been a heavy user of "batch jobs" to perform different kinds of logic in systems. Over the last couple of weeks, I've been thinking and reading about other approaches and I wanted to ...
0
votes
1
answer
165
views
How to seamlessly interact with a message broker when the underlying system (SQS, RabbitMQ) can change depending on the environment?
Having a poor knowledge of the extended features of the available message brokers, I was wondering how to approach this.
We have some environments where only RabbitMQ is available, others where ...
- 111
0
votes
2
answers
754
views
Message broker design pattern best practice [closed]
I'm designing the architecture of a platform to introduce a message broker in an existing data collection web application.
This web application is currently used to upload data from excel files that ...
- 109
0
votes
1
answer
63
views
Which techniques to follow to make a fast and consistent message consumer that writes to a RDBMS [closed]
I am building a consumer application that reads messages from a queue and runs basic sanitization and deduplication and persists them to an RDBMS.
The functional requirements are
messages must not be ...
- 131
-1
votes
3
answers
496
views
How can i handle really long-term running tasks?
Im not dealing with a x minute task that can use an message queue and some workers to be done. Lets say that i have an cctv camera that capture 24/7, but can be stopped or started again via Web API ...
- 1
2
votes
2
answers
3k
views
Implement message queue in API Gateway
Recently we have this client who asks for a API Gateway solution, in this case it’s Kong. Currently, they have 10 services (200 APIs) that are running on really legacy stuff (built with C++ and ...
- 129
1
vote
0
answers
124
views
How to have only one consumer handle a long running task at a given point?
I need to design an application which makes requests to an external API. External API has a hierarchy of entities: User which contains ListOfItemss which contain Items. I need to get data on all Items ...
- 167
0
votes
1
answer
74
views
Queue system with priority on Consumers
Imagine a queue system with different consumers - some are low cost whist (like 'reserved/spot' instances in the AWS world) others would be high cost.
The system should ensure the messages are ...
- 320
0
votes
1
answer
217
views
Bubbling errors upstream in async message-based services
Imagine a simple set up of an API and a 2nd service, where the API pushes some msgs to the message queue and the service pulls them and processes them.
Now, if an error occurs while processing a msg, ...
- 191
0
votes
0
answers
67
views
How to handle maintenance to your queuing server (e.g. rabbitMQ)
I'm new to queuing and looking into options that could be done operationally. I'm sorry if I have any conceptual misunderstandings related to queuing.
I watched this video where the presenter made a ...
- 341
6
votes
6
answers
557
views
Best practice to architect a system in which local users may override global definitions
Our travel industry client, operating across three continents and several countries, wants to standardize the process of choosing travel packages, air routes, hotels, and hundreds of other travel ...
- 77
1
vote
0
answers
482
views
Do we really need a message brocker in WebSocket system?
I'm engineering architecture of a new web based software.
I've never worked on high-scale softwares before and I'm reading a lot about it.
To increase client-side speed and reduce load on servers, and ...
- 19
0
votes
1
answer
618
views
How to optimize average rating calculation in a review system?
I'm thinking of a designing a review system (restaurant, hotel etc) where users can drop star reviews. Typically in a such a application, you can see the average rating of an entity along with all ...
-1
votes
1
answer
790
views
Message queuing systems: sequential processing with dependencies between queues
I am trying to understand Rabbitmq and how I can accomplish sequential message processing in certain circumstances only. For example if I have 2 queues. One for pricing an order and one for sending ...
- 107
-1
votes
1
answer
93
views
Any pattern for delaying processing of a (JMS) message until another task completes
Is there a good pattern in microservices systems for delaying the processing of a JMS message until some other task has been completed. Let's say, for example, a message is sent to a topic when a ...
- 179
0
votes
1
answer
136
views
Expose api to third parties to send PDF files (invoices)
Our system needs to process PDF invoice files from generic external third parties.
Using HTTP is the most obvious, but this isn't the greatest of ideas in case we need to deal with bulk uploads.
Even ...
- 185
2
votes
2
answers
1k
views
Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?
We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
- 83
0
votes
2
answers
228
views
Prevent missing updates using a message broker after creating a task
I'm trying to design a data updates mechanism in my micro-services architecture.
For the sake of simplicity, let's assume we have two micro-services A and B, B exposes an API for creating some tasks, ...
- 109
3
votes
1
answer
1k
views
Long-running compute-intensive tasks in APIs: background workers?
I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing.
To give an impression of the compute requirements, ...
- 139
0
votes
0
answers
48
views
Design a Program which Forwards Work Pieces to Corresponding Modules
I am working on a program that will listen to a queue of incoming message and will process each message differently depending on the metadata of that message. The messages must be processed in order ...
- 101
2
votes
2
answers
1k
views
A service having to have both an API and listen to events - Is this an anti-pattern for microservices?
I have a use case to create a service that consume messages from a message queue, process them, store them in the DB, and expose the processed results via an API. Therefore, the service I'm going to ...
- 129
4
votes
1
answer
6k
views
Event bus vs PubSub
I have been using queues and PubSub patterns for years but never really came across the terminology "event bus". After some googling I didn't really find a clear distinction between "...
- 189
0
votes
2
answers
377
views
Can messages from single partition be forwarded to multiple partitions to increase parallel processing in Kafka?
I'm wondering if the following approach can be used in order to increase parallel processing of Kafka messages. Suppose there's a topic with N partitions. At some point the system hits a bottleneck in ...
- 167