Create temporary queue and subscribe using queue trigger in Azure Function

I agree with @Sean and @Jesse Squire When a trigger is created, it is fixed to a single queue and cannot be altered. Making a hosted service that directly utilises the Service Bus client library would be your best option. If you want to use numerous queues you can use ServiceBusClient to generate a ServiceBusReceiver. Send a certain number of messages, then shut off the receiver and move on to the next. You may choose to accomplish that simultaneously or assign different services to different portions of it.

Below is the C# code illustrating how to use the ServiceBusClient to spawn a ServiceBusReceiver to handle messages from multiple queues simultaneously:-

using Azure.Messaging.ServiceBus;
using System;
using System.Threading.Tasks;
using System.Threading;

class Program
{
    const string ServiceBusConnectionString = "Endpoint=sb://siliconsb45.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxxxxMubQxxxx";
    const string QueueName1 = "customer";
    const string QueueName2 = "customer1";

    static async Task Main(string[] args)
    {
        await ProcessQueuesConcurrently();
    }

    static async Task ProcessQueuesConcurrently()
    {
        ServiceBusClient client = new ServiceBusClient(ServiceBusConnectionString);

        var receiver1 = client.CreateReceiver(QueueName1);
        var receiver2 = client.CreateReceiver(QueueName2);

        // Start receiving messages from Queue 1
        var processingTask1 = ProcessMessagesAsync(receiver1);

        // Start receiving messages from Queue 2
        var processingTask2 = ProcessMessagesAsync(receiver2);

        // You can add more queues and tasks as needed...

        // Simulate processing for a certain duration (e.g., 30 seconds)
        await Task.Delay(TimeSpan.FromSeconds(30));

        // Set a flag to stop processing
        bool stopProcessing = true;

        // Wait for the processing tasks to complete
        await Task.WhenAll(processingTask1, processingTask2);

        // Close the receivers and client
        await receiver1.CloseAsync();
        await receiver2.CloseAsync();
        await client.DisposeAsync();
    }

    static async Task ProcessMessagesAsync(ServiceBusReceiver receiver)
    {
        try
        {
            while (true)
            {
                ServiceBusReceivedMessage message = await receiver.ReceiveMessageAsync();

                if (message != null)
                {
                    Console.WriteLine($"Received message: {message.Body}");
                    // Process the message here

                    // Complete the message to remove it from the queue
                    await receiver.CompleteMessageAsync(message);
                }
                else
                {
                    // No more messages, exit the loop
                    break;
                }
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine($"Error processing messages: {ex.Message}");
        }
    }
}

Output:-

Received messages from both the queues:-

• You can also create multiple Function instances. To handle messages from various queues simultaneously, each instance might have a different queueName configuration parameter.

• Use the Azure Service Bus SDK to programmatically create a new queue whenever a new customer is created. Reference - Refer code from other languages in the left.

Queue triggers appear to require the name to be hardcoded for the queue

This is incorrect, you should not be hardcoding the connection but you can also make the queue name configurable. You only hardcode the Key that is used to access the setting:

• How To Configure Queue Name for Queue Trigger In Azure Function App

The reason that there are not a lot of examples out there using an infinite number of queues is an indication that this is not a common solution to this problem. I can however assure you that this problem comes up quite frequently as a product request, it just doesn't often make it into the requirements ;)

I don’t want customer 3 messages having to wait whilst customer 1 And 2 are being processed

As a general concept, this is very hard to ensure, at some point in the transaction chain, there is always going to be a pinch point where one customer will have to wait. But there are ways to minimise the delays.

I talked briefly about this in my first response but there are techniques that are designed for high compute needs per user, and there are different techniques for supporting high volume of low compute needs. Either can work but the costs can quickly blow out if you pick the wrong one.

From a sales and payment point of view, shopping carts and reserving inventory items before the actual transaction is generally all you need. The speed of actual processing of a payment is often governed by the payment merchant and because the allocation of the inventory items being purchased has been pre-approved or allocated before the physical transaction, there is no reason that any customer waiting would negatively affect the others, because their sale is already guaranteed. Scenarios like this usually work really well with a minimal number of queues.

• The important aspect of this model is that all of the information needed to process the payment is collated into a single payload, which ultimately is a single message on the queue.

This works well for online stores that have fixed inventory with high demand, you might incorporate quantity limits and a timeout on the user session such that one user can't select items and sit on them specifically to prevent other users from having access. One the session is cancelled or times out, the reserved stock becomes available again.

• The processing of the initial request may involve a number of steps, a microservice implementation might model these steps as separate queues, 1 queue per step, being chained such that the completion of step 1 enqueues the message for step 2.
• To determine if this is appropriate requires understanding about WHY it is important that Customer 3 is not waiting for Customer 1 and 2 to be processed.

This wouldn't work for peer-to-peer trading scenarios where spot prices are affected by demand or there is a general ledger of trades for sale and bids to buy. This is the type of scenario where you specifically do want each customer to wait while other transactions are processed, so the advice here is not suited to those scenarios.

10000 queues because I have 10000 customers is just ludicrous, your plan is to grow but lets say 1 million customers and queues... is that a reasonable way to utilise the technology we have available?

Ask yourself these questions first:

• How many concurrent users will be connected?
• How many of those users will be making transactions?
• How many transactions are you expecting to be made concurrently?
• What is the average time between the user logging in and making a transaction?
• Are there specific or known peak times when concurrent transactions are high and other times where the system might be idling?
• What is an acceptable wait time between making a transaction and waiting for a reply?
• How long does it take to process a single transaction?
• What is the impact on the customer, other than time, if they do have to wait?

The reason to ask these questions is that if the actual number of concurrent transactions is low, and the transaction processing time is low, then is there a business case to bother worrying about potential waits that users might experience?

If the concurrency and transaction rates are high, then you first need to use technology stack that can handle the load. You also need to identify if the maximum throughput of your end to end networking can even handle the expected peak loads.

• If you already have a solution in place, scaling out sideways through multiple queues to allow an infinite degree of parallel processing is not a good solution, all parallel solution inherit some form of orchestration overhead, there is always a sweet spot in parallel solutions where the cost and effort starts to outweigh the benefits, or where managing the parallel threads starts to slow or negatively impact the entire throughput.

For high compute loads, like game engines, one solution that has been used is to spin up entire environments per user session, but only when those sessions are active. From a practical sense, this has significant cold-start times compared to many other solutions, but if there is a delay between logging in and needing the compute, then perhaps the cold-start time can be hidden from the user who is spending time in the front end experience anyway. If there isn't a natural delay in the user experience, but the quality of service is important to you, then either make them wait, or specifically re-engineer the user experience such that there is enough of a delay that the service will be ready by the time the customer makes a transaction.

In this scenario, your code only needs a single queue, what changes between the environments is the connection strings.

• This plays well into the standard Azure Queue storage trigger for Azure Functions examples because the queue name is fixed or hardcoded, the connection is always a configuration setting.

At the end of the user session the environment can be decommissioned, or you could use it for the next customer.

This is a different implementation of the Partition concept I described in my first response but ultimately the same underlying principal. Depending on your compute needs and the criticality of your business logic, partitioning the whole environment might make sense. You might even partition the environments, and partition the queues within them and also implement shopping cart style pre-allocation combining the benefits of each different approach to optimise the user experience and throughput of transactions.

At the core of this will always be an analysis of the acceptable costs per customer/transaction as well as the development effort to get this to market at all. You might find that you start in one direction at first then as you expand you might need to implement something new.

Minimizing the customer interactions into complete batches is the best place to start, implementing pre-allocation if that makes sense in your scenario.

You can then implement partitioning of the queue processing is a pragmatic next step.

Partitioning the users into specific queues should be manageable after the previous two steps have been optimised to their practical limits.

Finally, once you need Hyper scale, partitioning across whole environment deployments should be a natural progression, knowing that the environments themselves have already been optimized.