RabbitMQ work queues using Dependency Injection in c#

Question

I am using rabbitmq in a "Work Queues" scenario.

I need eg. a pool of 5 consumers, (each with its own channel), so one consumer doing I/O operations, won't block other consumer of the same queue.

Eg. If I have on my queue: Message 1, Message 2, Message 3, Message 4. Each instance of (FistConsumerHandler) will take 1 message from the queue using Round Robin (default rabbitmq behavior)

The problem I am facing is I need to do this using Dependency Injection.

Here is what i have so far:

On Windows service start (my consumers are hosted in a windows service):

    protected override void OnStart(string[] args)
    {
        BuildConnections();
    
        // Register the consumers. For simplicity only showing FirstConsumerHandler.
        AddConsumerHandlers<FistConsumerHandler>(ConstantesProcesos.Exchange, ConstantesProcesos.QueueForFirstHandler);
    
        BuildStartup();
    
        var logger = GetLogger<ServicioProcesos>();
    
        logger.LogInformation("Windows Service Started");
    
        Console.WriteLine("Press [enter] to exit.");
    }


    protected virtual void BuildConnections(
        string notificationHubPath = "notificationhub_path",
        string rabbitMQHostname = "rabbitmq_hostname",
        string rabbitMQPort = "rabbitmq_port",
        string rabbitMQUserName = "rabbitmq_username",
        string rabbitMQPassword = "rabbitmq_password")
    {
        ContextHelpers.Setup(ConfigurationManager.ConnectionStrings[appContextConnectionString].ConnectionString);
   
        if (_connection == null)
        {
            var factory = new ConnectionFactory
            {
                HostName = ConfigurationManager.AppSettings[rabbitMQHostname],
                Port = int.Parse(ConfigurationManager.AppSettings[rabbitMQPort]),
                UserName = ConfigurationManager.AppSettings[rabbitMQUserName],
                Password = ConfigurationManager.AppSettings[rabbitMQPassword],
                DispatchConsumersAsync = true,
    
            };
    
            // Create a connection
            do
            {
                try
                {
                    _connection = factory.CreateConnection();
                }
                catch (RabbitMQ.Client.Exceptions.BrokerUnreachableException e)
                {
                    Thread.Sleep(5000);
                }
            } while (_connection == null);
        }
    
        _startupBuilder = new StartupBuilder(_connection);
    }
    
    protected void AddConsumerHandlers<THandler>(string exchange, string queue)
    {
        var consumerHandlerItem = new ConsumerHandlerItem
        {
            ConsumerType = typeof(THandler),
            Exchange = exchange,
            Queue = queue
        };
    
        _startupBuilder._consumerHandlerItems.Add(consumerHandlerItem);
    }


protected void BuildStartup()
{
    ServiceProvider = _startupBuilder.Build();
}

Startup Builder:

using Microsoft.Extensions.DependencyInjection;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System;
using System.Collections.Generic;

public class StartupBuilder
{
    private static IConnection _connection;

    private IModel _channel;

    public List<ConsumerHandlerItem> _consumerHandlerItems;

    public IServiceCollection Services { get; private set; }

    public StartupBuilder(IConnection connection)
    {
        _connection = connection;
        _consumerHandlerItems = new List<ConsumerHandlerItem>();
        Services = new ServiceCollection();
    }

    public IServiceProvider Build()
    {
        _channel = _connection.CreateModel();

        Services.InitSerilog();

        // Add channel as singleton (this is not correct as I need 1 channel per ConsumerHandler)
        Services.AddSingleton(_channel);

        // Register the ConsumerHandler to DI 
        foreach (var item in _consumerHandlerItems)
        {
            // Add FirstHandler to DI
            Type consumerType = item.ConsumerType;
            Services.AddSingleton(consumerType);
        }

        // Finish DI Setup
        var serviceProvider = Services.BuildServiceProvider();

        // Bind the consumer handler to the channel and queue
        foreach (var item in _consumerHandlerItems)
        {
            var consumerHandler = (AsyncEventingBasicConsumer)serviceProvider.GetRequiredService(item.ConsumerType);
            _channel.AssignNewProcessor(item, consumerHandler);
        }

        return serviceProvider;
    }
}

Helpers:

public static class QueuesHelpers
{
    public static void AssignNewProcessor(this IModel channel, ConsumerHandlerItem item, AsyncEventingBasicConsumer consumerHandler)
    {
        channel.ExchangeDeclare(item.Exchange, ExchangeType.Topic, durable: true);
        channel.QueueDeclare(item.Queue, true, false, false, null);
        channel.QueueBind(item.Queue, item.Exchange, item.Queue, null);
        channel.BasicConsume(item.Queue, false, consumerHandler);
    }
}

Consumer handler:

public class FistConsumerHandler : AsyncEventingBasicConsumer
{
    private readonly ILogger<FistConsumerHandler> _logger;

    private Guid guid = Guid.NewGuid();

    public FistConsumerHandler(
        IModel channel,
        ILogger<FistConsumerHandler> logger) : base(channel)
    {
        Received += ConsumeMessageAsync;
        _logger = logger;
    }

    private async Task ConsumeMessageAsync(object sender, BasicDeliverEventArgs eventArgs)
    {

        try
        {
            // consumer logic to consume the message
        }
        catch (Exception ex)
        {
            
        }
        finally
        {
            Model.Acknowledge(eventArgs);
        }
    }
}

The problem with this code is:

There is ony 1 instance of FistConsumerHandler (as is reigstered as singleton). I need, for instance 5.
I have only 1 channel, I need 1 channel per instance.

To sum up, the expected behavior using Microsoft.Extensions.DependencyInjection should be:

Create a connection (share this connection with all consumers)
When a message is received to the queue, it should be consumed by 1 consumer using its own channel
If another message is received to the queue, it should be consumed by another consumer

score 0 · Answer 1 · answered Jan 12 '21 at 19:08

TL;DR; Create your own scope

I've done something similar in an app I'm working on, albeit not as cleanly as I would like (and thus why I came across this post). The key for me was using IServiceScopeFactory to get injected services and use them in a consumer method. In a typical HTTP request the API will automatically create/close scope for you as the request comes in / response goes out, respectively. But since this isn't an HTTP request, we need to create / close the scope for using injected services.

This is a simplified example for getting an injected DB context (but could be anything), assuming I've already set up the RabbitMQ consumer, deserialized the message as an object (FooEntity in this example):

public class RabbitMQConsumer
{
    private readonly IServiceProvider _provider;

    public RabbitMQConsumer(IServiceProvider serviceProvider)
    {
        this._serviceProvider = serviceProvider;
    }

    public async Task ConsumeMessageAsync()
    {
        // Using statement ensures we close scope when finished, helping avoid memory leaks
        using (var scope = this._serviceProvider.CreateScope())
        {
            // Get your service(s) within the scope
            var context = scope.ServiceProvider.GetRequiredService<MyDBContext>();
            
            // Do things with dbContext
        }
    }
}

Be sure to register RabbitMQConsumer as a singleton and not a transient in Startup.cs also.

RabbitMQ work queues using Dependency Injection in c#

1 Answers1

TL;DR; Create your own scope

References: