Architecture for RabbitMQ in tandem with AngularJS/.NET Web API application

[lukegf]

Hi all,

So I have a SPA (single page application) with AngularJS as the front-end and .NET Web API the back-end (C#). I have a requirement to generate SQL Server Reporting Services (SSRS) reports upon some user action, i.e. select some data and press a button. Here is a rough illustration of my existing architecture:

    FRONT END                                                             BACK END                                                                                 SSRS Report

    ==========                                                             =========                                                                                 ===========

   AngularJS Web page  -->  .NET Web API controller --> Intermediary class (passes arguments to report) -->  Generate report and save to file

This works, but the problem is while the reports are being generated the application has to sit there and wait for them to finish and in the meantime the user can't do anything. If there are many reports to create, 100+ for example, it can take upwards of a minute or two, which is not acceptable for me.

So this is where RabbitMQ comes in. The idea I have is for the .NET Web API controller to send a message to a server, once the server receives it a call is performed to the intermediary class above, which generates the report, then a response is sent back to the client letting them know the report has been generated. I think I would need to use the RPC scenario for that so I've done the corresponding tutorial on the RabbitMQ site and it seems straightforward enough.

What I don't know though, and this is what my question is about, is how to start-up a client and a server from a .NET Web API application. Frankly I am a bit lost at the moment as this is my first real experience with any sort of messaging and I am not quite sure how to approach it, architecture wise. If I can get a detailed picture in my head of how the whole thing would work I can code it from there, but at this point I am having trouble visualizing how to structure the application.

So, questions:

1. How do i start a RabbitMQ server and client from a .NET Web API controller? Should the server and client be class libraries or console applications, or something else entirely?

2. Is the architecture I am envisioning going to work or is there a better approach?

3. Any general advice about how to approach what I am trying to accomplish would be appreciated.

[Andres Gonzalez]

Hi,

I have no experience with .NET/C# and I am also trying to come up to speed and understand messaging and RabbitMQ, so take whatever I say cautiously. But I am also working on an Angular client that communicates with RabbitMQ so I thought I would at least comment because I feel your learning pain that is associated with any new API.

Your client should not "start the RabbitMQ server"  The RabbitMQ broker is a process and can be thought of as a peer (either locally or distributed on remote a server) of your web server. To take the simple case, the RabbitMQ broker could possible be running on the same box with your back end HTTP server that is serving your Angular code.  I am doing that in my case and have node.js serving Angular code but if you are using something like Apache architecturally it is the same. Remember that the RabbitMQ broker is only a transport mechanism so it can be located anywhere. So to answer your question as to how to start the RabbitMQ server, it should be started like your HTTP server is started because it is just another network resource/service that your client and worker threads will send messages to. 

There are Javascript packages that you can use in your Angular code so the client could send a message directly to the RabbitMQ broker or, alternately, to your web server that would then send the request to the RabbitMQ broker (I am using the node.js package amqplib).  You have to have some other process that has subscribed to a RabbitMQ queue to then receive the message request, do the report generation, then send the reports back to RabbitMQ in a message. There are RabbitMQ/AMQP libraries for C# so it could be coded into your existing C# code base.

In my opinion, report generation (your requirement) is a perfect fit for a messaging-base solution because generating reports is time/processing intensive.  In my experience, it is going to appear to be overkill initially because it is a big step to get a RabbitMQ broker (and the messaging architecture) integrated into an existing non-messaging architecture. So don't be discourage by that because, once the migration is done, your whole application is much more flexible and you will be glad you took the pill and went through the pain. It is not trivial to migrate an existing non-messaging-based application to a messaging-based architecture but it is worth it.

So....good luck...

-Andres

[lukegf]

Hi Andres,

 

Thank for your answer. A couple of questions.

 

In my case, I am using an IIS server for my Angular code. I basically have one project in Visual Studio 2015 that has all my Angular code and another one, which is a

ASP.NET Web API project, that has my backend Web API code in it. So far I haven't even had to mess with the IIS web server. So what you are saying is the RabbitMQ Server (broker) could be running on the same box as my IIS server?

 

In my case I would be sending messages to the RabbitMQ Server from my Web API code (C#), i.e. not the Angular code. When you say "some other process that has subscribed to a RabbitMQ queue", you mean the Client, right? That's assuming we are talking about the Remote Procedure Call (RPC) pattern which consist of a client and server sending messages between each other?

 

And thank you for that last paragraph as well. That encourages me in picking a messaging based solution, because I wasn't sure if that was a good way to go. While

doing research online, I came upon several different background tasks solutions (such as Hangfire and Azure Webjobs)) that could do what I need also and I was in fact worried RabbitMQ might be an overkill.

[Andres Gonzalez]

Hi lukegf,

So what you are saying is the RabbitMQ Server (broker) could be running on the same box as my IIS server?

Yes. It doesn't have to be local on the same box, but it certainly can be running as a separate process on the same box.  It might be helpful to NOT think of RabbitMQ in the typical client/server model. So I would suggest using the term "broker" instead of "server."  At least this mind-shift in how to think of it was helpful to me.  Think of RabbitMQ as a message router or message inter-mediator and not as a message server.  

 

When you say "some other process that has subscribed to a RabbitMQ queue", you mean the Client, right?

No, I meant another process that actually generates the reports (the client does not generate the reports). Again, it is perhaps best not to think in terms of a client. The only thing RabbitMQ will do is route or distribute messages (which can be the requests and results like the reports).

Here is one possible architecture using a publisher/consumer model.  Your Angular client makes a request for reports which your IIS server receives. The IIS server sends a message to RabbitMQ, the message contains all of the details about which reports, the appropriate date/time ranges, etc for the request. The IIS server has "published" a message to RabbitMQ and now does not care how that request is processed.  The IIS server also "subscribes" to a queue on RabbitMQ over which it will receive the completed results of the report generation request when it is completed. The reason why this is somewhat overkill is because the IIS server could very easy generate the reports without using RabbitMQ. But using a message broker as an inter-mediator makes the request asynchronous and decouples the request from whatever actually processes the request.

For the actual report generation, another server or process (or even the same IIS server) also communicates with RabbitMQ. It will subscribe to a queue over which it receives the message requests, generates the reports, then publishes the reports back to RabbitMQ.

There are other patterns of course, for example your Angular code could connect directly to RabbitMQ and issue the request directly to RabbitMQ and then receive the reports directly from that RabbitMQ connection.

Again, I have no experience with .NET/C# and I myself am just starting with messaging and RabbitMQ, so take whatever I say cautiously. As you mentioned, there are indeed many ways to architect your application. I am not familiar with Hangfire nor Azure Webjobs, but I am really sold on the basic concept of messaging as part of the overall design of applications. In my opinion, the benefits far outweigh the added complexity or any possible overkill. Plus, having an application designed with messaging interfaces opens up all sorts of new possibilities and use cases for the application. So I am a convert and have chosen RabbitMQ to be part of my application architecture.  

-Andres

[lukegf]

Hi Andres,

I think i understand better now. The confusion with the Client and Server terms originated from the RPC pattern tutorial on the RabbitMQ web site (https://www.rabbitmq.com/tutorials/tutorial-six-dotnet.html) where they refer to a Publisher as a Client and a Subscriber as a Server. I created a diagram (see attached file) that reflects my current architecture, based largely on what you suggested above.

[Andres Gonzalez]

Yes, from a high level view, that is one possible architecture.  I personally view the queues as contained within RabbitMQ, so I view it as your red queue graphics within your RabbitMQ blue oval instead of above and below it.   I also view it as each endpoint (your .NET web API and your Service Application boxes) operating in both publish and consume modes.  You indeed show this with two arrows from each endpoint to the center Windows Services rectangle. But technically each arrow represents a different roll or function, so I view each of those arrows as terminating or originating to different separate ovals, one for when that endpoint is operating as a publisher and the other for when that endpoint is operating as a consumer.

But the way these pictures are drawn depends on how one abstracts the various concepts/functionality. So, the way I view it would draw the architecture is certainly not the only way to depict it. I think you do indeed have a correct high level understanding of the architecture.

BTW, nice graphic. What application did you use to draw the png?  I assume it is a Windows application. 

-Andres

[lukegf]

I see. Yes, it does make more sense for the queue graphics to be inside the RabbitMQ server graphic, because they are not separate concepts/objects, like Publisher and Consumer are. It is true that both the Publisher and the Consumer do send and receive messages, so from that point of view you are correct. However I wanted to have a clear separation of roles conceptually, that's why I drew it like that. And also, that's how the diagram look in the RabbitMQ tutorials.

As per the diagramming software, I used Gliffy. It's a pretty good alternative to Visio. It's an online tool, so there is nothing to install and the first five diagrams are free.