[Sample Request] Multi-Tenancy with MT3

[Maverik Gately]

Hi there,

I've been scouring the internet for days for any real direction on how to actually get multi-tenancy sorted with MT3 but so far my best lead ends up being "IConsumerFactory extension via AutoFac" answer. However, being brand new to both MT3 & Autofac, I'm at a complete loss when it comes to actual implementation.

Is there anybody who could put a simple sample together (preferably on github) that shows how to discriminate message handling based off a TenantId (doesn't matter if you do it via header or property, I can extend the sample myself for my own scenario).

My specific scenario is as follows:

1. Send a authentication message ValidateLogin.
   
2. Message is received and can be handled potentially by multiple authenticators (competing consumers for each tenant) but not all authenticators can handle all tenants.
3. One of the competing consumers that can actually handle the Tenant in question responds with a LoginValidated response.

To roughly draw how the queues would look like for an MT exchange:

ValidateLoginMessageMTExchange

  |  |

  |  |--TenantA-Authenticator1

  |  |--TenantA-Authenticator2

  |

  |--TenantB-Authenticator1

So Authenticator1&2 for TenantA will need to be competing consumers but TenantB-Authenticator1 will be a separately queued consumer (and of course TenantA shuold only get messages meant for TenantA while B gets for B).

Is there any hope for this scenario? If there are even snippets of information on internet that can be put together to make this work, I'm happy to do the digging if somebody can point me in direction and put it together.

@ Chris Patterson I really like your TenantContext idea in one of the SO answers and would love a sample along that line if at all possible.

I've also seen suggestions to use Distributor from MT2, but again.. all I have is a class name with no real clue on how the thing will fit together. I can see huge potential in MT3 (i've been semi following MT3 for the past few months as I really want to make it the backbone of our new project) but unfortunately with the number of subtle changes in MT3 and tons of ancient examples from MT2 I'm only left utterly confused. Thank you for all your hard work guys!

Kind regards,

Maverik

[Tomer Yoskovich]

Hi Maverick, like you, I have also requested for this feature, which apparently existed in MT2 (by the name of some sort of "Selector" interface which allows a consumer to say "i don't want to handle this message, throw it back in the queue") and was removed in MT3 (Chris Patterson mentioned this was a welcomed change after several years of having introduced this design fallacy)

I will paraphrase what I remember from his answer:

if the consumer doesn't want to handle a message, it gets thrown to the end of the queue. this could lead to waste: for example if only one consumer can handle IMessage1 but there are a lot of IMessage2 in the queue, then every message it pulls gets fetched and sent right back to the end of the queue. and then what would happen if we don't have any IMessage2 consumers at all? the same messages will be consumed again and again.

If I remember correctly, Chris's offered to separate messages to different queues dealing with different contracts.
To me, it seems like it reduces the managability of the system (because of the changes that will be necessary if a new type was introduced).

If there is a viable solution to this using MT +1 from me :)

[Maverik Gately]

Hi Tomer,

I've read that announcement and I have no problem splitting it into more queues but my real question is still: How?! I can make a million queues but I'm not the one glueing things together, MT is. So unless somebody can point me at how to tell MT to do this sort of dispatch, I'm at a complete loss. I'm running out of time to make this work and management is already considering ditching message queues completely and go back to old tightly coupled architecture. I'd really like not to go back and embrace all the neat stuff that's possible in message queue world coupled with a tasks driven framework.

I can even just drop packets for tenants that my consumer can't handle. The problem is that I can't figure out how to manage competing consumers then. I end up with multiple consumers that all respond to fanout message giving the same reply back that then has to be dropped at the other end. I'm completely open to alternative ideas as long as I can make it work and somebody can point me at some code snippet to build from.

Again, this is not a request along the lines of "do my homework for me".. but rather a pointer / guidance mini-sample that handles just one message even but handles the core tenancy problem that I have.

Kind regards,

Maverik

[Chris Patterson]

A sample at this scale given what you requested is not something I can just hack out over lunch and release.

I would suggest reconsidering your competing consumer design of authentication providers and do that as a single consumer that uses the credential information to make a smart decision about which authentication provider to use and then uses it. Fanning this out into something else is just over complicating the solution. 

__
Chris Patterson

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/0def575f-d23f-4bd9-bfab-dbaf3ce40d8f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Maverik Gately]

Hi Chris,

Thanks for responding. I feel you misunderstood a bit. The fanout isn't mine but MT's fanout topology that i was referring to. Competing consumers are meant only for redundancy (we don't want an azure scenario where one authenticator goes down for a bit of time and nobody can login anymore!). I suppose I could drop them to one consumer per tenant and use monitoring to keep their health in check. I'll have to talk it out but might be an acceptable compromise if there's no easy way out. Thanks for pointing out.

Also, about the sample: no there's no such urgency to make a sample over a lunch break but whenever there's time (and anybody who can invest that time), given the number of tenancy related questions I've seen while researching, I think it'll make an excellent sample to be added in samples repo. I was hoping I could contribute something myself when I started the research but seems I was out of my depth with this one and had to request some help. Still if I can get something going the way I really want to make it, I'll be happy to contribute a sample PR back to your repo.

Kind regards,

Maverik

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

[Chris Patterson]

Can you link the SO article? I feel like Tenant and Authentication are confusing me slightly. Since tenants are authentication providers in Azure AD. 

We are talking about different consumers for different customers based on a header/property in the message right?

__
Chris Patterson

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/1fde7bad-470d-43eb-bdca-92c57f4b069f%40googlegroups.com.

[Maverik Gately]

Ok I feel I've confused you even more! My apologies, I'll try to do this again.

1. The `TenantContext` suggestion that you mention in this SO question, I feel this is my way forward.
   
2. I mentioned Azure login issue as it just happened days back when MS had downtime with Azure but has no real relevance to the immediate issue, sorry. (My point here was to point out need for resilience but i can achieve it in another way if this is out of my depth for now)
   
3. Different Consumers per different customer based off header/property is correct. (again I believe this is what you were tackling in the above linked SO question, which is why I feel you're dead on with your TenantContext approach and I need to figure out how to move forward in that direction)

For reference this is how I visualize my problem in head (recap from earlier post):

ValidateLoginMessageMTExchange

  |  |

  |  |--TenantA-Authenticator1

  |  |--TenantA-Authenticator2

  |

  |--TenantB-Authenticator1

Vertical lines are individual Queues/Endpoints per customer and Horizontal lines on Vertical line indicates competing consumer. If I'm mistaken in this, please feel free to call me out.

Thank you again for the swift responses.

Kind regards,

Maverik

[Maverik Gately]

Thought I'll add this:

So far with my limited grip on MT3, my naive approach to solve this is by messing with queue name using `Namespace:ClassName:TenantId` combo per message. Not the most elegant way of doing things but I think this can work though I've no idea about its ramifications in long term (besides a lot of queues / exchanges per micro service -- not sure how bad a thing this can be?).

Input from any of the experienced members is welcome on this approach.

Thank you,

Kind regards,

Maverik

[Chris Patterson]

So this just happened:

https://github.com/MassTransit/MassTransit/blob/a5aa866a8b95ee78b70fd53c7602ad5bdc790cd4/src/Containers/MassTransit.Containers.Tests/Autofac_TenantContainerSpecs.cs

It's still in develop, and not released yet, but essentially the support we have for multi-tenancy (where different tenants have their own lifetime scope off of a parent lifetime scope) so that per-customer dependencies can be maintained (and in some cases, using InstancePerLifetimeScope(tag) to keep the instance around to avoid the first-call impact on every subsequent call).

Hopefully this helps.

Maverik

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/0def575f-d23f-4bd9-bfab-dbaf3ce40d8f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/1fde7bad-470d-43eb-bdca-92c57f4b069f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/62a57eb6-9ba9-4aa2-bdfc-fdfb17365bfc%40googlegroups.com.

[Maverik Gately]

Hi Chris,

Following your commit, I've managed to create the sample I intended to contribute to your project and the Consumer side (Authentication validators - the repositories changing per tenant using .Named<IRepository>(tenantId) pattern) are working like a charm. Thank you very much!

However, I've now run into a new snag: clients requesting validation get responses from all other tenants too. This makes perfect sense to me considering they're all just listening in on the main message type and aren't yet aware that they should only be listening to a specific tenant identifier.

Will it be possible to point me in the direction of any other commit you may have that shows how to solve this?

I've uploaded everything I've managed so far here and code review / advice is very welcome who'd like to offer it: https://github.com/Venomed/MassTransit-MultitenantCredentialValidator

(The project is using your CI build nugets to access your most recent commits)

I've opened an issue to deal with client side multi-tenancy here: https://github.com/Venomed/MassTransit-MultitenantCredentialValidator/issues/1

Many thanks for all your hard work and help :)

Maverik

PS: I noticed Autofac.Multitenancy is coming in autofac 4.0.0 as a core feature. Are there any plans to leverage that in foreseeable future? Seems like MultitenantContainer may solve some of the things I've been trying to do (mainly dependency override per tenant).

[Maverik Gately]

Hi all,

So I figured out my mistake (i was using Publish to message back to client rather than using Send) but now I'm stuck with a new problem: How do I figure out the Send(Uri, T)'s Uri bit on a remote host? 

All the examples are using typical trivial localhost send receive messages & internet is littered with MT2 ReceiveFrom examples. Is there any extension method on ConsumerContext or even on Sender bus that gives me an address to post directly back to?

I've tried client side's bus.Address as the send address but I'm clearly going something wrong as I can't consume the message sent on bus address.

Any tips on how to approach this please?

Kind regards,

Maverik

[Chris Patterson]

You can just use Respond() and it will automatically use the ResponseAddress or SourceAddress as the destination. Otherwise, it will publish.

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/2c11d424-868e-449e-a399-d4d0ffae7614%40googlegroups.com.

[Dru Sellers]

I'm pretty sure there is a method on the context

--

You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-dis...@googlegroups.com.
To post to this group, send email to masstrans...@googlegroups.com.

[Maverik Gately]

Thank you both. Respond works though I think i need to dig more as I'm ending up in a publish scenario when responding. I think I need to explore Request/Response model on client side. I'll keep reading.

On Monday, 14 December 2015 18:30:27 UTC, Dru wrote:

I'm pretty sure there is a method on the context

On Mon, Dec 14, 2015 at 11:49 AM, Maverik Gately <ven...@gmail.com> wrote:

Hi all,

So I figured out my mistake (i was using Publish to message back to client rather than using Send) but now I'm stuck with a new problem: How do I figure out the Send(Uri, T)'s Uri bit on a remote host? 

All the examples are using typical trivial localhost send receive messages & internet is littered with MT2 ReceiveFrom examples. Is there any extension method on ConsumerContext or even on Sender bus that gives me an address to post directly back to?

I've tried client side's bus.Address as the send address but I'm clearly going something wrong as I can't consume the message sent on bus address.

Any tips on how to approach this please?

Kind regards,

Maverik

On Friday, 11 December 2015 17:31:27 UTC, Maverik Gately wrote:

Hi Chris,

Following your commit, I've managed to create the sample I intended to contribute to your project and the Consumer side (Authentication validators - the repositories changing per tenant using .Named<IRepository>(tenantId) pattern) are working like a charm. Thank you very much!

However, I've now run into a new snag: clients requesting validation get responses from all other tenants too. This makes perfect sense to me considering they're all just listening in on the main message type and aren't yet aware that they should only be listening to a specific tenant identifier.

Will it be possible to point me in the direction of any other commit you may have that shows how to solve this?

I've uploaded everything I've managed so far here and code review / advice is very welcome who'd like to offer it: https://github.com/Venomed/MassTransit-MultitenantCredentialValidator

(The project is using your CI build nugets to access your most recent commits)

I've opened an issue to deal with client side multi-tenancy here: https://github.com/Venomed/MassTransit-MultitenantCredentialValidator/issues/1

Many thanks for all your hard work and help :)

Maverik

PS: I noticed Autofac.Multitenancy is coming in autofac 4.0.0 as a core feature. Are there any plans to leverage that in foreseeable future? Seems like MultitenantContainer may solve some of the things I've been trying to do (mainly dependency override per tenant).

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

[Maverik Gately]

I'm happy to report that I've finally got this working the way I wanted. There's definitely room for improvement, but It's enough PoC for me to get started on real implementation.

Final PoC is available at https://github.com/Venomed/MassTransit-MultitenantCredentialValidator

I'd love to offer my lessons learned as a helping hand to anybody who wishes to handle a similar situation so would appreciate if you guys could validate what I've done as correct (whenever you find time) and would be honoured if the sample work is good enough to join official samples list. No hurries of course!

Thank you for bearing with me while I navigated through all of this.

Kind regards,

Maverik

[Chris Patterson]

If they are using the same host/vhost and/or namespace, then they would communicate with each other. Depends upon the host configured for the broker.

On Wed, Mar 21, 2018 at 12:31 PM, Sipke Schoorstra <si...@ideliverable.com> wrote:

Hi Chris,

Does that mean that if I register one IBus instance in a child service container A (tenant A), and another one in child service container B (tenant B), and I publish a message within tenant B, only consumers within tenant B will receive that message?

Op dinsdag 8 december 2015 06:38:12 UTC+1 schreef Chris Patterson:

Maverik

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/0def575f-d23f-4bd9-bfab-dbaf3ce40d8f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/1fde7bad-470d-43eb-bdca-92c57f4b069f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsubscribe...@googlegroups.com.

To post to this group, send email to masstrans...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/62a57eb6-9ba9-4aa2-bdfc-fdfb17365bfc%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "masstransit-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to masstransit-discuss+unsub...@googlegroups.com.
To post to this group, send email to masstransit-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/masstransit-discuss/821cd690-32d8-43a1-aac5-723923b512cd%40googlegroups.com.