Sadly, leaving crossbar and wamp is imminent

[Greg Keys]

I'm sad to say that we have determined we need to migrate off of crossbar and the wamp protocol.

The number 1 reason is that crossbar does not scale well and it doesn't appear that it ever will. 

It's currently our linch pin, when crossbar crashes it takes our entire site with it.

There is some promise with the nexus router https://github.com/gammazero/nexus/issues/144 

but there isn't enough movement on this to stay our fears or fix our current issues.. 

I've considered deploying duplicate stack clusters and add load balancing to our clients but this is a costly and potentially problematic solution.

We are seriously looking into gRPC as a replacement, Im wondering if anyone else is having scaling issues?

Has anyone made the change from crossbar to gRPC?

[Zaar Hai]

Have you tried Wiola? I didn't with not too great success, but YMMV. The architecture looks promising though.

[Gareth Bult]

Hi Greg,

As a long-term Crossbar user, I feel your pain and indeed posted a similar question here three years ago.

That said, there is a lot happening on Crossbar that isn't really visible at the moment, maybe I could tempt you to hold-fire for a short while?

Scaling is now very much on the roadmap from a Crossbar perspective, and I believe there is "currently" work happening on the WAMP spec to facilitate this.
(I think there a developer discussion later today to cover some of the WAMP spec stuff)

In terms of Crossbar itself, there's a lot of new (maybe hidden?) functionality that is becoming available, some will be in the OS version and some will
only be available in the 'Enterprise' version, I'm still waiting to hear exactly how this will be split.

So, this is a screenshot from the new Crossbar UI showing;

• A connection to a Crossbar Fabric Controller
  
• Three management realms (currently selected "MyRealm")
• Two connected Crossbar Nodes (Node1/Node2)
• .. etc
• Screen is looking at the docker instances it can see (split over the two nodes) [Crossbar FX has docker management built-in]
• Tab to the right lists instances and allows containers to be created
• Current screen allows stop, start etc
• Current screen is showing an in-UI console pane connecting to the docker console of the docker container called "Demo1" running on "Node1" .. over WAMP. (and the container is running a standard Crossbar cookiecutter example app)
  i.e. [ UI -> pane within UI (xterm.js) -> WAMP -> Crossbar Fabric Controller -> WAMP -> Node1 -> Dockerd -> Container -> Cookie Cutter App ]
• 'Start shell' option on the console window lets you start a new tab in the frame that runs a shell in the same container as the application so you can debug (and alter the code) of the running application inside the running docker container - in real time.

UI is written using Vue.js and is fully reactive to the extent that someone else creates a container or "pulls" a docker image on a connected node, it'll popup in the listing in real-time.
So .. stuff *is* moving behind the scenes .. any chance you could document your specific needs / requirement within the context of scaling, maybe I can compare it to the capabilities of stuff currently under development?

Video demo here; https://www.youtube.com/watch?v=RFTRIWQovjw&t=4s

[Greg Keys]

The main features we use from crossbar are the websocket connections, rpc and pub sub. 

We don't really use realms or any other wamp/crossbar features.

We are not terribly dependent on the actual wamp protocol so much as we are the router, because of that I can't really see us using crossbar fabric

Our services are deployed and orchestrated into kubernetes clusters, we deploy a stack (the attached image represents a stack) per environment into their own namespaces

e.g. demo-dev.example.com goes into a demo-dev namespace, demo-qa.example.com goes into demo-qa etc... 

we typically run 5 environments per customer. A customer called demo would get the following environments: dev, qa, uat, blue, green

Each environment has its own crossbar router, the only environments that need HA are blue and green, when the router dies so does the entire stack (currently about 50 services).

We have been able to mitigate restarts by increasing resource limits but that can only realistically go so far on any given machine in our clusters. 

we really need to be able to run multiple instances of the router each with their own resource limits so that they can spread out in the cluster.

We've considered a lot of different options and technologies and have come to the conclusion that we could replicate our current crossbar usage with gRPC and envoy, 

envoy scales infinitely, we'd basically run envoy sidecar to all of our services which register to the envoy cluster, which provides the connections to our clients, this would allow

us to scale every aspect of our stack and eliminate the single point of failure.

If crossbar could do something similar we would not need to migrate away from it, but given it's current path im not sure it fits our needs anymore, it seems that crossbar is positioning itself 

on more of a commercial path (Congratulations on that, I understand everyone needs to pay the bills) but that doesnt really work for us because the cost would very likely be prohibitive.

[Gareth Bult]

Ok, well firstly thanks for taking the time to detail your setup, sounds like you have something fairly substantial there. Secondly I'm going to add a caveat with regards
to the next bit; the specifics of licensing have yet to be set in stone, so what I'm about to say contains a degree of personal opinion and 'ideas so far'.

If you look back in these groups, you'll see I've been a user for many users and started asking questions about scalability maybe three years ago (!) so for me, it's
right there at the top of my 'must have' list. Given my background (as an OS user) , having this as an Enterprise only feature makes no sense at all because in order

for people to see that the product 'does' scale, you need to be able to see it in action. To this end I'm expecting there to be a 'free tier' which includes HA features.

i.e. the product hasn't suddenly gone 'all commercial' :)

(which I must admit, some of the documentation might imply)

Whereas the UI is designed to be an Enterprise feature, the same sort of thing applies, if nobody can see it, getting people to pay for it is going to be difficult, so again
there will need to be some sort of free tier in order to promote adoption. If you want to give FX a try with the current public instance of the UI, let me know and I'll see
if I can sort you out with copies of stuff. (CrossbarFX is currently available as a public download)

The other thing that I had initially envisaged was "Enterprise version == prohibitively expensive" and historically I had ignored all postings relating to the Enterprise 
version for just that reason. However, my understanding is that we're talking (in terms of scale) "Google Docs pricing" rather than "Enterprise Exchange Server pricing",

i.e. on a small scale it's probably free, but if you're working on a larger scale, you pay a very modest fee for a commercially supported product. So if I were to go back
to the OS community, I'm relatively convinced that I would continue to develop HA applications on the free tier, on the basis that I would probably be happy to pay for
(and be able to pay for) the enterprise stuff should I scale that far.

Just going back to your diagram, I'd not seen a use for nor used realms prior to May this year .. however ( :) ) .. after discovering what they do and how easy they are
to use (specifically with the UI) I now use them (literally) for everything. In the context of your configuration for example, typically I would make 1 realm == 1 environment,

which means you get 5 routers / processes instead of 1, which immediately gives 5x the routing capacity vs using a single realm (?!) (not to mention 'hard' partitioning
between environments)

(the Rabbit bit is interesting, I kind of came to the conclusion queues could be done in Crossbar which makes for a single messaging library, I'd be interested to hear

about the use-case if it's something you could share?)

With regards to gRPC, I've been using this over the last month as an alternative interface to HTTP when talking to a service that supports both .. and in terms of performance
it's much better than http, maybe between 3 and 6 times faster in real terms. However .. (again in terms of performance) it doesn't seem to compare fare all that well against

WAMP / Crossbar .. if you get round to doing any benchmarking and come up with a comparison, again I'd be interested in seeing any results?

(also, I found gRPC relatively painful to develop, maybe I'm too used to Autobahn .. ;) )

With regards to a commercial path .. on the one hand there is no future in developing a product like this just for a few corporate users, but on the other hand it's not possible
to develop the product with zero income .. ideally someone would pay us lots of money to develop a completely free Enterprise version .. that may still happen .. ;)

Let me know if I can help or clarify .. 

Regards,
Gareth.

[Zaar Hai]

On Thu, 11 Oct 2018 at 21:27, Gareth Bult <garet...@gmail.com> wrote:

(the Rabbit bit is interesting, I kind of came to the conclusion queues could be done in Crossbar which makes for a single messaging library, I'd be interested to hear

about the use-case if it's something you could share?)

Gareth, can you implement persistent queues with crossbar? (those where subscriber can pick up messages he missed while being down)

 

 

--
You received this message because you are subscribed to a topic in the Google Groups "Crossbar" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/m0yE-wNAbYU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/f7200fb0-500c-4e83-a4b4-76708c9830a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Zaar

[Gareth Bult]

Hi Zaar,

Ok, bear in mind "came to the conclusion .." so I've not done this myself, but previously when looking at infrastructure options, that was my thinking.

Firstly you can obviously do this yourself 'relatively' easily depending on your architecture with a simple persistent store and a "get_since" call, however
crossbar has a message history option;

https://crossbar.io/docs/Event-History/

Again depends on your architecture / requirement .. but I figured from the available options, "another" messaging library wasn't necessarily a requirement ...

.. one of the issues we're seeing atm is documentation and it's organisation .. there are *lots* of lesser-known features kicking around in Crossbar
which aren't maybe as easy to find as they could be .. :)

hth

Gareth

[Greg Keys]

Ok, well firstly thanks for taking the time to detail your setup, sounds like you have something fairly substantial there. Secondly I'm going to add a caveat with regards
to the next bit; the specifics of licensing have yet to be set in stone, so what I'm about to say contains a degree of personal opinion and 'ideas so far'.

If you look back in these groups, you'll see I've been a user for many users and started asking questions about scalability maybe three years ago (!) so for me, it's
right there at the top of my 'must have' list. Given my background (as an OS user) , having this as an Enterprise only feature makes no sense at all because in order

for people to see that the product 'does' scale, you need to be able to see it in action. To this end I'm expecting there to be a 'free tier' which includes HA features.

i.e. the product hasn't suddenly gone 'all commercial' :)

(which I must admit, some of the documentation might imply)

Glad to hear its hasn't gone all commercial, that is the sense I got when reading the 11,000 per node in the documentation

 

Whereas the UI is designed to be an Enterprise feature, the same sort of thing applies, if nobody can see it, getting people to pay for it is going to be difficult, so again
there will need to be some sort of free tier in order to promote adoption. If you want to give FX a try with the current public instance of the UI, let me know and I'll see
if I can sort you out with copies of stuff. (CrossbarFX is currently available as a public download)

 

I keep wanting to try FX but havent really had a chance, it's hard to justify investing more time in something when 

it's future is so unclear. Any free time I have has been spent and defining our needs and looking for things to meet them. 

The other thing that I had initially envisaged was "Enterprise version == prohibitively expensive" and historically I had ignored all postings relating to the Enterprise 
version for just that reason. However, my understanding is that we're talking (in terms of scale) "Google Docs pricing" rather than "Enterprise Exchange Server pricing",

i.e. on a small scale it's probably free, but if you're working on a larger scale, you pay a very modest fee for a commercially supported product. So if I were to go back
to the OS community, I'm relatively convinced that I would continue to develop HA applications on the free tier, on the basis that I would probably be happy to pay for
(and be able to pay for) the enterprise stuff should I scale that far.

Just going back to your diagram, I'd not seen a use for nor used realms prior to May this year .. however ( :) ) .. after discovering what they do and how easy they are
to use (specifically with the UI) I now use them (literally) for everything. In the context of your configuration for example, typically I would make 1 realm == 1 environment,

which means you get 5 routers / processes instead of 1, which immediately gives 5x the routing capacity vs using a single realm (?!) (not to mention 'hard' partitioning
between environments)

I tried really hard at the beginning to use realms but they only ever complicated things for us. especially now that

we are using namespaces in kubernetes clusters. We'd have to stop deploying crossbar in the namespaces and give it it's 

own namespace, this would make it a single point of failre for 5 environments versus 1 which only compounds the problem

(the Rabbit bit is interesting, I kind of came to the conclusion queues could be done in Crossbar which makes for a single messaging library, I'd be interested to hear

about the use-case if it's something you could share?)

we are actually working on eliminating rabbitmq ourselves, it's a huge pain, although we will still use a queue system;

we are currently looking at amazon sqs. 

We try to only use the bare minimum features in crossbar, my thinking on that comes from several recommendations from micro service experts to keep your pipelines dumb and build the smarts into 

your services, this is a good read https://medium.com/@nathankpeck/microservice-principles-smart-endpoints-and-dumb-pipes-5691d410700f

 

With regards to gRPC, I've been using this over the last month as an alternative interface to HTTP when talking to a service that supports both .. and in terms of performance
it's much better than http, maybe between 3 and 6 times faster in real terms. However .. (again in terms of performance) it doesn't seem to compare fare all that well against

WAMP / Crossbar .. if you get round to doing any benchmarking and come up with a comparison, again I'd be interested in seeing any results?

(also, I found gRPC relatively painful to develop, maybe I'm too used to Autobahn .. ;) )

Regarding gRPC, the most attractive thing to me about that is actually envoy, if crossbar were to become more like envoy

that would probably solve all our problems. things like hot restarts without loosing connections, and drop dead simple 

clustering, where we could add new instances and it would just work. 

With regards to a commercial path .. on the one hand there is no future in developing a product like this just for a few corporate users, but on the other hand it's not possible
to develop the product with zero income .. ideally someone would pay us lots of money to develop a completely free Enterprise version .. that may still happen .. ;)

Agreed, Im always curious how so many companies give their product away and still make money. I guess its because what they give away isn't really their primary product?

[Gareth Bult]

Hi Greg,

> Glad to hear its hasn't gone all commercial, that is the sense I got when reading the 11,000 per node in the documentation

Erm, this page?

https://crossbario.com/products/enterprise-support/

This is the Enterprise support package .. (there are some *very* large Crossbar users about) .. whereas I rather suspect that may need to be updated, I believe it's quoting the price for support, as opposed to the product.
(but I take your point, some more definitive pricing would be good)

> I keep wanting to try FX but havent really had a chance, it's hard to justify investing more time in something when it's future is so unclear. Any free time I have has been spent and defining our needs and looking for things to meet them.

Mmm, I'm getting a distinct sense of Deja vu .. I think I may've said the same thing myself ... 

> We'd have to stop deploying crossbar in the namespaces ..

I've a CrossbarFX deployment on Google/K8, not sure the use of multiple realms would make any difference in that context ... (?)

> We try to only use the bare minimum features in crossbar

It is interesting, on the one hand if the intelligence is "all" in the client, then you miss-out on some of the optimisations available by doing bits in the pipeline, and on the other hand you're potentially far more portable and any bugs will be easier to find as they will be in local code.

> Regarding gRPC, the most attractive thing to me about that is actually envoy,

Yeah, I had a look at this when you mentioned it, looks really cool .. on the downside (for me) it looks like it's http2 only and doesn't support websockets (?)

gRPC does seem to be getting a lot of press / traction, it'll be interesting to see how it develops .. 

> Agreed, Im always curious how so many companies give their product away and still make money

Well I guess it's different for every company, but I think in general, support contracts with large customers and custom coding projects probably figure. I think there

are larger companies, without core developers, or with staff retention issues, who use free software. So having someone on the end of a phone who can fix

anything for a price, is an attractive safety net for the well heeled .. :)

I'll see if I can find out if there's anything available yet re; pricing plans ...

[Zaar Hai]

On Fri, 19 Oct 2018 at 11:12, Gareth Bult <garet...@gmail.com> wrote:

> Agreed, Im always curious how so many companies give their product away and still make money

Well I guess it's different for every company, but I think in general, support contracts with large customers and custom coding projects probably figure. I think there

are larger companies, without core developers, or with staff retention issues, who use free software. So having someone on the end of a phone who can fix

anything for a price, is an attractive safety net for the well heeled .. :)

I'll see if I can find out if there's anything available yet re; pricing plans ...

Yes, please keep us posted once you have anything. The one company I have in mind is Elastic (did an IPO couple of weeks ago) which have their ElasticSearch database open-sourced since day one, including scaleability;

and they provide support and enterprise goodies like authn/z and SSL for a fee. 

Though their model is "all or nothing" and "all" comes for rather high price starting from 5k USD per node per year - that's why those 11k for crossbar rang a bell, at least for me.

[Gareth Bult]

Current Crossbar FX /CFC pricing can be found here; https://crossbario.com/products/edge_and_cloud/ ...

Little bit cheaper .. than enterprise support .. :)

[Zaar Hai]

That's really interesting.

If all my "sensors" talk to the internet, I don't need any Edge devices, right? Same for users coming from web-browser - they just connect to Crossbar.io Fabric Cloud directly, right?

Also, for Router in Starter pack it says "1 x 4 cores". Does it mean that crossbar router scales across multiple cores for a single realm?

--

You received this message because you are subscribed to a topic in the Google Groups "Crossbar" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/m0yE-wNAbYU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/0e4d482a-4e65-4bbd-8a9d-a1d11ddcab83%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zaar

[Gareth Bult]

Mmm, not quite, all your sensors will talk to a (CrossbarFX) edge node, but you'll get a lot of connections per node ...

CFC is a mechanism for linking / controlling all your (CrossbarFX) edge nodes, rather than for connecting sensors to.

(which comes with a UI for doing dynamic config etc)

To be honest, I don't understand the 'cores' specifications so I'm going to pass on that one .. 

If you have a potential configuration, I'm happy to try to clarify the configuration / cost .. 

[Zaar Hai]

Te "cores" was actually the most interesting part :) I hope the below snapshot (from the link you sent) will help to explain what I am asking about.

So, apologies if I'm taking the discussion off-topic, but I would like to understand the product.

You say CFC is for "linking / controlling edge nodes". What do you mean by "linking"? - Will I be able to have one WAMP realm spread over multiple Edge Crossbars?

Also, the page says "Crossbar.io Fabric Edge is priced per edge device" - does it mean "per edge crossbar router"? or "per edge sensor/actuator/etc. that connects to edge crossbar router"?

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/80de67e9-e8d2-4ab6-9220-e379acd3b02e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zaar

[Gareth Bult]

Yeah sorry, I'm going to have to ask specifically how the cores / pricing relate.

The idea is to have one WAMP realm spread over multiple Edge Nodes, indeed one of the challenges I'm looking at currently is how to get the UI to cope with 1M edge nodes on 1 realm.

I know there have been lots of discussions around pricing, and looking at the table I can see how it could be interpreted in different ways .. let me see if I can an authoritative comment / answer and come back to you ... :)

[Tobias Oberstein]

Hi Zaar,

let me chime in;) I'll add some infos ..

Am 22.10.2018 um 09:52 schrieb Zaar Hai:
> Te "cores" was actually the most interesting part :) I hope the below
> snapshot (from the link you sent) will help to explain what I am asking
> about.
>
> So, apologies if I'm taking the discussion off-topic, but I would like
> to understand the product.
> You say CFC is for "linking / controlling edge nodes". What do you mean
> by "linking"? - Will I be able to have one WAMP realm spread over
> multiple Edge Crossbars?

"linking"

this means an edge node maintains a management connection (WAMP) to CFC
so that it can be remotely managed and monitored. this connection does
not carry any app traffic - only control rgd crossbar itself.

"one realm spread over multiple edge nodes?"

you can define one app realm, and have all settings and configuration
automatically _replicated_ to multiple edge nodes

ultimately, you will also be able to have one app realm also _span_
multiple edge nodes, which will require at least one cloud anchored node
too (with public routable IP)

in such a topology, an edge node routes events/calls destined for the
other edge node over the cloud node (because the assumption is that the
edge nodes are not reachable from public internet)

>
> Also, the page says "Crossbar.io Fabric Edge is priced per edge device"
> - does it mean "per edge crossbar router"? or "per edge
> sensor/actuator/etc. that connects to edge crossbar router"?

the former.

an edge device (hardware, like an industrial PC, or a Pi) running a
Crossbar.io Fabric Edge node counts as one.

an Autobahn (or general WAMP) client does _not_ count (is free)

Cheers,
/Tobias

> <https://groups.google.com/d/msgid/crossbario/0e4d482a-4e65-4bbd-8a9d-a1d11ddcab83%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Zaar
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "Crossbar" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/crossbario/m0yE-wNAbYU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> crossbario+...@googlegroups.com

> <mailto:crossbario+...@googlegroups.com>.

> To post to this group, send email to cross...@googlegroups.com

> <mailto:cross...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/crossbario/80de67e9-e8d2-4ab6-9220-e379acd3b02e%40googlegroups.com
> <https://groups.google.com/d/msgid/crossbario/80de67e9-e8d2-4ab6-9220-e379acd3b02e%40googlegroups.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Zaar
>
> --

> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com
> <mailto:crossbario+...@googlegroups.com>.

> To post to this group, send email to cross...@googlegroups.com

> <mailto:cross...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/crossbario/CAGQ_5kVdqFP2B0qJ3cr0C3LNwRv_atF3%3Di4sKNMfsuyGrOiJAA%40mail.gmail.com
> <https://groups.google.com/d/msgid/crossbario/CAGQ_5kVdqFP2B0qJ3cr0C3LNwRv_atF3%3Di4sKNMfsuyGrOiJAA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Greg Keys]

That sounds quite complex. 

I have to be honest I'm a bit lost on the crossbarFX, none of it looks like it fits our needs.

I'm getting the sense that crossbarFX is moving more toward being a router in the cloud that iot can connect to, which 

I can see being a great choice for iot users. 

We don't use crossbar for iot. 

we use it more like nginx where our client connects to the router in order to access api endpoints instead of restful end points.

It's really import that we run everything on our hardware in our data centers (AWS), I can't imagine we would ever have all of our services connecting to 

a router service hosted elsewhere. We just need the router to scale so that its no longer a single point of failure.

I'm getting the sense that we are probably not the target market for crossbar, am I wrong about that?

>     <mailto:crossbario+unsub...@googlegroups.com>.

>     To post to this group, send email to cross...@googlegroups.com
>     <mailto:cross...@googlegroups.com>.
>     To view this discussion on the web visit
>     https://groups.google.com/d/msgid/crossbario/80de67e9-e8d2-4ab6-9220-e379acd3b02e%40googlegroups.com
>     <https://groups.google.com/d/msgid/crossbario/80de67e9-e8d2-4ab6-9220-e379acd3b02e%40googlegroups.com?utm_medium=email&utm_source=footer>.
>     For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Zaar
>
> --
> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com

> <mailto:crossbario+unsub...@googlegroups.com>.

[Gareth Bult]

> I'm getting the sense that we are probably not the target market for crossbar, am I wrong about that?

Target market is an interesting question. From a personal perspective I started using Crossbar (quite some time ago now) for an IoT project .. well .. sort of .. 

custom door entry system .. so lots of Raspberry Pi's with Camera's maglocks and RFID readers all talking to a central Crossbar server over WAMP. As a part

of the project I had to produce a UI showing open doors, who was in each room etc .. so I hung that off Crossbar too .. at which point it dawned on me that IoT

aside, Crossbar / WAMP was a great solution for real-time reactive websites and online applications, and that's the use-case that's sort of held my interest.

So the pundits reckon 20 billion IoT devices out there by 2021, and there are already a billion websites .. so in theory both markets have lots of potential.

... from a product capability perspective, I would say "yes" .. from a marketing perspective, maybe not at the moment (?) , but that's not so say

the use-case isn't being considered .. if I can expand a little on the basis of my understanding .. (someone will hopefully correct me if I get a bit wrong :) )

CrossbarFX comes as a binary which will run in either master or edge mode.

Master mode is for linking edge node's together (and managing edge nodes)

Edge mode provides something for your WAMP clients (and/or edge devices) to connect to.

So, to run a realm across multiple nodes you would need;

a. crossbarfx running in master mode on an ip that all desired edge nodes can connect to

b. two crossbarfxs running in edge mode such that they can connect to the master

c. the as-yet unreleased bit of software that provides same-realm routing between edge nodes via the master node

(so the realm would run on the edge nodes and the master node would provide the interconnect)

My understanding is that there will be a free-tier version of this such that you could set up two edge nodes for basic scaling, but once you start to expand, then

you expand into a paid-for per-node tier.

So, whereas FX will do a lot of things you don't need, there are some important features going in there that you might find useful. In addition to the scaling and the

delivery format (deploying Crossbar as a pre-compiled [python] binary is very convenient) FX has the ability to access and control "docker" on the machine

on which it's deployed .. so if you build your client applications / microservices as docker images, it can pull, deploy, start, stop, update and monitor your client

applications from within the UI, and changing the crossbar configuration no longer means having to edit config.json  .. :) .. 

At first glance Docker might not appear of any use of you're not a docker user, but it's effectively providing package management and partitioning for your microservices with very little fuss. 

(I wasn't really a Docker fan, however in this context it's sort of growing on me ...)

Anyway, hope this helps a little ...

[Zaar Hai]

On Tue, 23 Oct 2018 at 13:19, Gareth Bult <garet...@gmail.com> wrote:

So, whereas FX will do a lot of things you don't need, there are some important features going in there that you might find useful. In addition to the scaling and the

delivery format (deploying Crossbar as a pre-compiled [python] binary is very convenient) FX has the ability to access and control "docker" on the machine

on which it's deployed .. so if you build your client applications / microservices as docker images, it can pull, deploy, start, stop, update and monitor your client

applications from within the UI, and changing the crossbar configuration no longer means having to edit config.json  .. :) .. 

May be that makes sense in IoT, but living with Docker & Kubernetes for the last 3 years, the above statements may sound like you'll be competing with the latter :)

I already have K8s/Helm for deployment/scaling/etc., Prometheus for monitoring, Graylog for log collection and so on. For me, WAMP is just another transport bus inside my

application cluster, along with HTTP/REST, persistent pub/sub, etc. With all do respect, I find it hard to imagine considering Crossbar as my deployment manager

(if this is what you mean).

 

At first glance Docker might not appear of any use of you're not a docker user, but it's effectively providing package management and partitioning for your microservices with very little fuss. 

(I wasn't really a Docker fan, however in this context it's sort of growing on me ...)

Anyway, hope this helps a little ...

--
You received this message because you are subscribed to a topic in the Google Groups "Crossbar" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/m0yE-wNAbYU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/a819a677-3995-425d-bd95-b0f94468847e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zaar

[Greg Keys]

Agreed, it does appear crossbarFX is going down the path of orchestration, scheduling and management.

We too are using kubernetes for that. We recently switch from using Rancher cattle, Im not looking to change to yet another

orchestrator, kubernetes does everything we ever wanted and more.

[Tobias Oberstein]

Hi Greg,

I guess there are 2 different use cases here:

A. for IoT scenarios, where people want to deploy and manage apps onto edge devices, which are essentially a whole different layer of "cloud" (not a regular data-center), and CrossbarFX nodes on edge devices are part of a bigger application spanning _also_ to datacenter

the management/orchestration features Gareth was talking about are mainly aimed at A. - of course we have no plans to replicate what k8 already does!

B. datacenter-only scenarios

for B. we have "phase 1" of the architecture below in the queue: proxy workers. these are fully WAMP aware (and WAMP authenticating) reverse WAMP proxy workers that terminate incoming WebSocket/TLS and forward to router workers based on (realm, authid). the proxy workers are dynamically managed (eg where to forward to) by the controller. the complete picture rgd CrossbarFX scaling (and HA) in the datacenter:

Cheers,

/Tobias

[Gareth Bult]

> May be that makes sense in IoT, but living with Docker & Kubernetes for the last 3 years, the above statements may sound like you'll be competing with the latter :)

Yeah, I could see how that might look :)

But the idea is really to work "with" your chosen infrastructure rather than trying to replace it. The first time I ran up FX / UI on Kubernetes it became very
apparent from the 30-40 support containers that appeared that a pure docker interface on Kubernetes really didn't make sense as it would be totally unsafe

to try to manage Kubernetes infrastructure from within Crossbar. So whereas the current FX has a docker driver, ultimately the idea is that this to make this

generic such that it supports multiple drivers depending on the platform. (i.e. K8, AWS Elastic containers, Docker, Cloud Foundry, Heroku etc) So whereas

you may just want to use your chosen cloud platform and use Crossbar as a messaging bus, for some people an optional interface to interact with their local

platform or environment is quite useful.

.. but this is just one feature, if you don't need that bit, all you have to do is not enable it .. ;)

That said ( :) ) from a HA/resilience perspective, supporting Nodes on two platforms (say K8 and AWS) and being to launch a single realm across the two nodes

and load-balance/fail-over between the two, might be attractive for users who don't want to put all their eggs on one platform (?)

[Gareth Bult]

I think it would be fair to say that CrossbarFX is looking to provide options that allow it to integrate with the platform it's

running on, rather than only providing a solution that assumes it's always running in isolation. I'm not seeing any suggestion that 

Crossbar shouldn't work on Kubernetes, I have a K8 cluster on Google running a CFC/FX setup, indeed I have a Helm script

that deploys the setup from scratch .. personally I find the overhead a little on the 'heavy' side, but in principle it works fine.

But once I have a bunch of nodes deployed, and a bunch of WAMP clients connecting to those nodes, from the CFC perspective
I can "see" all resources on all nodes .. so I can monitor cluster-wide sessions for all nodes, aggregated user connections on all

nodes .. I can subscribe to any topic on any/all nodes and monitor events in real-time etc.. and of course I can reconfigure any

node on the fly .. none of these features are platform dependent, but are pretty useful, at the very least from a development perspective.

If you just have one Crossbar node then monitoring and managing it is a relatively known quantity .. once you have a number of 

connected / cooperating crossbar nodes, doing it all by hand becomes more of a challenge .. so I guess this is one of the issues FX is 

trying to address.

The Docker module specifically is really aimed at users running on raw virtual machines, rather than users with their own container

management infrastructure, so raw AWS or DO users for example.

[Greg Keys]

trying to use a single cluster of crossbar instances for all environments and all customers would add an exponential amount of complexity for us,

the way it works for us now is very simple, we basically use crossbar for connections and routing, the only piece of the puzzle we are missing is the HA

If we have to re-engineer our stack in order to achieve that HA there is a good possibility we'll just move away from crossbar. Please understand we

do not want to move away from crossbar, its been very good to us so far, it's the unknown future that's bringing up all these questions and concern

[Gareth Bult]

Ok, I'm just trying to get a feel for what HA would look like for you .. (load balancing aside for a second just to keep it simple) .. would it be something like this,

where multiple crossbar's share pub/sub/rpc for a common realm, with WAMP clients either connecting to multiple Crossbar's concurrently and using one

that's available?  (or for more scale, the WAMP client has a list of Crossbar instances and connects round-robin until if finds one that works)

[then rolling updates to Crossbar itself so there's always at least one Crossbar up]

Or something else?

[Zaar Hai]

Here is what I mean - something a bit simpler:

Or K8s version if you prefer:

Today the only missing piece is the interconnected crossbar routers.

If you support that, it would really be enough - I can handle everything else using other tools (monitoring, config mgmt, etc).

Again, I'm a single-realmer :)

--

You received this message because you are subscribed to a topic in the Google Groups "Crossbar" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/m0yE-wNAbYU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.
To post to this group, send email to cross...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/58e4b990-9901-4e67-b45c-4295329a273c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zaar

[Gareth Bult]

Ok, this makes sense, sort of what I was looking at, but you're handling client side with your own load balancer.

So .. I have a couple of questions if you would indulge me .. :)

With HTTP and a load-balancer, when a server goes down, handling fail-over is relatively straightforward in terms of sharing a session

(and maybe db connections) between servers .. every request is potentially a new connection anyway .. however in this scenario with

Crossbar, it's a persistent connection so taking out a router will actually sever the connection and force a reconnect. So .. (!) .. firstly is this

acceptable? Do your WAMP clients for example make use of the Crossbar message history, so when you reconnect, do you examine the history

and do a 'catchup' on pub/sub events potentially missed during the re-connection phase? 

(or, would you see this sort of thing as a new feature to be handled within Autobahn?)

Secondly (please don't shoot, it's an honest question! ;) ), do you think this feature should be a free / Open Source component of Crossbar,

or would you see this more of commercial feature with an attached license fee? i.e. from your perspective, is this something you would be

prepared to pay 'something' for, or would you expect this to be provided as a part of the free / Open Source package?

[Zaar Hai]

Hi Gareth,

A disclaimer: we only start incorporating WAMP into our existing infrastructure, though I'm watching this project for several years already. As a general rule of thumb, we don't plan on using any features that are not available in similar products. It's that right now, WAMPs provides the best package. Consequently, we try not to plan on any features that are specific to corssbar.io router (or at least not easily implementable on other WAMP routers).

So for restarts, if one router goes down/restarted, its clients on the other side will have to reconnect and LB will forward them to another, healthy router. Your original slide suggests that you'll fail-over inside a client, so it will be transparent to the client app, right?

However industry seems to be moving towards "smart clients, dump pipes", and this aligns with my personal experience. Hey, eventually there still be cross-system hiccup, so I'll have to handle it in my app code anyways; so what's the difference? (I mean, there is a difference, during router rolling updates for example, but does it worth the complexity?)

Similarly for pub/sub - we are not interested in message history - we'll use pub/sub only for updates and REST to backfill the data (e.g. on page load or reconnect). I.e. we won't be looking into a pub-sub with message persistence, since it introduces operational complexity (routers now have a state!) and feels too unique.

To answer for your crème de la crème question, I'll put a disclaimer first that what's written below is my personal opinion, and it's just that - an opinion. So without a further ado I'm shooting it:

I think that if crossbar router will not support scale-out pretty soon as an open-source product, it will exacerbate its niche-ness and community will simply move on either:

• Either alternative WAMP router that provides just "simple" scalability. E.g. Wiola - I failed to get it running, but I believe it can be polished to do the job with reasonable amount of effort (Yes, Redis is not the easiest platform to scale, but it's a "solved problem" and there are plenty SaaS solutions for Redis)
• Or just to alternative pub/sub + RPC solution. For example - socketcluster.io, deepstream.io, nats.io. Yes, they are not providing a complete WAMP-like feature set, but they are getting close...

I think developers these days prefer scalability & elasticity way above other features. We don't want to cherish our precious services anymore. We want cattle, or may be even "bacteria" that dies and breeds all the time and we don't care as soon the swarm is doing the job.

I apologies for foretelling bad news :), but again, this is only my opinion. And I think it's not that bad overall. Looks at Elastic.co - they provide completely functional open-source product (majority of their development is open-sourced actually) yet they still made enough revenues from support contracts / premium features to make an IPO. Or may be I just encouraged you to leave the cloud alone and concentrate on IoT where you may have a higher leverage :)

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/ff266143-070f-424a-890d-1e53a99b65b6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zaar

[Gareth Bult]

Hi Zaar,

Mmm, "smart clients dumb pipes" plays very well in a DevOps / pipeline context (as does the single realm model)  as you consciously

want 100% separation between environments. That said, there is a question of "how dumb"?

From a Crossbar perspective, I think it's a "smart pipe", but you can enable the various degrees of smartness .. (or not). So it will behave

like a "dumb pipe" if you want it to, however for users who want it to do more .. it will .. :)

From my perspective I like the idea of fail-overs being totally transparent to the client, which removes some edge case handling

or 'what do I have to think about if I have to connect to a different crossbar'. This can (in theory) be done if the client holds  > 1 crossbar

connection open. If you're set up to handle disconnections as 'restarts', that should work fine either way so long as it's an acceptable

outage from the application's perspective.

Personally I understand the point you make with regards to simple scaling, and I don't disagree.

From your point of view, would you see "simple" as a two node solution, or an (n) node solution?

Just as an aside on the client restart issue, if the client is able to get a full-state-snapshot quickly on a reconnect, as a solution this

would seem to be workable with regards to relatively low numbers clients surviving restarts. If on the other hand current-state is

'substantial', the ability to reconnect while just obtaining incremental / missed events, might be appealing from a performance 

perspective. If a state refresh is a few MB's, and you have 10,000 clients switching from CrossbarA to CrossbarB, and each client

just has to request a message or two for the fraction of a second it's disconnected, this is relatively trivial. If on the other hand the

server has to perform a state generation routine 10,000 times and transmit a few MB's 10,000 times, it becomes a less trivial operation .. 

(different use-cases, different perspectives .. :) )

Still it's got me thinking, specifically, "what if we had 'two' scaling options" .. :)

Gareth.

[Zaar Hai]

Hi Gareth,

Please see inline below

On Thu, 25 Oct 2018 at 05:00, Gareth Bult <garet...@gmail.com> wrote:

Hi Zaar,

Mmm, "smart clients dumb pipes" plays very well in a DevOps / pipeline context (as does the single realm model)  as you consciously

want 100% separation between environments. That said, there is a question of "how dumb"?

From a Crossbar perspective, I think it's a "smart pipe", but you can enable the various degrees of smartness .. (or not). So it will behave

like a "dumb pipe" if you want it to, however for users who want it to do more .. it will .. :)

From my perspective I like the idea of fail-overs being totally transparent to the client, which removes some edge case handling

or 'what do I have to think about if I have to connect to a different crossbar'.

Well, I think it does not really remove edge cases, because you always need to handle "what happens on reconnect" situation - crossbar restarts aside,

a user can just walk with his laptop to a lift and out, or be on the moving train with flaky LTE reception.

 

This can (in theory) be done if the client holds  > 1 crossbar

connection open. If you're set up to handle disconnections as 'restarts', that should work fine either way so long as it's an acceptable

outage from the application's perspective.

Don't get me wrong - I think >1 crossbar connection makes sense as a feature. It's just a nice-to-have feature IMHO, and my point that it does not save

application code development - may be only smooths user experience in some cases.

Personally I understand the point you make with regards to simple scaling, and I don't disagree.

From your point of view, would you see "simple" as a two node solution, or an (n) node solution?

N-nodes. If it says it scales, then it scales :) Following the above thoughts, I would prefer to have larger amount of smaller crossbars to minimize

"disconnect impact" of a single node failure.

 

Just as an aside on the client restart issue, if the client is able to get a full-state-snapshot quickly on a reconnect, as a solution this

would seem to be workable with regards to relatively low numbers clients surviving restarts. If on the other hand current-state is

'substantial', the ability to reconnect while just obtaining incremental / missed events, might be appealing from a performance 

perspective. If a state refresh is a few MB's, and you have 10,000 clients switching from CrossbarA to CrossbarB, and each client

just has to request a message or two for the fraction of a second it's disconnected, this is relatively trivial. If on the other hand the

server has to perform a state generation routine 10,000 times and transmit a few MB's 10,000 times, it becomes a less trivial operation .. 

(different use-cases, different perspectives .. :) )

Having several MB worth of state IMHO puts your app in a realm (pun intended) of distributed state/record/database system, and if I couldn't

avoid it all together, I would think about using dedicated tools for that (Firebase, GUN).

From my experience, delta-only approach will always go out of sync eventually. Therefore I treat pub-sub updates as a mere gap-filler that

improves UX responsiveness between full syncs going on background periodically. E.g. if I display live network bandwidth chart, I would use

pub/sub to display updates as they come every second, but would issue REST call to my API server every minute or so to backfill data from the

source of truth. (That's only one use case of course).

 

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/92301872-2c5e-4821-9e09-8b1c837482e0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Zaar

[Gareth Bult]

Hi Zaar,

Re; Edge cases .. I guess it depends on your perspective, but to me, "I've switched to a new crossbar" is quite different

from "I have no more crossbars", and the latter is (hopefully) way less likely and potentially a lot easier to deal with. At the

end of the day, scaling shouldn't really be dependent on the client connection, so multi-connection and single through a

load balancer should both be valid potential options against a scalable crossbar core.

n-Nodes .. scalability means scalability .. yeah, I don't really disagree there either .. :)

>From my experience, delta-only approach will always go out of sync eventually

Ok, I think we may we differ pretty wildly on that one, I work on the basis that a lost event is an error :)

Anyway, many thanks for the feedback, you've now got me (and maybe one or two other people) thinking .. :)

Gareth.

[Zaar Hai]

You are most welcome. Thank you for listening. Looking forward for news :)

To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/19285918-054e-459e-8c47-f8485b4c1234%40googlegroups.com.

[Greg Keys]

As an enterprise company we do pay for services that provide a unique commercial value.

For example, 

- security, we pay for a commercial security product because security is hard with containers and paying for a commercial product makes sense, (twistlock, stackrox, sysdig secure etc....)

- logging, metrics and alerts - similar to security we have tried to roll our own methods for handling these (icinga, prometheus, syslog etc..) but they fail in comparison to products that combine them all and offer really nice features

  like datadog, sematext, new relic etc. 

Things that are very difficult to replicate at a commercial scale are the things that we pay to use. 

Unfortunately with crossbar there are alternatives, granted they may not offer everything that crossbar does, but there are good alternatives non the less. 

There are other WAMP routers as well as other comparable technologies e.g. gRPC and envoy.

The most appealing technology for us right now is gRPC and Envoy, primarily because of envoy (which acts as the router in this scenario) it does

everything we need it to do. Granted we would take a hit on some features by switching to gRPC but the HA pay off would warrant refactoring our services around 

the missing features (which are on the roadmap btw) 

With options,  it's very hard to justify paying crossbar a licensing fee just for HA. Or dealing with a limit on the HA nodes, that just serves as an annoyance.

I personally tend to switch products when they get annoying. e.g. rabbitmq

I could potentially see us paying for crossbarFX as an interface to look at all of our crossbar instances in a single location to monitor sessions,

debug issues and tune performance etc... that would have some value for us. deploying docker containers does not appeal to us at all, we have kubernetes for that.

Trying to charge for things we can get for free elsewhere will only serve to drive us into the hands of your competitors

If you want to build your community, grow your product and ultimately get paid I think you need to compete at the same level as your

competitors. 

Docker, for instances gives away so much of their code, but at the same time they have a lot of SaaS products that offer really nice features.

Those SaaS features are very difficult to replicate, those things justify the cost.

In summary, I think crossbar should make the router as flexible as possible and give it away to gain mass adoption. 

Then build SaaS services that compliment that product as the profitable business model. 

Regarding how we need the HA to work.. 

I need to be able to add additional instances of the router (1,3 or 5 instances at minimum). 

my services and clients should be able to connect to any of the instances (not all of them at once but any of them at one time)

my services and clients should be able to call any uri and receive any pub/sub regardless of which router in the cluster that its connected to.

Optional features that would really be a home run,

- the ability to restart/upgrade the router without loosing any connections.

- working similar to envoy where we can deploy a mini crossbar sidecar with our services, which in turn connects

  to all the other mini crossbar instances to form a single cluster. this way our services would always just connect to localhost

  rather than a particular url  (this looks really appealing for development)

[Gareth Bult]

Hi Greg,

Personally, in general I think we're on the same page .. :) .. but a couple of questions if I may?

- assuming the HA you describe, the sidecar option I guess would be implicit, however any sort of sidecar

  implementation would be a performance hit to an extent as it would rely on an additional "hop" in terms of

  message forwarding .. do I take it from this that performance is not a critical factor for you?

- when you say 'without losing any connections', do you have an idea of how this could work? I'm just thinking of

  all the associated structures and dependencies associated with a connection that might change if the underlying

  software was updated and the time it takes to re-establish a lot of them .. a more transparent mechanism for

  rolling updates for example might be a staggered migration to other routers prior to a shutdown?

  (i.e. what's the target in terms of not losing connections, transparency, lack of outage .. ?)

- how quickly would the HA solution need to become available to keep you interested in Crossbar?

- would you be interested in being a tester?

Also, in terms of the stuff you would be interested in paying for ( I gotta ask :) ) , I'm familiar with a number of the

tools you mention .. and we already have a bunch of interesting toys for network-wide event and message monitoring,

but what sort of feature-set would be interesting to you from this perspective?

Gareth.

[Greg Keys]

On Friday, October 26, 2018 at 1:59:27 AM UTC-7, Gareth Bult wrote:

Hi Greg,

Personally, in general I think we're on the same page .. :) .. but a couple of questions if I may?

- assuming the HA you describe, the sidecar option I guess would be implicit, however any sort of sidecar

  implementation would be a performance hit to an extent as it would rely on an additional "hop" in terms of

  message forwarding .. do I take it from this that performance is not a critical factor for you?

I'd take a nano second performance hit over complete unavailability any day. I did a quick drawing for how I imagine it could work.

I included both a sidecar concept and a shared roouter conecpt. its really ugly I know, hopefully you get the idea though.

- when you say 'without losing any connections', do you have an idea of how this could work? I'm just thinking of

  all the associated structures and dependencies associated with a connection that might change if the underlying

  software was updated and the time it takes to re-establish a lot of them .. a more transparent mechanism for

  rolling updates for example might be a staggered migration to other routers prior to a shutdown?

  (i.e. what's the target in terms of not losing connections, transparency, lack of outage .. ?)

I have no idea how envoy does this, but my guess is that because it HA you can replace any part of it without loosing connections. but I think its even better than that. 

- how quickly would the HA solution need to become available to keep you interested in Crossbar?

Really we just need to be confident that it's coming and that its not going to sink us, most of the problems we  have been dealing with can be fixed by putting more resources behind the routers

but I can see a day where that becomes impractical. Also we can mitigate loosing connections with scheduled maintenance windows. Our goal though is to get to a 

point where we can do zero downtime deployments for any part of our system.

 

- would you be interested in being a tester?

Yes of course.

 

Also, in terms of the stuff you would be interested in paying for ( I gotta ask :) ) , I'm familiar with a number of the

tools you mention .. and we already have a bunch of interesting toys for network-wide event and message monitoring,

but what sort of feature-set would be interesting to you from this perspective?

To be honest we haven't gotten to a point where I can answer that, I will gladly take a serious look at crossbarFX though if we 

do end up sticking with crossbar.

[Greg Keys]

is google groups dieing along with google plus because it seems like these groups wont show any images anymore?

[Gareth Bult]

Yeah, I think there is a refresh issue when you post images .. try a page reload a couple of times .. I can see your diagram :)

Ok, leave it with us for a second, there are two prototypes I'm aware of that address at least some of the requirements
you mention, but there's now more discussion with regards to scope and timing. Much of this is down to focus, priorities 

and resources, as you will appreciate, the stuff that puts food on the table is sometimes tough competition for the stuff one

actually wants to do ... :)

[Greg Keys]

That is excellent news, thank you.

[Dominique Burnand]

I just wanted to say that we suffer from the exact same limitations (HA) as you do. 

We also think that your proposed way of implementing it is the way to go (this includes the free/pricing feature set). We are also using k8s.

At the moment it is okay for us, and we do not need to scale to that extends yet. But if our product does scale, it is obvious that we need to be able to scale crossbar as well.

Just thinking of being limited by the router is forcing us to lookout for alternatives... And even at our current state: It hurts to know that the single-point of failure is crossbar.

[Ben S]

Hi Greg/Dominique, 

Just out of interest, how many many clients do you have connected?

[Dominique Burnand]

Around 200.

[Greg Keys]

Ben,

Our highest use crossbar instance is handling about 5000 clients at it's peek, this is growing every week as we roll out to more users. 

This morning we suffered a complete outage which lasted 2 hours. I don't think crossbar was to blame, it looks more like a kubernetes

node issue but it highlights the point of crossbar being a single point of failure for us.

As you can probably imagine this has become a high priority for us to solve.
I have received a recommendation to look at Bondy which is what I will be doing this week,

it doesn't look like its at a good place for automatic deployments which is what we need but it seems to be the best

course of action I have at this point in time.

Does anyone know if crossbar made any progress on a HA solution?

[Alexander Gödde]

Hi Greg!

We are making progress on a HA solution as part of our commercial offerings. Contact me at alexande...@crossbario.com to get more information about this if this is of interest to you.

Regards,

Alex

[Tobias Oberstein]

Hi Greg,

please find attached 2 screenshots of a HA setup test (4 nodes + HA
proxy + load client on 1 test box).

CrossbarFX is doing:

- delivers 180k events/sec for subscribers from publishers doing 14k
publications/sec
- load is spread across the 4 routers
- when you kill 1 of the nodes, a client affected has reconnected (to
one of the remaining nodes) and back (at the app level) immediately (<2s
here)

So this is a 4 nodes active-active setup .. the pairing into a cluster
is "per realm".

What we want to have in place before distributing that more widely first
is indeed more load testing .. and robustness testing (all new code, so
bugs to shake out ..)

Cheers,
/Tobias

PS: the box all of above setup was running is a quad core i7 intel nuc
.. nothing server like at all.

Am 28.02.19 um 10:31 schrieb 'Alexander Gödde' via Crossbar:

> Hi Greg!
>
> We are making progress on a HA solution as part of our commercial
> offerings. Contact me at alexande...@crossbario.com to get more
> information about this if this is of interest to you.
>
> Regards,
>
> Alex
>
> Am Donnerstag, 21. Februar 2019 01:28:46 UTC+1 schrieb Greg Keys:
>
> Ben,
>
> Our highest use crossbar instance is handling about 5000 clients at
> it's peek, this is growing every week as we roll out to more users.
> This morning we suffered a complete outage which lasted 2 hours. I
> don't think crossbar was to blame, it looks more like a kubernetes
> node issue but it highlights the point of crossbar being a single
> point of failure for us.
>
> As you can probably imagine this has become a high priority for us
> to solve.
> I have received a recommendation to look at Bondy

> <https://gitlab.com/leapsight/bondy/tree/develop> which is what I

> will be doing this week,
> it doesn't look like its at a good place for automatic deployments
> which is what we need but it seems to be the best
> course of action I have at this point in time.
>
> Does anyone know if crossbar made any progress on a HA solution?
>
> On Wednesday, January 9, 2019 at 5:49:59 AM UTC-8, Ben S wrote:
>
> Hi Greg/Dominique,
>
> Just out of interest, how many many clients do you have connected?
>
> On Wednesday, 9 January 2019 13:33:30 UTC, Dominique Burnand wrote:
>
> I just wanted to say that we suffer from the exact same
> limitations (HA) as you do.
> We also think that your proposed way of implementing it is
> the way to go (this includes the free/pricing feature set).
> We are also using k8s.
>
> At the moment it is okay for us, and we do not need to scale
> to that extends yet. But if our product does scale, it is
> obvious that we need to be able to scale crossbar as well.
> Just thinking of being limited by the router is forcing us
> to lookout for alternatives... And even at our current
> state: It hurts to know that the single-point of failure is
> crossbar.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com
> <mailto:crossbario+...@googlegroups.com>.

> To post to this group, send email to cross...@googlegroups.com

> <mailto:cross...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/crossbario/b1008610-001e-47d3-8ecf-c9a1f1991d9d%40googlegroups.com
> <https://groups.google.com/d/msgid/crossbario/b1008610-001e-47d3-8ecf-c9a1f1991d9d%40googlegroups.com?utm_medium=email&utm_source=footer>.