Need help with Multi-frontend setup

[Mokshartha BV]

Hi Team,

Greetings of the day!

We are looking to deploy Velociraptor client on 70k endpoints/servers in our environment.

We obviously would like to go with Multi-frontend deployment/architecture. Our current plan is to keep 1 Master and go with 7 minions (each minion serving upto 10k clients).

So far, we got couple of references on how to configure this and we were hoping if we can get much detailed documentation (if it is documented anywhere else).

https://docs.velociraptor.app/docs/deployment/cloud/multifrontend/

https://velociraptor.velocidex.com/scaling-velociraptor-57acc4df76ed

 

Below are some of the questions we have:

1. REPEAT – Do you offer/have additional information on how to deploy large scale multi-front-end deployments ?
2. As per your knowledge, what is the highest count of clients that were deployed using this model (since it still says it is experimental) ?
3. Since we mentioned we would like to go with 1 Master and 7 minions and each minion serving upto 10k clients. I would assume use 7 different dns names for each minion/frontend, which will inturn have 7 client config files to manage etc. Do you agree with this (or) it can be done in any other way ?
4. Do you recommend NFS for File datastore ? We will be doing in On-Prem and not Cloud.
5. How do we handle the upgrades ? I would assume server side will be easy from the videos i have seen/followed. How can we do it efficiently for all the clients ?

Regards,

Mokshartha

[Mike Cohen]

Hi Mokshartha,

    These are the documentation pages regarding multi tenant deployments. It is pretty simple - once you add one additional frontend you will get two different deb packages for minion and master then you just deploy as many minions as you need. Alternatively you can use any other technology to deploy servers (eg. puppet, chef, docker k8s etc).

You can either have the minions behind a load balancer or you can deploy them using round robin DNS (with dyndns optionally). If you add say 10 frontends with different DNS names then the clients will randomly choose one to connect to and this will spread the load without a load balancer. 

A user shared with me a screenshot of a large deployment here https://present.velocidex.com/velocon_2022_year_in_review/#/36 . I believe they might have about 10 or 12 frontends to reach 125 endpoints.

There are some configuration parameters that can be tweaked here https://github.com/Velocidex/velociraptor/blob/master/docs/references/server.config.yaml#L301

you will need to increase the expected number of clients at least to match the expected number (say to around 100k)

You need to use some kind of distributed filesystem - we use NFS and EFS (which is the amazon version of NFS). You might be able to tweak the NFS to make it faster (lots of resources on Google like https://tldp.org/HOWTO/NFS-HOWTO/performance.html ). You should be able to get faster NFS on prem than in AWS because AWS uses TLS which adds overheads not necessary on prem.

As far as upgrades - it makes no difference to the clients what type of frontend they talk to - so clients upgrade schedule is the same as before - they do not need to be upgraded in step with the server but it is recommended to keep them mostly in sync. Server upgrades are just the same - build new debs and deploy using existing config files.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

--
You received this message because you are subscribed to the Google Groups "velociraptor-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to velociraptor-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/9aa23881-4b8f-4448-a624-c280d6996624n%40googlegroups.com.

[Mokshartha BV]

Thanks a lot Mike for your responses.

For ExtraFrontends configuration, does it add to the Minion server config file (or) for the Master server config file ?

Apologies if this is a stupid question. I was wondering why would we keep the expected clients limit to 100k if it is just a minion.

[Mike Cohen]

The extra frontend part is used by the minion - you technically only need one there if you have a load balancer because they are all the same in that case.

The expected number of clients controls the size of various caches in the binary (e.g. the size of the session key cache). Since clients can migrate randomly between minions they might all be serving the same client at various times so it helps to put it in the cache. I dont think the memory cost is significant either way.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/557b10bc-8494-4336-8b01-f22d87d4f77bn%40googlegroups.com.

[Mokshartha BV]

Thank you very much Mike 😊