Multi-frontend/minion questions

[Mario DeTore]

Hello all,

I have been experimenting with a multi-frontend deployment placed behind an internet facing load balancer (with clients configured to report to the single LB URL in Client.server_urls). From what I can tell minion frontends aren't able to handle client enrollments and you must also expose the "master" frontend via the load-balancer to ensure a clean enrollment. Is my understanding correct? The minions don't appear to be running the interrogation/enrollment service by default (https://github.com/Velocidex/velociraptor/blob/master/services/frontend/frontend.go#L341-L355). Is there a flag I'm missing possibly (other than --minion)?

I'm working from this blog post - https://docs.velociraptor.app/blog/2021/2021-04-29-scaling-velociraptor-57acc4df76ed/ . Specifically this sentence regarding putting the frontends behind an LB: "In order to spread the load evenly between the multiple frontends, it is possible to use a load balancer in front of all the frontends."

Have also encountered spurious HTTP 409 errors after a client re-connects (example at end of email) when all minions and the master frontend are behind the LB. They tend to resolve after some amount of time.

Also, it appears that the master/minion relationship at the network layer (internally behind the LB) is pretty much one way - the minions connect to the API port via TLS on the master to participate in the replication service, but the master does not connect to the minions over the network. Please correct me if I'm wrong. I've verified the network path both ways between the minions and the master frontend, but have only observed TCP connection from the minions to the master frontend.

Lastly, it appears all the minions write to the same logs as the master. This complicates troubleshooting a bit. Maybe I misconfigured something? 

Running 0.6.3 fwiw. 

[INFO] 2022-02-15T01:26:45+02:00 Receiver: Connected to https://<redacted>.net:443/reader
[DEBUG] 2022-02-15T01:26:45+02:00 Connection Info {"IdleTime":0,"LocalAddr":{"IP":"10.0.2.15","Port":49931,"Zone":""},"Reused":true,"WasIdle":true}
[INFO] 2022-02-15T01:26:45+02:00 Compiled all artifacts.
[INFO] 2022-02-15T01:26:45+02:00 Receiver: sent 690 bytes, response with status: 409 Conflict
[INFO] 2022-02-15T01:26:45+02:00 Post to https://<redacted>.net:443/reader returned 409 - advancing
[INFO] 2022-02-15T01:26:45+02:00 Waiting for a reachable server: 1m25s

--

Thanks for your time & guidance!

 Mario R. De Tore

 TechOps Engineer - Global Services Practice

 

 Mobile  | +65.8141.5385

 

---

---

Confidentiality Notice: The information contained in this email communication, including without limitation, any attachments, is confidential and may be legally privileged. It is intended solely for the individual(s) or entity named above and others who have been specifically authorized to receive it. If you are not the intended recipient you may not review, copy, use or disclose the contents of this communication (or any attachment hereof) to others. Please notify the sender that you have received this e-mail communication in error by replying to the above e-mail address or telephone and delete all copies of this message and any attachments.

[Mike Cohen]

Hi Mario,

   The minions are connecting to the master on the API port for the replication service. The replication service is bidirectional so there is no need to connect back to the minions.

This is how enrollment is happening -

The server (minion) realises the client is not enrolled and sends an enrolment message:

https://github.com/Velocidex/velociraptor/blob/f7b50326d54a33de75fb38970f524b98287b2203/server/enroll.go#L56

This goes through the replication service to the interrogation service on the master

https://github.com/Velocidex/velociraptor/blob/f7b50326d54a33de75fb38970f524b98287b2203/services/interrogation/interrogation.go#L91

Where the client is interrogated and scheduled.

So the master initiates the enrollment (and initiates all collections actually) but the minions are just communicating to clients and storing the results from them. 

The minions should write their logs to a subdirectory of the logs directory. 

https://github.com/Velocidex/velociraptor/blob/f7b50326d54a33de75fb38970f524b98287b2203/logging/logging.go#L269

The node name is derived from the node's hostname and bind port in the minion config.

https://github.com/Velocidex/velociraptor/blob/f7b50326d54a33de75fb38970f524b98287b2203/services/frontend.go#L58

This the latest documentation for this setup https://docs.velociraptor.app/docs/deployment/cloud/multifrontend/

This feature is currently experimental but is reported to be working in several large deployments. If you see issues we can work together through them to make it more robust.

Feel free to send me a zoom invite if you want to chat more about it.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

--
You received this message because you are subscribed to the Google Groups "velociraptor-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to velociraptor-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/CADQeWcxXUTKCdUPVvoB2SEg23yv%2B8YjPEuCFsShaXTiTTZ6uxA%40mail.gmail.com.

[Mike Cohen]

BTW there has been a lot of work on this feature through the 0.6.3 release cycle so make sure you are running the final release and not any of the release candidates

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

[Mario DeTore]

Thanks Mike, I was able to partially resolve some of this (my node naming convention was off - didn't realise I needed a hyphen between hostname & port as the example uses a colon), but minions still seem unable to facilitate new client enrollment when placed in a pool behind an LB. I just get continuous 502s client-side unless I also expose the master frontend. Will follow-up with you off-list about that Zoom call.

Appreciate your time!

-Mario

Please report a suspicious email using the Phish Alert fishhook icon or forward the email to sp...@cybereason.com