VFS GUI inconsistently refreshing file finder listing

[Dominic Bunch]

I've tried searching around to see if this is a know bug or if there is some deployment caveat, but I have not found anything on this issue:

When refreshing directories in the VFS GUI, the file finder collection will execute successfully and return rows, but the artifact results do not get populated in the VFS GUI, so you do not see the directory or files. This has been functioning inconsistently across various clients in my fleet.

Client version: 0.73.3

Any thoughts or feedback are welcome!

---

Any information contained in or attached to this e-mail is intended solely for the use of the intended recipient(s) and may contain certain information that is confidential, proprietary and/or legally privileged. If you are not the intended recipient, you may not review, copy or distribute this message. If you receive this in error, please notify the sender and destroy all copies of this message and attachments.

[Mike Cohen]

The VFS GUI does not launch the file finder artifact - it launches the System.VFS.ListDirectory artifact https://docs.velociraptor.app/artifact_references/pages/system.vfs.listdirectory/ and that is the only one that populates the VFS in the GUI.

Launching the regular file finder does not populate the VFS data structures. The VFS GUI is just a convenience for quickly viewing files on the endpoint; it is not meant to represent all known files about the end point.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

--
You received this message because you are subscribed to the Google Groups "velociraptor-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to velociraptor-dis...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/velociraptor-discuss/0a29f67a-06f1-463f-a495-dce28f8da33bn%40googlegroups.com.

[Dominic Bunch]

My mistake, I did indeed mean the ListDirectory, not FileFinder, artifact that you just described. 

The issue remains the same, though, that when I try to populate the VFS on the GUI page, the files and directories fail to appear, even though the artifact successfully executes. 

This is the same behavior for all users on our instance. I’ve observed the network requests in Chrome, and AFAICT, the request to populate the results of the artifact do not contain the response data required to populate the UI component.

I could provide more details if that would be helpful.

[Mike Cohen]

The ListDirectory artifact collects two sources:

1. The file listing

2. High level stats about the listing

The server combines these into the GUI view in the datastore.

Can you confirm that the results from the client look correct?

Are you running a master/minion style deployment or a single server?

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

[Dominic Bunch]

I can confirm that these artifacts indeed exist:

I just tested this on about 5 macOS clients and the VFS GUI search appears to be WAI, which to your last point, we are running a single server instance. So, I wonder if the refresh is slow because of the server capacity, resulting in what I'm mistaking to be a bug?

Our concurrent connections peak at around ~1500 (~3000 total clients), but our resource consumption appears to be relatively minimal (we have plenty of memory and CPU capacity):

Would you suggest that we explore an additional frontend considering our number of clients and potential max concurrent connections? During the test I just mentioned, our concurrent connections were down to ~500.

[Mike Cohen]

No I dont think you want to go to multi server architecture - it is quite a bit different and should not be needed for your number of clients.

The way this works is that there is a single VFS processor which post-processes the VFS flows - it could be falling behind if you have some really large VFS listings and might need to give it some time to catch up.

You can see this in the queue manage debugging queue - on the server https://docs.velociraptor.app/docs/gui/debugging/#the-debug-server 

Select the queue manage debug screen for the relevant org 

You will see the pending size > 0 when there are still flows waiting to be processed and the service is falling behind. We should probably add some more debugging here to help but typically the VFS post processor should not be too heavily used - it is pretty quick to post process those flows.

What sometimes happens is that the GUI fails to refresh the VFS itself after results come in - hitting browser refresh will rule out a JS issue.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

[Dominic Bunch]

Got it. When we encounter this issue again I'll inspect the queue debugging and see if that reveals anything noticeably wrong. In the event the number is much larger than 0, is there anything we can do to improve performance, other than reduce VFS utilization?

--

Dominic Bunch

He/Him/His

Engineering Manager, Threat Response

klaviyo.com

[Mike Cohen]

No this is set to be a low priority operation which should not use too much server resources - it should eventually catch up.

I dont personally use the VFS that much and generally I dont do very large listings. 

If you find you use the VFS often in this way (e.g. a recursive VFS listing from C: drive) then you can schedule these collection in other ways (e.g. automatically or through the API) they do not need to be scheduled from the GUI at all - there is no difference between GUI scheduling the collection and e.g. in a hunt.

This way the post processing can be ready when you are ready to inspect the VFS interactively.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

[Dominic Bunch]

That’s a good idea and is probably how we will better optimize this VFS search functionality for our particular usage.

I agree, and we don’t always use VFS, but when we do it’s usually all at once and can appear laggy due to the sheer volume of data.

Appreciate the help and talking me through this Mike!

[Mike Cohen]

Thanks for the feedback too - I will add some tracking for the VFS post processing too.

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com