Missed labels?

[Xavier Mertens]

Another day, another issue? :-)

My Velociraptor instance was stuck (high CPU/mem usage) due to a lot of clients connected and hunting rules. I had to restart it.

Now, it seems that all my labels are gone!? When I search "label:xxx", Nothing. If I search all clients, I see them tagged with the previous labels!? 

Any idea? Bug? Fix? I hope that it can be solved because I've 3000+ clients tagged :(

/x

[Xavier Mertens]

Something strange... (maybe to help in debugging). When I tried to tag another client, I had to create a new label with the same name.

Now, when I search for this label, only the *newly* clients are seen. But when I list all clients, I see them tagged with the same label!? Like of the label was duplicated...

[Mike Cohen]

Try to remove the index snapshot from the file store client_idx directory and restart the server. It should rebuild the index automatically.

--
You received this message because you are subscribed to the Google Groups "velociraptor-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to velociraptor-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/934d6483-65dd-4a06-b744-78a564c83fbdn%40googlegroups.com.

[Xavier Mertens]

Hi Mike,

Thank you for the quick followup!

I see these files in client_idx:

- snapshot.json.index

- snapshots/1666350529.json.index

- snapshots/1666351251.json.index

I remove them? Others?

/x

[Xavier Mertens]

Hi Mike,

I tried to remove the snapshot.json file, it was recreated but I now see a very low amount of entries dumped into the index.

Example:

{"level":"debug","msg":"<green>Indexing Service</>: Wrote index on /opt/velociraptor/client_idx/snapshot.json in 447.959813ms (1793 entries)\n","time":"2022-11-30T10:31:21Z"}

Before the problem, I had >20000 entries...

On Tuesday, November 29, 2022 at 6:59:15 PM UTC+1 mi...@velocidex.com wrote:

[Mike Cohen]

It should rebuild the index by scanning all the client records. How many client records do you have? (an `ls <filestore>/clients/ | wc`)

You can also rebuild it by doing a hunt for Generic.Client.Info - this will reindex all the currently active clients (omitting dead clients).

It is good to remove the index periodically to eliminate old outdated clients anyway.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/97ebea0d-3ea6-4b41-a2b9-61db78f91b2cn%40googlegroups.com.

[Xavier Mertens]

4524 clients... 

Running now the hunt but when I select "run everywhere", the estimation is only 312 clients!?

When I check the graph, I see all of them (>3000 clients connected)