Server generated msi not checking in

[racloir]

I'm working though learning about Velociraptor and setting it up to see how to deploy it in our environment. Currently I've got 3 physical machines setup and are all actually plugged into the same switch. One with Ubuntu Server 22.04 LTS to use as the server and two Windows 10 machines for clients. Velociraptor server is setup using the self-signed cert and I can access the dashboard no problem.

I used Server.Utils.CreateMSI to create a msi for install on the 2 Windows 10 client machines. I've got it installed on the clients and Windows show a velociraptor service running but they never check in with the server.  At one point I tried installing while Wireshark was running and no dns lookups are run and no traffic tries to go to the server's ip address. And that I could find I don't know where any other logs for the client might be.

However if I take the config that's placed in C:\Program Files\Velociraptor when you install the msi and run it with

velociraptor --config client.config.yaml client -v

using a velociraptor exe it immediately finds the server.

I've been using the docs on the velociraptor website but even searching the internet hasn't turned anything up for me although maybe I just haven't found the right incantation of search terms. So at this point I'm not sure how else to troubleshoot this and/or where I might have messed up.

[Mike Cohen]

The troubleshooting guide https://docs.velociraptor.app/docs/deployment/troubleshooting/ suggests to run the client manually which is what you did and since it is working then I would assume the network connectivity is good and configuration is fine.

If the server is unable to connect when running as system but can connect as a regular user then there is something about the user account that is running the binary. Maybe firewall settings or a security software.

You can try to:

1. Use psexec to elevate to system shell (psexec.exe /s cmd) and then try to connect manually

2. Change the server to run as another user and restart it.

3. Also check the logs because it might show that it is being blocked or something.

Those are the next steps i would try to debug what is happening

Thanks

Mike

[racloir]

Unfortunately work happened and I haven't been able to get back to this yet but like you mentioned I am wondering what the logs say. My question is I don't know where to find them? I see a writeback.yaml on the client and if I sign into the server gui I see Server Audit Logs under Server Events.

Searching the actual server itself only came back with

/etc/systemd/system/multi-user.target.wants/velociraptor_server.service
/etc/systemd/system/velociraptor_server.service
/etc/systemd/system/velociraptor_server.service.d
/etc/velociraptor
/sys/fs/cgroup/system.slice/velociraptor_server.service
/opt/velociraptor
/var/lib/dpkg/info/velociraptor-server.postinst
/var/lib/dpkg/info/velociraptor-server.conffiles
/var/lib/dpkg/info/velociraptor-server.list
/var/lib/dpkg/info/velociraptor-server.prerm
/var/lib/dpkg/info/velociraptor-server.md5sums
/usr/local/bin/velociraptor
/usr/local/bin/velociraptor.bin

The writeback and Server Audit Logs don't seem to have anything useful in them. So I apologize if it's listed on the website somewhere and I'm just overlooking it but I don't know where to look.

[Mike Cohen]

The clients do not normally log to disk to avoid disclosing sensitive information. I was actually referring to the windows event logs (looking for something like firewall blocked or something). 

As a test you can actually enable logging on the endpoint:

1. Open regedit to Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Velociraptor find the ImagePath setting and add a --logfile parameter

"C:\Program Files\Velociraptor\Velociraptor.exe"  --config "C:\Program Files\Velociraptor\/client.config.yaml" service run --logfile=c:/test.log

2. Go to the services app and restart the service

This will cause the service to write the logs to that file and you can use this to understand what is happening - it is the same as client -v but the actual service is logging that data.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

--
You received this message because you are subscribed to the Google Groups "velociraptor-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to velociraptor-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/068b601d-7800-43da-8cfb-e7967dc4f609n%40googlegroups.com.

[racloir]

Good news everyone! It's working.. but I don't understand why. I was out sick the last half of last week so I'm just now getting back to this.

I added the log file option in the registry and started up the service and preceded to get this one line over and over and over just with different time stamps in the log file.

{"level":"prelog","msg":"Failed to load yaml: unmarshal errors:\n  line 36: field level2_writeback_suffix not found in type proto.ClientConfig: C:\\Program Files\\Velociraptor\\/client.config.yaml will try again soon.\n","time":"2023-12-05T17:21:26.4221773Z"}

The log file contained nothing else but that line ad infinitum whereas when I started velociraptor up in the console I got the usual starting up, connecting etc.

So, because it's a config file and I could always add it back I just deleted that line saved the file and restarted the service.

This time when I checked the log file it looked just like it does when started from console with the -v option and the server picked up almost instantly.

Whhyyy??? Don't get me wrong. I'm happy it's working now and I can get on with my testing but I'm shrug'n over here. I don't understand.

[Mike Cohen]

The  level2_writeback_suffix option was added as part of the 0.7.0-3 fix to implement more robust config writeback writing.

This option is missing in earlier versions so earlier versions will reject that config file (but the server might generate the client config including this option if the server is the latest version).

Removing the option is fine - the older clients can't use it anyway.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/4ce049ab-aba2-4317-9e30-4ccd2a27119en%40googlegroups.com.

[racloir]

After spending yesterday figuring out what exactly was going on to cause my issue and with the help of the previous posts above. And also to help those that come behind and are also installing velociraptor for the first time.

xkcd

I initially installed the server with v0.7.0-3. I've been working on this as work permits on and off for awhile.
Then I saw the CVE and upgraded the server to v0.7.0-4.

Over time I realized that clients connected to the server when run from the console using velociraptor --config client.config.yaml client -v.

But not when installed from a msi using the Server.Utils.CreateMSI artifact.

Adding logging to ImagePath in HKEY_LOCAL_MACHINE>SYSTEM>CurrentControlSet>Services showed me something wasn't right. (see above for how to enable logging)

The log file just contained

{"level":"prelog","msg":"Failed to load yaml: unmarshal errors:\n  line 36: field level2_writeback_suffix not found in type proto.ClientConfig: C:\\Program Files\\Velociraptor\\/client.config.yaml will try again soon.\n","time":"2023-12-05T17:21:26.4221773Z"}

Over and over and nothing else.

Removing that line from the client config file made made everything work.
Well that's weird. So I opened up and looked at the contents of the msi Server.Utils.CreateMSI created.
And that's when I found that the client config file packaged in the msi was version v.0.7.0-4. (as mentioned above)
But the version of velociraptor packaged in the msi was version .v.0.7.0.
Apparently I messed something up when installing velociraptor on the server initially and/or upgrading to v0.7.0-4.
So I removed everything from the server and reinstalled v0.7.0-4 and re-ran Server.Utils.CreateMSI.
Now when I inspected the msi everything had the correct version and clients connect right up to the server.

So if you're having issues getting clients to talk to the server enable logging and also try checking the contents of the msi if your using Server.Utils.CreateMSI as a start.

[Mike Cohen]

Thanks for spending the time to debug and report this issue.

I actually think there is something wrong with the code responsible for upgrading the inventory. When you install 0.7.0-4 it should automatically start to repack 0.7.0-4 in the msi and not stay on the old client version. Because it was packing the old version with the new config there was an incompatibility.

This likely only showed up in upgrades. You can reset the entire inventory by just removing the <data store>/config/inventory.json file to start again. 

I will look more into this particular upgrade path today.

Thanks again

Mike

To view this discussion on the web visit https://groups.google.com/d/msgid/velociraptor-discuss/30f58c98-c273-4ddf-ade1-29c37bdbf48fn%40googlegroups.com.