Simple JSON log?

[Xavier Mertens]

Hi,

On the Artefact Exchange, they're lot of solution to send server events to Splunk, Mattermost, Slack, Elastic, ...

I was wondering if there is a simple way to dump the same events into a simple JSON file on the Velociraptor server file system?

Kr, 

Xavier

[Mike Cohen]

You can easily write a jsonl file using 

https://docs.velociraptor.app/vql_reference/other/write_jsonl/

Or you dont even have to do that because Velociraptor is storing results in jsonl anyway so you can always copy the files out of the datastore externally.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

--
You received this message because you are subscribed to the Google Groups "velociraptor-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to velociraptor-dis...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/velociraptor-discuss/f03f0cdb-3f6d-4c4e-ac69-d8743d380bb2n%40googlegroups.com.

[Xavier Mertens]

Tx for the tip! Here is what I'm trying. The JSON file is created on disk but... empty!

...

LET events = SELECT * FROM foreach(
     query={
        SELECT *, "Artifact_" + Artifact as _index,
              Artifact,
              timestamp(epoch=now()) AS timestamp,
              client_info(client_id=ClientId).os_info.hostname AS Hostname
       FROM watch_monitoring(artifact=Artifact)
    })

SELECT * FROM write_jsonl(
  filename=Filename,
  query=events
)

Am I in the right direction?

/x

[Mike Cohen]

Almost, you don't have a rows parameter to the foreach so there is no artifacts to watch. Also for event artifacts you need to specify async = true to ensure they are all combined in parallel. Otherwise the first one will run and block the others.

To view this discussion visit https://groups.google.com/d/msgid/velociraptor-discuss/274754d9-832f-409c-a4c1-3aea1a520de8n%40googlegroups.com.

[Xavier Mertens]

Ok, now, I see my events in the Server Event tabs but my file remains empty on disk... Tx for the tip!

But I've a doubt: The document says that the parameter "filename" is mandatory  and described like this: "CSV files to open". Do I have to read this file and convert it to JSON? What's the destination in this case?

I'm a bit lost :)

SELECT * FROM write_jsonl( filename=Filename, query=events )

[Mike Cohen]

That's the path to the file to write . It needs to be writable and the directory must exist

To view this discussion visit https://groups.google.com/d/msgid/velociraptor-discuss/8faf1211-9412-4720-b8ea-e0b2baf5d863n%40googlegroups.com.

[Xavier Mertens]

Ok, I understood properly :-)

The file is created on the file system, I see event in the Velo console but nothing is dumped in the file!? Did I miss something else?

1LET artifacts_to_watch = SELECT Artifact FROM parse_csv( 2 filename=artifactParameterMap, accessor='data') 3 WHERE get(item=scope(), member=Parameter) AND log( 4 message="Dumping artifact " + Artifact + " into JSONL file") 5LET events = SELECT * FROM foreach( 6 row=artifacts_to_watch, 7 async=TRUE, 8 query={ 9 SELECT *, "Artifact_" + Artifact as _index, 10 Artifact, 11 timestamp(epoch=now()) AS timestamp, 12 client_info(client_id=ClientId).os_info.hostname AS Hostname 13 FROM watch_monitoring(artifact=Artifact) 14 }) 15 16SELECT * FROM write_jsonl( 17 filename=Filename, 18 query=events 19)

[Xavier Mertens]

Nevermind,  there was not event processed, no I see some :)

How do you recommend to use the 'buffer_size' parameter? When I check my dumped json file, the last event looks "incomplete". Seems to be related to this buffer size.

[Mike Cohen]

The json file will be queued in memory up to the buffer size and then written. If you make the buffer size smaller it will write it sooner but there is no guarantee the file will be complete. Probably we also need a time based flush parameter here to ensure the file is complete at fixed points in time.

this plugin was originally written to allow dumping of data into a jsonl file not for passing live events so it does not ensure the file is valid json until it is complete.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

To view this discussion visit https://groups.google.com/d/msgid/velociraptor-discuss/8a1e8c77-4ac7-4e6f-8e15-e7f8e322ebf1n%40googlegroups.com.

[Xavier Mertens]

Hi Mike,

My JSON file is properly populated with my events but I detected an issue this morning. I added the JSON file to my log rotate configuration to rotate the file every day (the file is pretty big).

I saw that Velociraptor keeps filling the rotated file (xxx.json.1) instead of filling the new one. It seems that it keeps using the same inode. I tried to change the way logrotate works by adding the "copytruncate" option. With this option enabled, no new JSON events are stored in the newly created JSON file. Any idea?

/x

[Mike Cohen]

You are right in that the write_jsonl() plugin first creates the file and then writes to it. As I mentioned the original use case was to just dump out a query not to accommodate live event writing. Perhaps we should allow for a way to rotate the file natively in the write_jsonl() plugin?

Right now, you can achieve a similar effect using the query() plugin - this plugin allows you to run any other query within different parameters. For example using the timeout parameter you can arrange for the write_jsonl() query to terminate after some time and write to a different file. This achieves the desired effect of changing the writing file with a very small window where events may possibly be lost while the query is re-executed

Let me know if you need help with writing this query 

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

To view this discussion visit https://groups.google.com/d/msgid/velociraptor-discuss/2ef064f2-f3ba-4a82-89fb-92e34980852cn%40googlegroups.com.

[Xavier Mertens]

Hi Mike,

I see the point but could you provide me an example of the query() plugin usage? I’ll try this way! Example: To write files likes events-yyyymmddhh.json and rotate every hour?

Tx!

On 26 Jan 2025, at 09:17, Mike Cohen <mi...@velocidex.com> wrote:



To view this discussion visit https://groups.google.com/d/msgid/velociraptor-discuss/CACREbPBakCP9ahdYXX76Xa1ohVkwLXAiDC8tpJtw9jxsR9UjHA%40mail.gmail.com.

[Mike Cohen]

Hi Xavier,

Here is an example of a query that rotates the file every five minutes. it works by timing the query out and writing a new file each time.

LET EventStream = SELECT Unix FROM clock()

SELECT * FROM foreach(row={
    SELECT * FROM range(end=10000)
}, query={
  SELECT * FROM write_jsonl(
    filename=format(format="C:/logfile_%d.jsonl", args=now() ),
    query={
       SELECT * FROM query(query=EventStream, timeout=300)
    })
})
-- Use this to suppress events from feeding into the artifact output.
WHERE FALSE

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

[Xavier Mertens]

Trying now but not successful…

How do add add my events to the query?

I had this before:

SELECT * FROM write_jsonl(

    filename=Filename,

    query=events,

    buffer_size=1024

)

“events” contains my events extracted via another query. 

In your example below, events -> EventStream?

/x

[Xavier Mertens]

Hi Mike,

I'm still struggling with my attempts to mix my old query with yours (to properly rotate the JSON files). Any idea how to solve this :-(

Tx!

[Mike Cohen]

Here is the full solution

```

LET Parameter <= "Foo"

LET artifacts_to_watch = SELECT Artifact

  FROM parse_csv(filename=artifactParameterMap, accessor='data')
  WHERE get(item=scope(), member=Parameter)
   AND log(message="Dumping artifact " + Artifact + " into JSONL file")

LET events = SELECT *
  FROM foreach(

    row=artifacts_to_watch,
    async=TRUE,
    query={
      SELECT *,
             "Artifact_" + Artifact AS _index,

             Artifact,
             timestamp(epoch=now()) AS timestamp,
             client_info(client_id=ClientId).os_info.hostname AS Hostname
      FROM watch_monitoring(artifact=Artifact)
    })

SELECT * FROM foreach(row={
    SELECT * FROM range(end=10000)
}, query={
  SELECT * FROM write_jsonl(
    filename=format(format="C:/logfile_%d.jsonl", args=now() ),
    query={

       SELECT * FROM query(query=events, timeout=30, inherit=TRUE)
    })
})

```

The inherit parameter is ensures that the symbols depended on by the events query are visible inside the isolated query scope - if you dont add it then it might not be visible and you get errors like artifacts_to_watch not found.

Thanks

Mike

Mike Cohen  Digital Paleontologist,  Velocidex Enterprises E mi...@velocidex.com

To view this discussion visit https://groups.google.com/d/msgid/velociraptor-discuss/e688dc52-89e2-4eea-8d52-6ff8b9c72095n%40googlegroups.com.