protobuf logs

[Conrad Leonard]

Hi;

I'm looking to read the audit log files produced by the server; I see they are in protobuf format - is there a .proto definition file available?

cheers,

Conrad.

[Michael Grove]

They normally are in the distro in the docs directory, but as you noticed, they're not there.  Thanks for pointing that out.  I've attached the file, and we'll get those included in the next release.

Cheers,

Mike

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[Conrad Leonard]

Thanks Mike;

I tried to open a small Stardog access.log (attached) with protobufeditor (http://sourceforge.net/projects/protobufeditor/) using log.proto definition  I get the following Exception:

While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either than the input has been truncated or that an embedded message misreported its own length.

Class=class com.google.protobuf.InvalidProtocolBufferException    Error=While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either than the input has been truncated or that an embedded message misreported its own length.

    com.google.protobuf.InvalidProtocolBufferException  :  70

    com.google.protobuf.CodedInputStream  :  1082

    com.google.protobuf.CodedInputStream  :  1068

    com.google.protobuf.CodedInputStream  :  1203

    com.google.protobuf.CodedInputStream  :  517

    net.sf.RecordEditor.ProtoBuf.JRecord.IO.ProtoDelimitedByteReader  :  41

    net.sf.JRecord.IO.LineReaderWrapper  :  69

    net.sf.RecordEditor.re.file.FileView  :  566

    net.sf.RecordEditor.re.file.FileView  :  496

    net.sf.RecordEditor.edit.open.StartEditor  :  51

    net.sf.RecordEditor.edit.open.StartEditorBackGround  :  29

    net.sf.RecordEditor.edit.open.StartEditorBackGround  :  8

    javax.swing.SwingWorker$1  :  296

    java.util.concurrent.FutureTask  :  262

    javax.swing.SwingWorker  :  335

    java.util.concurrent.ThreadPoolExecutor  :  1145

    java.util.concurrent.ThreadPoolExecutor$Worker  :  615

    java.lang.Thread  :  745

The log file was created entirely during a clean server start/stop cycle so I am surprised a truncated message is reported.

I tried both binary and text formats with the same result, also no joy with logs produced in audit logging mode.

Am I doing something wrong?

[Michael Grove]

On Fri, May 8, 2015 at 10:19 PM, Conrad Leonard <conrad....@hotmail.com> wrote:

Thanks Mike;

I tried to open a small Stardog access.log (attached) with protobufeditor (http://sourceforge.net/projects/protobufeditor/) using log.proto definition  I get the following Exception:

While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either than the input has been truncated or that an embedded message misreported its own length.

Class=class com.google.protobuf.InvalidProtocolBufferException    Error=While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either than the input has been truncated or that an embedded message misreported its own length.

    com.google.protobuf.InvalidProtocolBufferException  :  70

    com.google.protobuf.CodedInputStream  :  1082

    com.google.protobuf.CodedInputStream  :  1068

    com.google.protobuf.CodedInputStream  :  1203

    com.google.protobuf.CodedInputStream  :  517

    net.sf.RecordEditor.ProtoBuf.JRecord.IO.ProtoDelimitedByteReader  :  41

    net.sf.JRecord.IO.LineReaderWrapper  :  69

    net.sf.RecordEditor.re.file.FileView  :  566

    net.sf.RecordEditor.re.file.FileView  :  496

    net.sf.RecordEditor.edit.open.StartEditor  :  51

    net.sf.RecordEditor.edit.open.StartEditorBackGround  :  29

    net.sf.RecordEditor.edit.open.StartEditorBackGround  :  8

    javax.swing.SwingWorker$1  :  296

    java.util.concurrent.FutureTask  :  262

    javax.swing.SwingWorker  :  335

    java.util.concurrent.ThreadPoolExecutor  :  1145

    java.util.concurrent.ThreadPoolExecutor$Worker  :  615

    java.lang.Thread  :  745

The log definitely seems borked.  We use the protobuf delimited format, all records are preceded by their size as an int, I don't know if that will be compatible with that protobuf editor.  There are corresponding classes in Stardog you can use, ProtobufLogReader/ProtobufTextLogReader.  I realize that's not as convenient as a GUI, but if that editor doesn't handle the delimited format, that's the easiest alternative.

In any case, the first 4 bytes of the file should be the size of the subsequent protobuf record.  Those bytes report a size of 35.6 MB, which is obviously incorrect.

 

The log file was created entirely during a clean server start/stop cycle so I am surprised a truncated message is reported.

How exactly is the log generated?  Can you tell me the steps to recreate?  Did the log get rotated at any point?

Cheers,

Mike

 

I tried both binary and text formats with the same result, also no joy with logs produced in audit logging mode.

Am I doing something wrong?

On Friday, 8 May 2015 15:19:40 UTC+10, Conrad Leonard wrote:

Hi;

I'm looking to read the audit log files produced by the server; I see they are in protobuf format - is there a .proto definition file available?

cheers,

Conrad.

--

[Conrad Leonard]

Hi Mike;

I was successful using the reader classes in com.complexible.stardog.logging.protobuf to read both text & binary logs produced by our server instance, so the logging itself seems to be working. I don't mind doing it this way; I have no attachment to Protobufeditor although I may try also the wireshark protobuf editor for interest.

A couple of questions remain for me though:

1. Is it expected that the text logs have non-text bytes in them? Tailing text log shows there is clearly non-character stuff in there. If that is expected, what is the difference really between the binary & text logging format?

2. Does the 4-byte marker only apply to binary logging format? I was also able to read even the log I attached earlier, using ProtobufTextLogReader, despite as you say the first 4 bytes being not sensible. The log was produced by: 

    i.   stardog-admin server stop

    ii.  mv $STARDOG_HOME/access.log  /tmp

    iii. stardog-admin server start

    iv. (some queries)

    v.  stardog-admin server stop

    vi. use the resulting $STARDOG_HOME/access.log

[Michael Grove]

On Mon, May 11, 2015 at 10:17 PM, Conrad Leonard <conrad....@hotmail.com> wrote:

Hi Mike;

I was successful using the reader classes in com.complexible.stardog.logging.protobuf to read both text & binary logs produced by our server instance, so the logging itself seems to be working. I don't mind doing it this way; I have no attachment to Protobufeditor although I may try also the wireshark protobuf editor for interest.

A couple of questions remain for me though:

1. Is it expected that the text logs have non-text bytes in them? Tailing text log shows there is clearly non-character stuff in there. If that is expected, what is the difference really between the binary & text logging format?

Yeah, it's not impossible.  The text logs just use protobuf's text format, which isn't plain text.  Once protobuf 3 is final, we'll migrate to that, and afaik, they'll have JSON based writers, which we'll probably adopt.  Mostly the text logs are good for debugging.  In production, I'd imagine you'd use the binary format.

 

2. Does the 4-byte marker only apply to binary logging format? I was also able to read even the log I attached earlier, using ProtobufTextLogReader, despite as you say the first 4 bytes being not sensible. The log was produced by: 

    i.   stardog-admin server stop

    ii.  mv $STARDOG_HOME/access.log  /tmp

    iii. stardog-admin server start

    iv. (some queries)

    v.  stardog-admin server stop

    vi. use the resulting $STARDOG_HOME/access.log

Did you just strip the first four bytes?

Yes, records have to be delimited otherwise the standard protobuf readers just read until the end of the input.

Cheers,

Mike

[Conrad Leonard]

On Tuesday, 12 May 2015 21:09:14 UTC+10, Michael Grove wrote:

On Mon, May 11, 2015 at 10:17 PM, Conrad Leonard <conrad....@hotmail.com> wrote:

Hi Mike;

I was successful using the reader classes in com.complexible.stardog.logging.protobuf to read both text & binary logs produced by our server instance, so the logging itself seems to be working. I don't mind doing it this way; I have no attachment to Protobufeditor although I may try also the wireshark protobuf editor for interest.

A couple of questions remain for me though:

1. Is it expected that the text logs have non-text bytes in them? Tailing text log shows there is clearly non-character stuff in there. If that is expected, what is the difference really between the binary & text logging format?

Yeah, it's not impossible.  The text logs just use protobuf's text format, which isn't plain text.  Once protobuf 3 is final, we'll migrate to that, and afaik, they'll have JSON based writers, which we'll probably adopt.  Mostly the text logs are good for debugging.  In production, I'd imagine you'd use the binary format.

 

ok, yes this is what I'll do 

 

2. Does the 4-byte marker only apply to binary logging format? I was also able to read even the log I attached earlier, using ProtobufTextLogReader, despite as you say the first 4 bytes being not sensible. The log was produced by: 

    i.   stardog-admin server stop

    ii.  mv $STARDOG_HOME/access.log  /tmp

    iii. stardog-admin server start

    iv. (some queries)

    v.  stardog-admin server stop

    vi. use the resulting $STARDOG_HOME/access.log

Did you just strip the first four bytes?

no; that's the source of my confusion - ProtobufTextLogReader reads the whole thing unchanged just fine.

Here's reading from the start of the same file:

groovy:000> p = new ProtobufTextLogReader()

===> com.complexible.stardog.logging.protobuf.ProtobufTextLogReader@30268da6

groovy:000> instr = new FileInputStream(new File('access.log'))

===> java.io.FileInputStream@61a7c7e7

groovy:000> rdr = p.read(instr)

===> com.complexible.stardog.logging.protobuf.ProtobufTextLogReader$1@e1b054c

groovy:000> println rdr.next()

KernelEvent[type=Authentication, user=grafli-rw, db=null, value=null]

 And here's the first 4 bytes:

groovy:000> instr = new FileInputStream(new File('access.log'))

===> java.io.FileInputStream@1e68d7a

groovy:000> buffer = new byte[4]

===> [0, 0, 0, 0]

groovy:000> instr.read(buffer)

===> 4

groovy:000> buffer

===> [2, 32, 118, 101]

 

which makes me wonder if the log is non-spec but ProtoBufTextLogReader is compensating...?

[Michael Grove]

Yeah, binary and text use fairly different formats.  There isn't really a delimited format build in for text, so we use ASCII STX and ETX to delimit records.

It's pretty straightforward to provide a new writer implementation if you'd prefer a format that works with some specific tool, or if you wanted to send the logs to a remote location rather than a local file.

Cheers,

Mike

[Conrad Leonard]

Good idea, thanks.