Flushing every write in logog::LogFile

[Guillaume Seguin]

I want to come back on the discussion about flushing every write to a logfile. I agree that flushing at every write is too expensive, but ability to flush would be nice. 

Perhaps flusing could be integrated as one of many operations that happen on every write. Each operation independent, but able to work together using the chain of responsability pattern. (Funny: Wikipedia's entry on the pattern uses a primitive logger as an example). 

Here are the processing objects I would need/write, in that order :

1. log file rotation (based on time or size)
2. log file cleanup (keep the last N files)
3. integrity values at the end each line (to detect line deletion)

Of course, the more we bring to logog, the less "optimised for games" it gets... 

+I really like the library. I have remplaced my own half baked logging implementation with it.

--

Guillaume

[John Byrd]

I'm glad you like it, and thanks for your comments :)  So let's talk about the best way to implement (or not implement) your ideas.

1) Log file rotation.  Let's think about how logog should work "out of the box" with respect to a typical game programmer.  I can imagine at least the following workflows:

a) Capture the last few (10?) runs of a game, in order to compare the current log with previous run logs.  Delete the oldest log before starting a new log.  (The basic unix log model)

b) Capture all logs as separate files (this can cause the log file directory to get full quickly, but never deletes data);

c) The current method of one log, one log file and it grows forever until you delete it.  This is simple and works but it does not scale well to larger projects.

We also need to remember to Do the Right Thing in all cases if we can't open a file. For some reason I feel like a dummy because I always open a log file, read it, and forget to close it before I restart my programs... logog log file rotation should never destroy data in these cases.

With regard to line integrity.  This sounds to me like a job for a custom Formatter class for you, unless I misunderstand what you're trying to accomplish.  What situations can you imagine where log output might be compromised and incorrectly output?

Again, I'm not against flushing on writes, but I don't think this should be the default.  I imagine just adding a protected boolean to the Output class, and some of the Log* classes would query it to decide whether to fflush after every fwrite.  Then you could just call logFile.FlushEveryWrite( true ) to enable that feature.

Your thoughts on these topics would be appreciated.

jwb

--
You received this message because you are subscribed to the Google Groups "Logog Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/logog-devel/-/WbqOK6pbIAYJ.
To post to this group, send email to logog...@googlegroups.com.
To unsubscribe from this group, send email to logog-devel...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/logog-devel?hl=en.

--
---

John Byrd
Gigantic Software
2102 Business Center Drive
Suite 210-D
Irvine, CA   92612-1001
http://www.giganticsoftware.com
T: (949) 892-3526 F: (206) 309-0850

[Guillaume Seguin]

Right now I have implemented 1a (in my defunct logging framework). But it lacks essential feature : parts of the name are auto generated, and I don't append to a previously closed logfile. Se each reboot gets its own logfile.

1c is simple and it works. 

You are right, a custom formatter could provide log integrity. The purpose : detect that a malicious user removes parts of a log file. Each line is hashed,  and encrypted. The hash of the previous line is XOR'ed with the one we just computed. Strating at line one, we can reconstruct the file and see that is has not been tampeered with. It is not perfect, but at least it is constant time O(1). The system logs random character data every 5 minutes or so to prevent a user to delete more than a few minutes worth of log data, at the end of the file. 

--

Guillaume

jwb

To unsubscribe from this group, send email to logog-devel+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/logog-devel?hl=en.

[John Byrd]

I think that for most logog applications, if people needed a log file
to be rotated then they'd want that to happen on each program
invocation, as opposed to each week or day or what have you. If that
were the case, some long running programs may want to have a mode of
operation where, when log file data is written, the log file is
opened, appended and closed on every write. This would be compatible
with unix-style log files, and would also interoperate with external
log rotation unix scripts.

I'm sure you thought of this but a bunch of warning bells go off in my
mind when you explain your log file anti-tampering strategy. In
general, if it looks like encryption and seems like encryption and
it's not encryption, it is dangerous (it can be hacked). Of course I
don't know your application... I just wonder what kind of application
might have trivial log file tampering as a problem.

jwb

On Wed, Mar 21, 2012 at 8:32 PM, Guillaume Seguin

>>> logog-devel...@googlegroups.com.

>>> For more options, visit this group at
>>> http://groups.google.com/group/logog-devel?hl=en.
>>
>>
>>
>>
>> --
>> ---
>>
>> John Byrd
>> Gigantic Software
>> 2102 Business Center Drive
>> Suite 210-D
>> Irvine, CA   92612-1001
>> http://www.giganticsoftware.com
>> T: (949) 892-3526 F: (206) 309-0850
>

> --
> You received this message because you are subscribed to the Google Groups
> "Logog Development" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/logog-devel/-/m9J5kXnJ6ewJ.

>
> To post to this group, send email to logog...@googlegroups.com.
> To unsubscribe from this group, send email to

> logog-devel...@googlegroups.com.

[Guillaume Seguin]

I think there is an easy way to add log file rotation using logog as it is right now. It moves the problems of dealing with the frequency of rotation and the naming of the log file to the application.

In short : the application creates a LogFile object, with the filename output.01.txt for example as usual. When it decides to "rotate" the file, it deletes the LogFile object and creates a new one, output.02.txt for example. The application can use its own synchronisation, but using logog mutex seems the logical way to implement this.

As for the integrity, hashes are encrypted. That value is to detect malicious removal of log events. It does not prevent anything, it only has detection power. I will look into your suggestion of using a custom formatter for that.


--
Guillaume

>>> logog-devel+unsubscribe@​googlegroups.com.

>>> For more options, visit this group at
>>> http://groups.google.com/​group/logog-devel?hl=en.
>>
>>
>>
>>
>> --
>> ---
>>
>> John Byrd
>> Gigantic Software
>> 2102 Business Center Drive
>> Suite 210-D
>> Irvine, CA   92612-1001
>> http://www.giganticsoftware.​com
>> T: (949) 892-3526 F: (206) 309-0850
>
> --
> You received this message because you are subscribed to the Google Groups
> "Logog Development" group.
> To view this discussion on the web visit
> https://groups.google.com/d/​msg/logog-devel/-/m9J5kXnJ6ewJ​.
>
> To post to this group, send email to logog...@googlegroups.com.
> To unsubscribe from this group, send email to

> logog-devel+unsubscribe@​googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/​group/logog-devel?hl=en.

[John Byrd]

Who's going to maliciously remove log events from a log file? On
which platform(s) is this a concern for you?

I think the only way to truly guarantee that any digital object has
not been tampered with is to sign the entire file using public-key
crypto. Anything else is security by obscurity, and that will not
stop a malicious user. For example, you could rearrange lines in a
logog log file and your tampering algorithm wouldn't detect that fact.
(Recall that ( a ^ b ) ^ c == a ^ (b ^ c).)

If you really really really must prevent tampering then consider
running pgp as a subprocess once your log file is written. (see the
command line options -sta) Dealing with signing, keys, revocations,
file verification and all that is an exciting set of problems, but pgp
has solved them all already.

Posix platforms don't use .txt suffixes, so I'm not sure that the
renumbering algorithm will work as specified -- maybe we have the user
pass in a file name in the form output%d.txt, where we replace the %d
with sprintf, and if there is no %d in the string, we append the
number.

jwb

On Mon, Mar 26, 2012 at 7:09 AM, Guillaume Seguin

>> >>> logog-devel...@googlegroups.com.

>> >>> For more options, visit this group at
>> >>> http://groups.google.com/group/logog-devel?hl=en.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> ---
>> >>
>> >> John Byrd
>> >> Gigantic Software
>> >> 2102 Business Center Drive
>> >> Suite 210-D
>> >> Irvine, CA   92612-1001
>> >> http://www.giganticsoftware.com
>> >> T: (949) 892-3526 F: (206) 309-0850
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Logog Development" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/logog-devel/-/m9J5kXnJ6ewJ.
>> >
>> > To post to this group, send email to logog...@googlegroups.com.
>> > To unsubscribe from this group, send email to

>> > logog-devel...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/logog-devel?hl=en.
>>
>> --
>> ---
>>
>> John Byrd
>> Gigantic Software
>> 2102 Business Center Drive
>> Suite 210-D
>> Irvine, CA   92612-1001
>> http://www.giganticsoftware.com
>> T: (949) 892-3526 F: (206) 309-0850
>
> --
> You received this message because you are subscribed to the Google Groups
> "Logog Development" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/logog-devel/-/tqzeOD54fvQJ.

>
> To post to this group, send email to logog...@googlegroups.com.
> To unsubscribe from this group, send email to

> logog-devel...@googlegroups.com.

[Guillaume Seguin]

Doesn't really matter who, since we can't prevent it, only detect it. An intruder, a software glitch, rogue admin, its all the same : it should not happen but we can't prevent it.

Any tampering will be detected, because there is encryption. It is called hash chaining. The Wikipedia article is pretty broad. This paper applies the concept to secure logging with more details than I understand. This post will probably do the best job in explaining the idea, but it goes something like this :

h[n] = sha1(log_line) //compute the hash of the current line

h[n] ^= h[n-1]        //XOR with the previous hash (wich was encrypted in the previous pass)

h[n] = aes(h[n])      //Encrypt the result

     write log_line, h[n]

     ++n 

A better choice of encryption algorithm can lead to tamper evident log that don't need a key to read or verify. 

Bottom line : you are right, this is a job for a custom formatter.

I didn't know about Posix and suffixes, thank you. Another reason not to put log file rotation in logog, but leave it to the application code.

Thank your for your time considering the idea !

--

Guillaume

>> >>> logog-devel+unsubscribe@googlegroups.com.

>> >>> For more options, visit this group at
>> >>> http://groups.google.com/group/logog-devel?hl=en.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> ---
>> >>
>> >> John Byrd
>> >> Gigantic Software
>> >> 2102 Business Center Drive
>> >> Suite 210-D
>> >> Irvine, CA   92612-1001
>> >> http://www.giganticsoftware.com
>> >> T: (949) 892-3526 F: (206) 309-0850
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Logog Development" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/logog-devel/-/m9J5kXnJ6ewJ.
>> >
>> > To post to this group, send email to logog...@googlegroups.com.
>> > To unsubscribe from this group, send email to

>> > logog-devel+unsubscribe@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/logog-devel?hl=en.
>>
>> --
>> ---
>>
>> John Byrd
>> Gigantic Software
>> 2102 Business Center Drive
>> Suite 210-D
>> Irvine, CA   92612-1001
>> http://www.giganticsoftware.com
>> T: (949) 892-3526 F: (206) 309-0850
>
> --
> You received this message because you are subscribed to the Google Groups
> "Logog Development" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/logog-devel/-/tqzeOD54fvQJ.
>
> To post to this group, send email to logog...@googlegroups.com.
> To unsubscribe from this group, send email to

> logog-devel+unsubscribe@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/logog-devel?hl=en.

[John Byrd]

By your answer, I don't feel much like I did help you :( I do see by
your web site that you know how to generate a GnuPG key though.

And since you were good enough to ask me the question about log file
tamper protection, I'll answer you as politely and as completely as I
can... IMNSHO I do think this problem is much harder than you might
think it is.

I glanced through the paper you hyperlinked. The paper clearly
describes that during initialization time, you need to set up a couple
symmetric keys to seed the first line of the generator and then store
the keys on an uncompromised machine. Furthermore, the scheme only
protects against reordering or deletion of intermediate log file
entries, not against deletion of the end of the log file.
Furthermore, you still gotta write a log file verifier that makes sure
all the log entries were correct. Furthermore, the scheme does not
protect against a rogue program logging spurious data after a
compromise of the target system.

As politely as possible I would like to caution you against trying to
implement this functionality in a logger unless you know EXACTLY what
you are doing. I am not convinced that a couple dudes from UC Irvine
writing an un-peer reviewed paper know exactly what they're doing.
There are myriad ways to implement a cryptographic function
incorrectly if you don't know EXACTLY what you're doing. And it is
probably better to avoid the appearance of security where it does not
exist at the level suggested.

So I hope I scared you a little :) and I wish you best of success on
your project!

jwb

On Mon, Mar 26, 2012 at 7:53 PM, Guillaume Seguin

>> >> >>> logog-devel...@googlegroups.com.

>> >> >>> For more options, visit this group at
>> >> >>> http://groups.google.com/group/logog-devel?hl=en.
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> ---
>> >> >>
>> >> >> John Byrd
>> >> >> Gigantic Software
>> >> >> 2102 Business Center Drive
>> >> >> Suite 210-D
>> >> >> Irvine, CA   92612-1001
>> >> >> http://www.giganticsoftware.com
>> >> >> T: (949) 892-3526 F: (206) 309-0850
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "Logog Development" group.
>> >> > To view this discussion on the web visit
>> >> > https://groups.google.com/d/msg/logog-devel/-/m9J5kXnJ6ewJ.
>> >> >
>> >> > To post to this group, send email to logog...@googlegroups.com.
>> >> > To unsubscribe from this group, send email to

>> >> > logog-devel...@googlegroups.com.

>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/logog-devel?hl=en.
>> >>
>> >> --
>> >> ---
>> >>
>> >> John Byrd
>> >> Gigantic Software
>> >> 2102 Business Center Drive
>> >> Suite 210-D
>> >> Irvine, CA   92612-1001
>> >> http://www.giganticsoftware.com
>> >> T: (949) 892-3526 F: (206) 309-0850
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Logog Development" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/logog-devel/-/tqzeOD54fvQJ.
>> >
>> > To post to this group, send email to logog...@googlegroups.com.
>> > To unsubscribe from this group, send email to

>> > logog-devel...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/logog-devel?hl=en.
>>
>> --
>> ---
>>
>> John Byrd
>> Gigantic Software
>> 2102 Business Center Drive
>> Suite 210-D
>> Irvine, CA   92612-1001
>> http://www.giganticsoftware.com
>> T: (949) 892-3526 F: (206) 309-0850
>
> --
> You received this message because you are subscribed to the Google Groups
> "Logog Development" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/logog-devel/-/6zsh91MGkE4J.

>
> To post to this group, send email to logog...@googlegroups.com.
> To unsubscribe from this group, send email to

> logog-devel...@googlegroups.com.