happstack-state / acid-state and disk full

[Petter Bergman]

Hi,

Today my processes running happstack-state stopped responding, upon investigation I discovered that the "disk" on my virtual server had become full.

Trying to (re)-start the 3 of my server processes(the processes connected to customers) yielded the following error:

> handleEvent FAIL: too few bytes. Failed reading at byte position 72

> booking: too few bytes. Failed reading at byte position 72

2 other processes started fine, these probably never had any activity on them.

I solved the error by reverting back to a backup.

What will happen if happstack-state can't write an event to disk because the disk is full?

What will happen in the same situation with acid-state?

Regards

Petter Bergman

[Felipe Almeida Lessa]

On Sun, Jan 15, 2012 at 7:55 PM, Petter Bergman
<jon.pette...@gmail.com> wrote:
> What will happen if happstack-state can't write an event to disk because the
> disk is full?

AFAIK, the log will become corrupted, just like you saw.

> What will happen in the same situation with acid-state?

Probably the same thing.

Cheers,

--
Felipe.

[Matthew Elder]

AFAIK you should be able to back up a little bit in the log to a clean spot (i.e shave off some bytes).. recommend looking at last few bytes of file (tail?) and try to find the boundary.

--
You received this message because you are subscribed to the Google Groups "HAppS" group.
To post to this group, send email to ha...@googlegroups.com.
To unsubscribe from this group, send email to happs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/happs?hl=en.

[Jeremy Shaw]

On Sun, Jan 15, 2012 at 3:55 PM, Petter Bergman <jon.pette...@gmail.com> wrote:

 

What will happen if happstack-state can't write an event to disk because the disk is full?

Who knows. One of the reasons for a clean rewrite as acid-state is that happstack-state had too many loose ends like that.

 

What will happen in the same situation with acid-state?

One of the things Lemmih worked on in acid-state was ensuring that events really got flushed to disk. That is the reason why it uses the low-level posix API for writing files to disk.

The correct behavior when the disk is full is for update transactions to fail. That is pretty much the only way to handle the 'durability' aspect of ACID. If we can not successfully flush the event log to disk, then we can not allow the transaction to complete. 

Internally, acid-state uses 'write(2)' to write the transaction to disk (or something similar on Win32). If the write fails, it will it raise an ERRNO exception ENOSPC.

Still, it is not entirely obvious what the result of that will be. Looking at the code,  I think it means that the 'update' call will never return because the thread that writes the log will die and never write the mvar that allows update to return. Obviously.. it might be nicer if the update function itself raised an exception instead of just blocking? Though blocking forever (or until happstack kills the request thread) does seem ACID compliant..

Hopefully Lemmih can provide a better answer.

- jeremy

[Lemmih]

It'll probably fail in some ugly way but it'll definitely be ACID
compliant. Making it fail more gracefully isn't high on my TODO list
since it isn't a torn in my side.

--
Cheers,
  Lemmih

[Petter Bergman]

Thanks for all answers, once I am done rewriting my app using acid-state I will try to simulate this situation again and see what happens.

Ironically, in this case the cure was the cause of the problem. I had a cron job taking backups on regular intervals, it was these backups that saved me this time, but also the backups that filled up the disk in the first place...