[hawtdb] Architecture Questions

[Mark McGranaghan]

Thanks for publishing HawtDB!

I have a few questions about HawtDB's architecture and its
implications for users. In everything that follows I assume that
transactional page files are used.

What are the consistency guarantees of the database files? Can I kill
-9 a process that is running HawtDB and be sure that the database file
will always be OK? That it will have all committed transactions? Will
it require a recovery phase? If so, is the time required for recovery
bounded?

Do pages on disk ever need to be garbage collected? Is this handled
automatically by a background thread? Is this behavior
paramaterizable?

How does one back up a database file? Can it be cp'd or rsync'd from
under a live read/write process? A read-only process? Does it require
a special tool/process/locking?

Under concurrent updating, if n transactions attempt to update the
same page, do all n get OptimisticUpdateExceptions, or just n-1?

Is there a way to specify the how much buffer to allow a paged filed
to use? Is this specified as just a maximum or are other options
available?

Sorry about asking so many questions! Answers to even a few would be
appreciated. Also, any pointers into particularly relevant pieces of
the source would be great.

Regards,
Mark McGranaghan



--
Subscription settings: http://groups.google.com/group/hawtdb/subscribe?hl=en

[Hiram Chirino]

Hi Mark

On Mon, Apr 19, 2010 at 11:43 PM, Mark McGranaghan <mmcg...@gmail.com> wrote:
> Thanks for publishing HawtDB!
>
> I have a few questions about HawtDB's architecture and its
> implications for users. In everything that follows I assume that
> transactional page files are used.
>
> What are the consistency guarantees of the database files? Can I kill
> -9 a process that is running HawtDB and be sure that the database file
> will always be OK? That it will have all committed transactions? Will
> it require a recovery phase? If so, is the time required for recovery
> bounded?
>

So another one of the features of HawtDB is that 'D' part in ACID is
decoupled from the transaction. In other words, a transaction is not
guaranteed to be durable once committed. If you follow it up with a
TxPageFile.flush() call, then you get the durability guarantee.

So yes, you can kill -9 a running process and the database will remain
consistent and all changes performed previous to the flush call will
be guaranteed to to have been applied.

> Do pages on disk ever need to be garbage collected? Is this handled
> automatically by a background thread? Is this behavior
> paramaterizable?
>

We keep a list of free pages so no garbage collection is needed. Now,
if you mean "can a big page file resize down?", currently no. Need to
do some work on this. If end of the file has unused pages, it should
be easy to truncate the page file size down. The harder part will be
if there is lots of unused space in the middle, can the pages be moved
around so that space can be compacted.

> How does one back up a database file? Can it be cp'd or rsync'd from
> under a live read/write process? A read-only process? Does it require
> a special tool/process/locking?

This is another area that could use some contributions. Ideally we
would expose and API method which flushes out and puts the database in
a read only mode that blocks all writers to the file so that it can be
copied and then another method to put it back in write mode. Since
that's missing the only safe bet is to close the page file copy it and
then re-open it.

>
> Under concurrent updating, if n transactions attempt to update the
> same page, do all n get OptimisticUpdateExceptions, or just n-1?
>

n-1. The first one to commit will win. Everyone else will get an
OptimisticUpdateException

> Is there a way to specify the how much buffer to allow a paged filed
> to use? Is this specified as just a maximum or are other options
> available?
>

There are lots of buffers in play. hopefully one day they will all be
configurable via options in the TxPageFileFactory.
By default HawtDB memory maps the page file in 64 meg chunks/segments.
The page file has to be at a minimum 1 segment big. You can tweak
this setting with the mappingSegementSize property on the
TxPageFileFactory.
The pageSize you pick can also be important. The default 4k page size
is probably too big.. should probably more like 512.

Other important options that have not yet been exposed as a config
option on the TxPageFileFactory are:
- HawtTxPageFile.updateBatchSize : controls the number of page updates
that we try to batch up in an update. Batching up multiple
transactions in one update allows us to reduce the average cost of the
file.sync().
- HawtTxPageFile.ReadCache : configured to only cache 1024 objects.
This is an Object cache, holding objects associated with recently
accessed pages. I reduced the cost of deserializing frequently
accessed pages. Some folks may want to make this much bigger.

> Sorry about asking so many questions! Answers to even a few would be
> appreciated. Also, any pointers into particularly relevant pieces of
> the source would be great.
>

No problems! I'm glad your interested. :)

> Regards,
> Mark McGranaghan
>
>
>
> --
> Subscription settings: http://groups.google.com/group/hawtdb/subscribe?hl=en
>

--
Regards,
Hiram

Blog: http://hiramchirino.com

Open Source SOA
http://fusesource.com/