Binary Stores with indexable keys for fast lookup and maxsize of babudb

[bayoda]

Hi

found babudb and we thought on using it in one project where we need
the following:

getting small pieces of binary data (some bytes up to 64kb chunks)
which are stored with an sha key as unique identifier.

this storage could grow very large -> 1 till 2 TB on disc.

is this possible with babu-db?

is it possible to delete entries in the database easily?

how much entries could be in the babudb index without any problems?
cause we need it for fast asking -> if the hashcode is available ->
have seen the todo list (bloom) - maybe we can contribute on this?
(indexing / or non indexing bloom)

thanks in advance for help on this


yours

Michael

[Jan Stender]

Hi Michael,

> Hi
>
> found babudb and we thought on using it in one project where we need
> the following:
>
> getting small pieces of binary data (some bytes up to 64kb chunks)
> which are stored with an sha key as unique identifier.
>
> this storage could grow very large -> 1 till 2 TB on disc.
>
> is this possible with babu-db?

Yes, BabuDB supports databases of virtually any size.

> is it possible to delete entries in the database easily?

You can remove an entry by doing a 'Database.singleInsert(...)' with a
'null' value. Alternatively, you can use
'DatabaseInsertGroup.addDelete(...)'.

> how much entries could be in the babudb index without any problems?
> cause we need it for fast asking -> if the hashcode is available ->
> have seen the todo list (bloom) - maybe we can contribute on this?
> (indexing / or non indexing bloom)

It depends on what you regard as a "problem". In theory, there is no
hard limit on the number of entries in your database. The most evident
"problem" that can occur if your database keeps growing is performance.
If the size of your database exceeds the size of your main memory by
far, it is likely that BabuDB will slow down. This is because lookups
will hit the disk more often as they cannot be served from the cache
anymore. Since we do not have any practical experience with
terabyte-scale databases and indices so far, however, I cannot provide
you with any concrete performance numbers.

Best regards,
Jan