A few questions

[Maxi]

What is your recommendation on storing/retrieving/modifying a list of
comments? I think I need to use the map datatype, as I don't see any
way to delete specific list items based on some kind of attribute in
them. (they can only retain strings or ints after all)

Also, two things: As I understand it, I have to call
hyperclient_loop() every now and then. This is confusing, shouldn't
this be treated by the database? What kind of timeout value should I
use and what do I need the return value for? This sounds like a
function that I have to put into the very core event loop of my
webserver, but like I said, I don't see what utility it has for me.
I feel the same about the thread synchronization. Handling a mutex for
the hyperclient instance is a little inconvenient.

Also, I guess I would appreciate documentation on the subject of how
the project is layed out. How can I start stepping through code in
hyperdex, basically? Where do the python modules fit in, what exactly
is their purpose, why do it in python, etc. Just some high level
documentation basically.

Appreciate it!

[Robert Escriva]

On Fri, Apr 27, 2012 at 03:45:59PM -0400, Maxi wrote:
> What is your recommendation on storing/retrieving/modifying a list of
> comments? I think I need to use the map datatype, as I don't see any
> way to delete specific list items based on some kind of attribute in
> them. (they can only retain strings or ints after all)

What kind of attribute are you thinking? I think a map from numeric
comment ids to strings would be best for comments. For a different kind
of "delete matching X", I'd possibly do it differently.

> Also, two things: As I understand it, I have to call
> hyperclient_loop() every now and then. This is confusing, shouldn't
> this be treated by the database? What kind of timeout value should I
> use and what do I need the return value for? This sounds like a
> function that I have to put into the very core event loop of my
> webserver, but like I said, I don't see what utility it has for me.
> I feel the same about the thread synchronization. Handling a mutex for
> the hyperclient instance is a little inconvenient.

hyperclient_loop is used only by the C API. If you use the Python
bindings (e.g., in a Django web application), you'll never even see it.

I explicitly avoided adding background threads, or any thread safety to
the client so that single-threaded clients do not pay a penalty. It is
safe to instantiate one client per thread. I actually prefer this
approach when developing.

> Also, I guess I would appreciate documentation on the subject of how
> the project is layed out. How can I start stepping through code in
> hyperdex, basically? Where do the python modules fit in, what exactly
> is their purpose, why do it in python, etc. Just some high level
> documentation basically.
>
> Appreciate it!

We always could use more documentation. Are you thinking about the
layout from the perspective of an application developer, administrator,
or HyperDex developer? If either of the former, I'd recommend going
through the Basic Operations[1] as it show the structure of the system.

Almost all HyperDex code is in hyperdex-daemon itself. The Python
modules are only used for managing membership in the system, and as a
scripting API.

Happy Hacking,
Robert

1. http://hyperdex.org/doc/tutorial/

[Maxi]

>What kind of attribute are you thinking? I think a map from numeric
comment ids to strings would be best for comments.

True, I'm probably just used to MongoDB. I was thinking of some kind
of nested list, where you can access the nested list and modify its
members.

> hyperclient_loop is used only by the C API.

I'd be using the C API, that's why I am confused on how /often/when to
use the loop() function.

> Are you thinking about the
layout from the perspective of an application developer,
administrator,
or HyperDex developer?

I'm thinking in terms of a HyperDex developer, because personally I
think it's always best to step through code and get a feel for the
program. (..I can't find the actual executable, I assume that's
because there is none and python code loads the c++ libraries. I'll
need to learn the autotools)

Since hyperdex has a pretty low version number, the question arises
how mature the database is. Would you recommend using it in a real
world website? How often do you still encounter / check with Valgrind
memory bugs, what kind of plans do you have for a stable release?

[Robert Escriva]

On Fri, Apr 27, 2012 at 05:41:49PM -0400, Maxi wrote:
> >What kind of attribute are you thinking? I think a map from numeric
> comment ids to strings would be best for comments.
>
> True, I'm probably just used to MongoDB. I was thinking of some kind
> of nested list, where you can access the nested list and modify its
> members.
>
> > hyperclient_loop is used only by the C API.
>
> I'd be using the C API, that's why I am confused on how /often/when to
> use the loop() function.

The loop function is used to complete operations. Here's an example
that is pseudo-C:

int64_t some_id = hyperclient_put(...)
int64_t some_other_id = hyperclient_loop(...)
if (some_other_id == some_id) {
/* The put has completed */

}

> > Are you thinking about the
> layout from the perspective of an application developer,
> administrator,
> or HyperDex developer?
>
> I'm thinking in terms of a HyperDex developer, because personally I
> think it's always best to step through code and get a feel for the
> program. (..I can't find the actual executable, I assume that's
> because there is none and python code loads the c++ libraries. I'll
> need to learn the autotools)
>
> Since hyperdex has a pretty low version number, the question arises
> how mature the database is. Would you recommend using it in a real
> world website? How often do you still encounter / check with Valgrind
> memory bugs, what kind of plans do you have for a stable release?

We have a few applications we are builing on top of HyperDex.

I periodically run the system under valgrind and electric fence with
great success (memory bugs are extremely rare). We try to keep each
release stable with no backwards regressions.

-Robert