using zodb with pyramid for simulations/lots of writing

[Iain Duncan]

I've got a sideproject brewing for which I think the ZODB might be a really good fit, but I'm totally new to it. I read Chris's blog post on it, and my only concern was that he said it's really good for stuff with mostly read access.

I expect this project will be doing a *lot* of writes. IE, maybe even all hits do some minor mutation. On the other hand, I have a hunch that the architecture could wind up being really clean in zodb as it's going to be a lot of object modelling (some non-realtime simulation related stuff) And I think transactions will be important, it will be all screwed up if there's any kind integrity jiggery pokery in the object model.

Wondering if anyone can weigh in on:

- how is zodb if you're doing a lot writing?

- experience using it as a persistence scheme for (slow) simulations

- any misc thoughts on zodb

thanks

Iain

[kusut]

On 28 August 2012 11:34, Iain Duncan <iaindun...@gmail.com> wrote:
> - any misc thoughts on zodb

zodb newbie here.

The problem with zodb for a newcomer like me is no querying/indexing
out of the box. It's probably a good idea to include best practice of
zodb in zodb scaffold (with
zeo/hypatia/catalog/anythingfromsubstanced/etc).

Is hypatia the replacement for repoze.catalog?

--
http://kusut.web.id

[Chris McDonough]

It's a fork.

- C

[Malthe Borch]

On 28 August 2012 06:34, Iain Duncan <iaindun...@gmail.com> wrote:
> I expect this project will be doing a *lot* of writes. IE, maybe even all
> hits do some minor mutation. On the other hand, I have a hunch that the
> architecture could wind up being really clean in zodb as it's going to be a
> lot of object modelling (some non-realtime simulation related stuff) And I
> think transactions will be important, it will be all screwed up if there's
> any kind integrity jiggery pokery in the object model.

The ZODB is not optimized for this use-case. There's a couple of
reasons for this:

1. You'll get B-tree conflicts that can't be resolved on the server.
2. You'll need to load non-current object data too frequently, which
is expensive.
3. There's a global commit lock (primarily to establish order) which
incurs some latency.

You'll probable fare better with PostgreSQL because it's got knobs you
can tune and native support for concurrent B-trees.

\malthe

[Iain Duncan]

Thanks Malte. I had been hoping that a pure object database would make the data modelling a lot simpler. Can you weigh in on how much the above issues would be a problem? Like is it so bad that I just abandon the idea and figure it out in SQLAlchemy, which I know well?

thanks

Iain

\malthe

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To post to this group, send email to pylons-...@googlegroups.com.
To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.

[Malthe Borch]

On 28 August 2012 21:20, Iain Duncan <iaindun...@gmail.com> wrote:
> Thanks Malthe. I had been hoping that a pure object database would make the

> data modelling a lot simpler. Can you weigh in on how much the above issues
> would be a problem? Like is it so bad that I just abandon the idea and
> figure it out in SQLAlchemy, which I know well?

I think if you've got a write concurrency requirement > 10 per second
– sustained – then you'll be unhappy with ZODB in its current
implementation.

It's entirely possible to write an object database like ZODB that
supports much more than the concurrency threshold suggested above, but
nobody has stepped up.

\malthe

[Iain Duncan]

Thanks, much appreciated!

Iain

\malthe