Using RelStorage 3.0a in production
46 views
Skip to first unread message
Brice Vissière
Sep 30, 2019, 7:01:51 AM9/30/19
to zodb
Hi,
We are considering replacing a ZEO/FileStorage instance by a RelStorage/postgresql one for our services, in order to solve some performance issues. Our tests showed that it's a good call.
But came two questions:
- all the tests were ran with RelStorage 3.0a (a1 to a11), which is still under development. If we choose to go with this development version in production, I guess we will do it "at our own risks". We are wondering what kind of risks? And especially, are there points we should focus on and check, in order to minimize those risks?
- our tests were performed with direct connections to postgresql; we should also still use a ZEO instance in front of postgresql. We did not try it. Is this a good/recommended setup?
Best regard,
Brice
We are considering replacing a ZEO/FileStorage instance by a RelStorage/postgresql one for our services, in order to solve some performance issues. Our tests showed that it's a good call.
But came two questions:
- all the tests were ran with RelStorage 3.0a (a1 to a11), which is still under development. If we choose to go with this development version in production, I guess we will do it "at our own risks". We are wondering what kind of risks? And especially, are there points we should focus on and check, in order to minimize those risks?
- our tests were performed with direct connections to postgresql; we should also still use a ZEO instance in front of postgresql. We did not try it. Is this a good/recommended setup?
Best regard,
Brice
Jason Madden
Sep 30, 2019, 7:39:17 AM9/30/19
to Brice Vissière, zodb
Hi,
> On Sep 30, 2019, at 02:51, Brice Vissière <br...@yaal.fr> wrote:
>
> - all the tests were ran with RelStorage 3.0a (a1 to a11), which is still under development. If we choose to go with this development version in production, I guess we will do it "at our own risks". We are wondering what kind of risks? And especially, are there points we should focus on and check, in order to minimize those risks?
We are also using RelStorage 3.0 in what I think is a fairly large production setting (for example, one large single database is around 100GB and 33 million objects; there are currently about 180 client connections to it). In fact, scaling RelStorage in several different ways has been the general theme for the 3.0 series.
We work hard to make sure that each release is as stable as possible. We try especially hard to be sure that it doesn't introduce chances for database corruption (but we can still make mistakes!). Any bugs we do spot we try to address as quickly as possible. We usually give each release one to two weeks in QA testing before putting it in production (though not always).
The main risks, I would say, are that things are still changing from release to release. Options and configuration settings may come and go; the format of the persistent cache may change; auxiliary tables may be added or altered; stored procedures may change (in such a way that rolling releases aren't possible and a complete shutdown is required[1]), and so on. That said, I think we're closing in on a final feature set for 3.0 and the rate of change in things like that is slowing down.
> - our tests were performed with direct connections to postgresql; we should also still use a ZEO instance in front of postgresql. We did not try it. Is this a good/recommended setup?
I wouldn't put ZEO in front of RelStorage on PostgreSQL (or any other RDBMS). ZEO doesn't understand and expose that RelStorage is natively an IMVCCStorage and never calls new_instance(), so you wind up attempting to multiplex all your client connections over a single PostgreSQL connection. That tends not to go well, *especially* not in history-free RelStorage.
HTH,
Jason
[1] https://github.com/zodb/relstorage/issues/332
> On Sep 30, 2019, at 02:51, Brice Vissière <br...@yaal.fr> wrote:
>
> - all the tests were ran with RelStorage 3.0a (a1 to a11), which is still under development. If we choose to go with this development version in production, I guess we will do it "at our own risks". We are wondering what kind of risks? And especially, are there points we should focus on and check, in order to minimize those risks?
We work hard to make sure that each release is as stable as possible. We try especially hard to be sure that it doesn't introduce chances for database corruption (but we can still make mistakes!). Any bugs we do spot we try to address as quickly as possible. We usually give each release one to two weeks in QA testing before putting it in production (though not always).
The main risks, I would say, are that things are still changing from release to release. Options and configuration settings may come and go; the format of the persistent cache may change; auxiliary tables may be added or altered; stored procedures may change (in such a way that rolling releases aren't possible and a complete shutdown is required[1]), and so on. That said, I think we're closing in on a final feature set for 3.0 and the rate of change in things like that is slowing down.
> - our tests were performed with direct connections to postgresql; we should also still use a ZEO instance in front of postgresql. We did not try it. Is this a good/recommended setup?
HTH,
Jason
[1] https://github.com/zodb/relstorage/issues/332
p...@thirdfloor.com.au
Oct 15, 2019, 7:50:43 PM10/15/19
to zodb
On Monday, 30 September 2019 21:39:17 UTC+10, Jason Madden wrote:
We are also using RelStorage 3.0 in what I think is a fairly large production setting (for example, one large single database is around 100GB and 33 million objects; there are currently about 180 client connections to it).
Out of interest is this a Zope app or something else? We are working through a prototype of a non-Zope app (Pyramid + ZODB) and coming from the usual SQL background we get tripped up on how to work through data structures and indexing etc. In an application like this what would you use? We are currently working with repoze.catalog for search requirements and seems to be working well but we have a feeling in the back of our minds that we are potentially missing the best options. Given that ZODB doesn't really need much documentation to get going, what is documented is great for the the basics, however something along the lines of a cookbook for how to model data and provide search/indexing etc would be nice.
Any thoughts on this are much appreciated - the reduction in code, layers of abstraction and performance improvements with ZODB over SQL/ORM are very appealling.
Regards,
Peter Wilkinson
Reply all
Reply to author
Forward
0 new messages