Sync pool between process

[Sergi Almacellas Abellana]

Hi,

Currently there is a weird behavior if new modules are installed when
running multiple instances of trytond (as separate process). The problem
is that only one process has the updated pool definition and the other
process will fail with a KeyError when try to get the new created models.

I think this can be solved like we sync the cache resets between instances:

1. Store on the database the pool version.
2. Each time the pool is updated, a new version is saved on the database.
3. Before staring a request that requires the pool, fetch the version
from database and check if it matches with the current pool version
(stored in memory). If it does not match, reload the pool before
processing the request.

This has the drawback that adds one additional query *per request*.

Thoughts?

--
Sergi Almacellas Abellana
www.koolpi.com
Twitter: @pokoli_srk

[Cédric Krier]

For me, activate modules from the UI has always been a gadget (or for
simple setup). On larger setup with multiple server, the activation
should be part of a larger maintenance process which should restart all
of them.

My second point, is that adding a extra query for each request just of
the very rare case of activating a new module is really too much costly.

Indeed I would prefer to have a configuration option that disable the
activation of module from the UI.

--
Cédric Krier - B2CK SPRL
Email/Jabber: cedric...@b2ck.com
Tel: +32 472 54 46 59
Website: http://www.b2ck.com/

[Sergi Almacellas Abellana]

El 28/08/17 a les 12:46, Cédric Krier ha escrit:

> On 2017-08-28 12:14, Sergi Almacellas Abellana wrote:
>> Hi,
>>
>> Currently there is a weird behavior if new modules are installed when
>> running multiple instances of trytond (as separate process). The problem
>> is that only one process has the updated pool definition and the other
>> process will fail with a KeyError when try to get the new created models.
>>
>> I think this can be solved like we sync the cache resets between instances:
>>
>> 1. Store on the database the pool version.
>> 2. Each time the pool is updated, a new version is saved on the database.
>> 3. Before staring a request that requires the pool, fetch the version
>> from database and check if it matches with the current pool version
>> (stored in memory). If it does not match, reload the pool before
>> processing the request.
>>
>> This has the drawback that adds one additional query *per request*.
>>
>> Thoughts?
>
> For me, activate modules from the UI has always been a gadget (or for
> simple setup).

Indeed this has nothing to do with the UI, as it can be reproduced with
trytond-admin also.

On larger setup with multiple server, the activation
> should be part of a larger maintenance process which should restart all
> of them.

The problem I see here is that there is no standard process. Let me
explain:

Everyone can deploy the processes in different ways, so each different
way of deployment should develop a larger maintenance process that fits
its needs.

So we should probably define a standard way or some common procedure.

For us, it will be enough to be able to query the server and return some
status code to know which instances are not updated.

>
> My second point, is that adding a extra query for each request just of
> the very rare case of activating a new module is really too much costly.

I will like to avoid this query on each request too, probably it will be
enough to do it on some time interval.

>
> Indeed I would prefer to have a configuration option that disable the
> activation of module from the UI.
>

That can be achieved by restricting permissions on the UI buttons and
actions. But for me it's out of the scope of this discussion.

[Cédric Krier]

On 2017-08-28 14:27, Sergi Almacellas Abellana wrote:
> El 28/08/17 a les 12:46, Cédric Krier ha escrit:
> > On 2017-08-28 12:14, Sergi Almacellas Abellana wrote:
> >> Hi,
> >>
> >> Currently there is a weird behavior if new modules are installed when
> >> running multiple instances of trytond (as separate process). The problem
> >> is that only one process has the updated pool definition and the other
> >> process will fail with a KeyError when try to get the new created models.
> >>
> >> I think this can be solved like we sync the cache resets between instances:
> >>
> >> 1. Store on the database the pool version.
> >> 2. Each time the pool is updated, a new version is saved on the database.
> >> 3. Before staring a request that requires the pool, fetch the version
> >> from database and check if it matches with the current pool version
> >> (stored in memory). If it does not match, reload the pool before
> >> processing the request.
> >>
> >> This has the drawback that adds one additional query *per request*.
> >>
> >> Thoughts?
> >
> > For me, activate modules from the UI has always been a gadget (or for
> > simple setup).
>
> Indeed this has nothing to do with the UI, as it can be reproduced with
> trytond-admin also.

So the admin knows he has to restart all the services.

> On larger setup with multiple server, the activation
> > should be part of a larger maintenance process which should restart all
> > of them.
>
> The problem I see here is that there is no standard process. Let me
> explain:
>
> Everyone can deploy the processes in different ways, so each different
> way of deployment should develop a larger maintenance process that fits
> its needs.
>
> So we should probably define a standard way or some common procedure.

For me, it will be a disadvantage to limit the way Tryton is deployed.

> For us, it will be enough to be able to query the server and return some
> status code to know which instances are not updated.

I do not see the point. Instead of querying just restart.

> > My second point, is that adding a extra query for each request just of
> > the very rare case of activating a new module is really too much costly.
>
> I will like to avoid this query on each request too, probably it will be
> enough to do it on some time interval.

That sounds odd. You can not expect correct behaviour with such design.

> > Indeed I would prefer to have a configuration option that disable the
> > activation of module from the UI.
> >
>
> That can be achieved by restricting permissions on the UI buttons and
> actions. But for me it's out of the scope of this discussion.

It is not for me because the problem is in the UI because admin user is
not necessary and server administrator. But if the feature is limited to
server administrator, there is no problem.

[Sergi Almacellas Abellana]

El 28/08/17 a les 15:04, Cédric Krier ha escrit:

> It is not for me because the problem is in the UI because admin user is
> not necessary and server administrator. But if the feature is limited to
> server administrator, there is no problem.

When using an orchestration system (like kubernetes), there is no tryton
server administration because the system administrator only administres
the orchestration system, and is this system which manages the
applications states.

In kubernetes this is managed by using Readyness proves [1], which will
query (on an interval time) to each server if it's ready to serve and
restart it if not ready. Having some pragmatic way to know if the
process pool have all the required modules, will allow to solve the
problem for us.

[1]
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/

[Cédric Krier]

On 2017-08-28 15:34, Sergi Almacellas Abellana wrote:
> El 28/08/17 a les 15:04, Cédric Krier ha escrit:
> > It is not for me because the problem is in the UI because admin user is
> > not necessary and server administrator. But if the feature is limited to
> > server administrator, there is no problem.
> When using an orchestration system (like kubernetes), there is no tryton
> server administration because the system administrator only administres
> the orchestration system, and is this system which manages the
> applications states.

This is too easy to say there is no administrator, there is and will be
always an administrator.

> In kubernetes this is managed by using Readyness proves [1], which will
> query (on an interval time) to each server if it's ready to serve and
> restart it if not ready. Having some pragmatic way to know if the
> process pool have all the required modules, will allow to solve the
> problem for us.

This looks like the wrong tools for the wrong problem.

[Jean Cavallo]

Hi Sergi,

2017-08-28 12:14 GMT+02:00 Sergi Almacellas Abellana <se...@koolpi.com>:

Currently there is a weird behavior if new modules are installed when running multiple instances of trytond (as separate process). The problem is that only one process has the updated pool definition and the other process will fail with a KeyError when try to get the new created models.

We got around this problem by using redis to communicate the information accross trytond instances :

https://github.com/coopengo/trytond/blob/master/trytond/iwc.py

https://github.com/coopengo/trytond/blob/master/trytond/pool.py#L109

Jean Cavallo

Coopengo