Adding different backend

[Nicolas Évrard]

Hello,

I am currently working on two patches that will make it easier to
create python modules adding a new backend to tryton:

- https://tryton-rietveld.appspot.com/33951002/
This patch moves the handling of the SQL type of the tryton
fields to the backend.

In order to have a kind of lingua franca between tryton and the
backend we specify on the field the "abstract" SQL types it
wants to use (a M2O will use an INTEGER, a TimeDelta an
INTERVAL, and so on).

- https://tryton-rietveld.appspot.com/26231002/
This patch uses the abstract SQL types to create the column in
the backend instead of the raw SQL type that was used before.

You can see those patch in action in the following patch adding a
postgis backend which inherits almost everything from the postgres
backend: https://codereview.appspot.com/322730043/

We wonder if there is some kind of interest to make this backend
supported by community or if we should keep it in the B2CK realm.
According to me we should definitely support this backend (I
personally know ∞-time more people that would use this backend then
the mysql one). And this last remarks makes me wonder if we shouldn't
make of the mysql backend this kind of python package (supported by
the community but not release critical at all).

What are your thoughts about it?

--
Nicolas Évrard - B2CK SPRL
E-mail/Jabber: nicolas...@b2ck.com
Tel: +32 472 54 46 59
Website: http://www.b2ck.com/

[Cédric Krier]

On 2017-03-20 18:34, Nicolas Évrard wrote:
> And this last remarks makes me wonder if we shouldn't
> make of the mysql backend this kind of python package (supported by
> the community but not release critical at all).

I'm in favor of that it will improve the quality of the MySQL backend
because we could finally run tests on it without fearing to have test
failure for trytond.

--
Cédric Krier - B2CK SPRL
Email/Jabber: cedric...@b2ck.com

[Sergi Almacellas Abellana]

El 20/03/17 a les 18:34, Nicolas Évrard ha escrit:

> Hello,
>
> I am currently working on two patches that will make it easier to
> create python modules adding a new backend to tryton:
>
> - https://tryton-rietveld.appspot.com/33951002/
> This patch moves the handling of the SQL type of the tryton
> fields to the backend.
>
> In order to have a kind of lingua franca between tryton and the
> backend we specify on the field the "abstract" SQL types it
> wants to use (a M2O will use an INTEGER, a TimeDelta an
> INTERVAL, and so on).

So in order to create a custom field type (I have JSONB in my mind), we
have to create a new backend? And in order to support this kind of
fields on more than one backend we have to create multiple backend? I
imagine some users having to maintain the following backends:

sqlite+spatialite
postgresql+postgis
sqlite+json
postgresql+jsonb
sqlite+spatialite+json
postgresql+postgist+json

Having said that I'm wondering if it won't be better to allow to
register the handlers of the abstract types in external modules, so with
the same backend we will be able to handle different types of fields,
which can be then reused.

>
> - https://tryton-rietveld.appspot.com/26231002/
> This patch uses the abstract SQL types to create the column in
> the backend instead of the raw SQL type that was used before.
>
> You can see those patch in action in the following patch adding a
> postgis backend which inherits almost everything from the postgres
> backend: https://codereview.appspot.com/322730043/

Instead of creating a new backend I will prefer to extend the existing one.

>
> We wonder if there is some kind of interest to make this backend
> supported by community or if we should keep it in the B2CK realm.
> According to me we should definitely support this backend (I
> personally know ∞-time more people that would use this backend then
> the mysql one). And this last remarks makes me wonder if we shouldn't
> make of the mysql backend this kind of python package (supported by
> the community but not release critical at all).

+1 for separating mysql backend.

I'm in favor of adding community supported packages but not release
critical. Indeed, i think more packages can be added in this realm, like:

- flask-tryton
- celery-tryton
- redis-tryton
- trydoc






--
Sergi Almacellas Abellana
www.koolpi.com
Twitter: @pokoli_srk

[Nicolas Évrard]

* Sergi Almacellas Abellana [2017-03-21 12:02 +0100]:

>El 20/03/17 a les 18:34, Nicolas Évrard ha escrit:
>>Hello,
>>
>>I am currently working on two patches that will make it easier to
>>create python modules adding a new backend to tryton:
>>
>> - https://tryton-rietveld.appspot.com/33951002/ This patch moves
>> the handling of the SQL type of the tryton fields to the backend.
>>
>> In order to have a kind of lingua franca between tryton and the
>> backend we specify on the field the "abstract" SQL types it
>> wants to use (a M2O will use an INTEGER, a TimeDelta an
>> INTERVAL, and so on).
>
>So in order to create a custom field type (I have JSONB in my mind),
>we have to create a new backend? And in order to support this kind of
>fields on more than one backend we have to create multiple backend?

Kind of.
The lingua franca is a just a mean so that the fields speak the same
language as the backend. It's then up to the backend to translate this
for the database.

>I imagine some users having to maintain the following backends:
>
>sqlite+spatialite
>postgresql+postgis
>sqlite+json
>postgresql+jsonb
>sqlite+spatialite+json
>postgresql+postgist+json

I think that an eventual support for JSONB would be implemented using
JSONB for postgres, TEXT for sqlite and MySQL.

>Having said that I'm wondering if it won't be better to allow to
>register the handlers of the abstract types in external modules, so
>with the same backend we will be able to handle different types of
>fields, which can be then reused.

Indeed registering marshallers in the backends could be a way to
tackle the complexity of mixing together GIS and JSONB and whatnot.

>>
>> - https://tryton-rietveld.appspot.com/26231002/
>> This patch uses the abstract SQL types to create the column in
>> the backend instead of the raw SQL type that was used before.
>>
>>You can see those patch in action in the following patch adding a
>>postgis backend which inherits almost everything from the postgres
>>backend: https://codereview.appspot.com/322730043/
>
>Instead of creating a new backend I will prefer to extend the
>existing one.

There is not that much people using geographic related data, and even
in those cases it seems the geographic data is not use in business
processes.

That's why I think it should live in a specific package.

On the other hand JSONB could be and (probably will be) used in
business processes that's why I think that those kind of fields should
be included in core. So I am not completely sold on the idea of
registering marshallers in the end ;).

>>We wonder if there is some kind of interest to make this backend
>>supported by community or if we should keep it in the B2CK realm.
>>According to me we should definitely support this backend (I
>>personally know ∞-time more people that would use this backend then
>>the mysql one). And this last remarks makes me wonder if we shouldn't
>>make of the mysql backend this kind of python package (supported by
>>the community but not release critical at all).
>
>+1 for separating mysql backend.

OK then.

>I'm in favor of adding community supported packages but not release
>critical. Indeed, i think more packages can be added in this realm,
>like:
>
>- flask-tryton
>- celery-tryton
>- redis-tryton
>- trydoc

+1 but we should discuss of each of those separately.

[Cédric Krier]

Indeed what we tried to achieve is to have the backend agnostic about
fields. So we created this abstract SQL type which of course is closed
to the common SQL types.
So the design of issue6351 is to have converter on the backend using the
abstract type. Maybe we could think of a way to make those converter
extensible a little bit like what we did for json. But before that it
will be good to know how much abstract types we have and/or we could add
in the future. For example, the JSONB is not really a good abstract
type name, it should probably be an HashMap or JSON because JSONB is too
much low level.
But the main difficulty in introducing a new kind of abstract type which
is not common/standard is to have it supported by all backends.
And this makes me think that Tryton should not do too much in this area
as far as it allows customization of the database schema transparently.
For example, the TEXT column for Dict field on postgresql should
transparently work with JSONB type.

--
Cédric Krier - B2CK SPRL
Email/Jabber: cedric...@b2ck.com

[Sergi Almacellas Abellana]

El 21/03/17 a les 14:33, Cédric Krier ha escrit:

What I miss is some mechanism to extend/modify the backend abstract
types without having to rewrite a new backend. So we can add new
abstract types directly from module and register the adapters for the
current backends. Of course and error must be throw if used with a
backend that's not aware of this abstract type.

Something like the function mapping of python-sql depending on the flavor.

> But the main difficulty in introducing a new kind of abstract type which
> is not common/standard is to have it supported by all backends.
> And this makes me think that Tryton should not do too much in this area
> as far as it allows customization of the database schema transparently.

The problem i see here is that you have to manually change the database
schema, which adds a lot of complexity when dealing with a big number of
databases or a big number of custom fields. That's why I prefer to
create a module that does it for me.

> For example, the TEXT column for Dict field on postgresql should
> transparently work with JSONB type.
>

I have tested it and it requires https://bugs.tryton.org/issue6382

[Sergi Almacellas Abellana]

El 21/03/17 a les 13:34, Nicolas Évrard ha escrit:

>
>> I'm in favor of adding community supported packages but not release
>> critical. Indeed, i think more packages can be added in this realm, like:
>>
>> - flask-tryton
>> - celery-tryton
>> - redis-tryton
>> - trydoc
>
> +1 but we should discuss of each of those separately.

Maybe we should define some criteria that should met this community
supported packages.

So better to open a discuss topic about it?

[Cédric Krier]

On 2017-03-21 17:47, Sergi Almacellas Abellana wrote:
> El 21/03/17 a les 14:33, Cédric Krier ha escrit:

> > But the main difficulty in introducing a new kind of abstract type which
> > is not common/standard is to have it supported by all backends.
> > And this makes me think that Tryton should not do too much in this area
> > as far as it allows customization of the database schema transparently.
>
> The problem i see here is that you have to manually change the database
> schema, which adds a lot of complexity when dealing with a big number of
> databases or a big number of custom fields. That's why I prefer to create a
> module that does it for me.

I see no reason it could only be done manually.

[Cédric Krier]

On 2017-03-21 17:49, Sergi Almacellas Abellana wrote:
> El 21/03/17 a les 13:34, Nicolas Évrard ha escrit:
> >
> > > I'm in favor of adding community supported packages but not release
> > > critical. Indeed, i think more packages can be added in this realm, like:
> > >
> > > - flask-tryton
> > > - celery-tryton
> > > - redis-tryton
> > > - trydoc
> >
> > +1 but we should discuss of each of those separately.
>
> Maybe we should define some criteria that should met this community
> supported packages.

-1 for criteria. We always worked by consensus.

--
Cédric Krier - B2CK SPRL
Email/Jabber: cedric...@b2ck.com