strategies for conversion from json to SQLalchemy model layer?

[Iain Duncan]

Hi folks, I'm working on an internal framework using Pyramid, Colander, and SQLAlchemy, and it's for much more enterprisey apps than I have previously done. We're looking at having a full fledged service layer, and using the ZCA as a DI framework a fair bit. I'm hoping folks can share their opinions on what they've found the best place and way to convert from validated dicts (originating from JSON from angular.js) to SA mapped objects is. I'm using Colander to validate the incoming json, which might be nested two or three levels deep, so after Colander conversion I know the fields in the dicts are ok, but they are still just python dicts.

In the past, I used formencode to do both validation and conversion, but in this case I'm hoping to keep SQLA logic and coupling to a business/model layer that is more insulated from the web request layer so it can be used without issue from non-web-request contexts (rabbitqm jobs, scripts, etc).  So I'm imagining right now that outside this business/model layer the web controllers are dealing with json and validating with colander, inside the business/model we are dealing with either proxy objects to the model objects or direct model objects, and somewhere (?) we convert. 

Any suggestions on what people have found to be good strategies or tools (or reading!) for localizing and controlling the conversion from dicts to mapped objects with relationships would be much appreciated. Or suggestions that I'm just wrong and why. 

thanks. 

(apologies for cross post if you already read this on the SQLAlchemy list!)

Iain

[Mikko Ohtamaa]

Hi,

What I have seen happening that many people use __json__(self, request) method on SQLAlchemy models. I find this dirty, as web subsystem should not relevant to data persistency subsystem and there is a bleeding of separation of concerns.

I'd rather have one adapter per model which gives serialize/deserialize process which permission support so that the same serializer could support e.g. anonymous and authenticated use cases.

In my ideal world

- All data would be described as Colander

- (semi-automatic) adapters would provide serializing between SQLAlchemy model instances and Colander JSON schema

- Python docstrings could be used do describe objects in Colander schema

- Open API (swagger) schema could be generated from Colander schemas, thus providing support for automatic Swagger client generation, API discovery and such

Some more related work

https://github.com/stefanofontanelli/ColanderAlchemy

http://websauna.org/docs/narrative/form/autoform.html (wrapper of above)

https://pythonhosted.org/dictalchemy/

https://github.com/striglia/pyramid_swagger

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAN9NcLzWeQRVEAZhU1OZMwrc0fDiahMxQw1W1f6QtjfJe4__LQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Mikko Ohtamaa
http://opensourcehacker.com
http://twitter.com/moo9000

[Vincent Catalano]

Hey Iain,

I basically use two patterns when it comes to handling the impedance mismatch between my persistence layer (DB with SQLAlchemy), domain models, and view models (JSON objects for Angular.js):

1.) This first pattern I use consists of maintaining two separate models for the domain and persistence layer. I personally prefer to use SQLAlchemy's declarative approach, but many times my domain models do not associate directly with all the fields in the persistence models. In these situations I rely heavily on the Repository Pattern (https://msdn.microsoft.com/en-us/library/ff649690.aspx) for dealing with the conversion of domain objects to the persistence objects. Unfortunately, this results in a quite a bit of redundant code since the domain models will contain most of the same fields and structures as the persistence models. However, I've found that the benefit of maintaining a good separation of concerns between the persistence layer and my domain models provides major benefits.

2.) The second method that I use is adding to_json and from_json static functions to my domain models or SQLAlchemy model objects (depending on how rich your domain model structure is and how big your project is). When I validate incoming JSON from the web (or any other service) I use Colander to validate the structure and turn the object to a dictionary. I will then use the from_json function to convert the object to the appropriate the persistence model or domain model. On the other hand, when I want to convert my domain object to JSON, I use the to_json function. This gives me the greatest control over the fields that I want to expose to JSON without needing an additional libraries or needing to define custom serializers/deserializers.

Overall, I've found that's it's best to maintain distinct models for my persistence layer, domain model and view model even if it tends to be redundant. When I need to convert between two different object types, e.g. model view (JSON) to domain model, I simply add the static function to the object I need to convert to.

These patterns have really helped me deal with a number of Pyramid applications that have grown quite large over the past few years.

-Vincent

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUv-p9jJhS%2BABxauZw3dc5QoY-voLfNVF7DYN6U1_BFodA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Vincent Catalano
Software Engineer and Web Developer,
(520).603.8944

[tonthon]

Hi,

It's maybe too coupled for your need, but for this purpose I mostly use
colanderalchemy.
http://colanderalchemy.readthedocs.io/en/latest/

It allows to generate colander schemas from sqla models and provide
usefull dictify and objectify methods.

Hope this helps
Regards

> --
> You received this message because you are subscribed to the Google
> Groups "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pylons-discus...@googlegroups.com

> <mailto:pylons-discus...@googlegroups.com>.

> To post to this group, send email to pylons-...@googlegroups.com

> <mailto:pylons-...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pylons-discuss/CAN9NcLzWeQRVEAZhU1OZMwrc0fDiahMxQw1W1f6QtjfJe4__LQ%40mail.gmail.com

> <https://groups.google.com/d/msgid/pylons-discuss/CAN9NcLzWeQRVEAZhU1OZMwrc0fDiahMxQw1W1f6QtjfJe4__LQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Jonathan Vanasco]

In our largest app we have a caching layer and do a lot of transitions between SqlAlchemy and a Dict (both directions)

some of these may apply to you:

1. dicts are wrapped in an 'attribute safe' container, so they behave like sqlalchemy objects.  certain sqlalchemy relationships are handled via lazyloading cache items from a readthrough cache.  

2. there is a per-request local cache of dict objects.

3. the dict containers have a 'readonly' flag, so they raise errors if we try to write.  the system loads a fresh sqlalchemy object when it needs to write, then updates the cached items.

4. there is a single method for converting the objects from one type to another.  after a while, it made more sense to keep the mapping logic between the two object types instead of nested into one.

[Iain Duncan]

Thanks for the input everyone. One thing I'm wrestling with is whether to use my deserializer/validator to go to app-level dicts or straight to SQLAlchemy objects. I'm not sure yet whether the service/action layer should work with SQLA objects all the time. Thoughts on that?

thanks

iain

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/2a6a8c2d-1d00-4ba9-b99e-2de88be16d56%40googlegroups.com.

[Vincent Catalano]

Personally, I've found that many times the logic and object structures that I'm dealing with in my service layer do not directly correlate to my SQLA Objects. This usually caused my SQLA Objects to become bloated with logic and many times these objects become awkward to use. Plus, this also meant that a lot of my business logic was being into the persistence layer (SQLA Objects) violating the principle of separation of concerns. My recommendation is this: strictly use SQLA Objects for persistence only (unless your project is relatively small) and use separate model objects (not dictionaries) to encapsulate behavior and pass around to your service layers. The one disadvantage to this approach can be the redundancy between your domain objects and your SQLA Objects, but I find it's still much easier to test individual components and easier to handle the differences between your persistence model and your domain model.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAN9NcLz8rjPTHK8wFrG97Hnx-UBB_K7GJ97UUZRSTb4WOZA6hA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Iain Duncan]

Thanks everyone for the input.

 Vincent, do you wind up making model objects that are per-instance adapters? or more class level service objects?

iain

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAEhnOszCW%3D1NgYm6hA6XaCmYb_U-LyZ-_%3DuN%2BaXCq-Z5P%2BMuuA%40mail.gmail.com.

[Vincent Catalano]

Assuming that by "class level service objects" you mean domain model objects then I would say I find myself using these most often. My applications tend to be REST APIs that integrate with Angular web apps. In these cases, my Python application is receiving a JSON structure that I want to translate to a domain model object to send to a service. I've found that I don't usually need to implement very many adapters, instead, I have static from_json functions for my domain model classes that build the model object from the JSON (validation with Colander is done before this step). Once my JSON object has been "converted" to the domain object I pass it off to the service layer. When returning JSON in my response (for example) depending on how complex the domain object is, I will use either adapters or simply have a to_json function on my domain model class.

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAN9NcLyL8tqHzmo2k8we7qOLdCTuO2jHG2WAfQE9097yA77AGg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.