How to inspect filters

[Andrew Martin]

I have a generic CRUDService for my web app. It's a pattern that was loosely suggested to me by Mike a while back. I've probably not implemented it the way he intended, but it works pretty well in a pretty small amount of code.

The main work, however, that needs to be done is in the read_all method. It needs to be able to handle different filters depending on which view is calling it. Which it can as it is now. But I want to fidget with the defaults a little. Basically, I almost never want to show objects that have been soft deleted. Only admin users/views should be able to fidget with things that we've "deleted". The way the method works now is that if I don't pass any filters in at all, it sets a filter to cls.is_deleted==False. But I'd like to be able to inspect incoming filters. Say the view looks at the resource context and determines that a view should only be allowed to see objects based on some ACL rule. I might create a filter for that view controller that looks like 

filters = (Project.usergroup==User.usergroup)

(defining the applicable usergroup for a given context is a separate service, but the relationship should be obvious, I think?)

and then have the view query with CRUDService.read_all(Project, request.dbsession, filters). The way I've got it now, that would just replace the default filter, and I would have to add Project.is_deleted==False to the tuple of filters every time I call this. I would like to be able to inspect the filters that are passed into the read_all (which, honestly, I should rename and call it read_many) method and say, if there isn't any reference to the is_deleted column in the filters, then set that as false, otherwise, accept the value in the function parameter. But I find the filter's tuple values quite opaque. Once they are instantiated, they are class-specific, and I'm trying to get at a general way of understanding them. I'll note that this column is guaranteed to be present in the framework. I've modified the declarative base such that every object has the is_deleted. I appreciate any help you might be able to give. It's very likely I'm overthinking this like a numbskull and the answer is painfully obvious. Here's the service class:

from pyramid.request import Request

from crudites.models.meta import Base

from sqlalchemy.orm.session import Session

from uuid import UUID

from typing import List, Tuple, Dict

from crudites.services.id_service import IDService

class CRUDService:

    '''TODO: docstring'''

    @staticmethod

    def create(cls: Base, s: Session) -> bool:

        unique_id, public_id = IDService.create_db_ids()

        cls.unique_id=unique_id

        cls.public_id=public_id

        s.add(cls)

        s.commit()

        return True

    @staticmethod

    def read_by_pkid(cls: Base, s: Session, id: int) -> Base:

        q = s.query(cls).get(id)

        return q

    @staticmethod

    def read_by_unique_id(cls: Base, s: Session, id: UUID) -> Base:

        q = s.query(cls).filter_by(unique_id=id).first()

        return q

    @staticmethod

    def read_by_public_id(cls: Base, s: Session, id: str) -> Base:

        q = s.query(cls).filter_by(public_id=id).first()

        return q

    @staticmethod

    def read_all(cls: Base, s: Session, filters: Tuple=None) -> List[Base]:

        if not filters:

            filters = (cls.is_deleted==False,)

        q = s.query(cls).filter(*filters).all()

        return q

    @staticmethod

    def update(cls: Base, s: Session) -> bool:

        s.add(cls)

        s.commit()

        return True

    @staticmethod

    def delete(cls: Base, s: Session)-> bool:

        cls.is_deleted=True

        s.add(cls)

        s.commit()

        return True

Thanks,

-andrew

[Mike Bayer]

On Sun, Jun 23, 2019, at 5:45 PM, Andrew Martin wrote:

I have a generic CRUDService for my web app. It's a pattern that was loosely suggested to me by Mike a while back. I've probably not implemented it the way he intended, but it works pretty well in a pretty small amount of code.

The main work, however, that needs to be done is in the read_all method. It needs to be able to handle different filters depending on which view is calling it. Which it can as it is now. But I want to fidget with the defaults a little. Basically, I almost never want to show objects that have been soft deleted. Only admin users/views should be able to fidget with things that we've "deleted". The way the method works now is that if I don't pass any filters in at all, it sets a filter to cls.is_deleted==False. But I'd like to be able to inspect incoming filters. Say the view looks at the resource context and determines that a view should only be allowed to see objects based on some ACL rule. I might create a filter for that view controller that looks like 

filters = (Project.usergroup==User.usergroup)

(defining the applicable usergroup for a given context is a separate service, but the relationship should be obvious, I think?)

and then have the view query with CRUDService.read_all(Project, request.dbsession, filters). The way I've got it now, that would just replace the default filter, and I would have to add Project.is_deleted==False to the tuple of filters every time I call this. I would like to be able to inspect the filters that are passed into the read_all (which, honestly, I should rename and call it read_many) method and say, if there isn't any reference to the is_deleted column in the filters, then set that as false, otherwise, accept the value in the function parameter. But I find the filter's tuple values quite opaque. Once they are instantiated, they are class-specific, and I'm trying to get at a general way of understanding them. I'll note that this column is guaranteed to be present in the framework. I've modified the declarative base such that every object has the is_deleted. I appreciate any help you might be able to give. It's very likely I'm overthinking this like a numbskull and the answer is painfully obvious. Here's the service class:

you can inspect a ColumnElement structure to look for a certain column like this:

from sqlalchemy.sql import visitors

for elem in visitors.iterate(Project.usergroup == User.usergroup, {}):

   if isinstance(elem, Column) and elem.name == "is_deleted":

       return True

internally, the ORM usually uses visitors.traverse(expression, {}, {"column": visit_column}) but the above is easier to illustrate quickly.

If it were me, I'd likely keep this as an explicit option on read_all(), e.g. read_all(... include_deleted=True), much simpler and won't be called "magic" by anyone, but it can go either way.

also if you know me at all you'd know I would not do the "s.commit()" inside all of the CR/U/D methods.   Your methods that do a "read()" are beginning a transaction and leaving it open implicitly (there's no rollback() or commit() at the end, nor should there be), whereas your CRUD methods are fully committing the transaction for individual INSERT/UPDATE/DELETE statements, and this inconsistency and the difficulty it introduces in what happens when multiple CRUDService methods are called sequentially will eventually lead to architectural problems.

 

--

SQLAlchemy -

The Python SQL Toolkit and Object Relational Mapper

 

http://www.sqlalchemy.org/

 

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.

---

You received this message because you are subscribed to the Google Groups "sqlalchemy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+...@googlegroups.com.

To post to this group, send email to sqlal...@googlegroups.com.

Visit this group at https://groups.google.com/group/sqlalchemy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/da43b503-1e7a-4a0e-bd32-619d96163c1c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andrew Martin]

Thank you, as always, for both the specific answer and the general advice. Much appreciated!

To unsubscribe from this group and stop receiving emails from it, send an email to sqlal...@googlegroups.com.

[Jonathan Vanasco]

FWIW, I found a better approach to a similar problem was to create a dict/object I used to log metadata about the query I wanted... then build the query or analyze it based on that metadata.  All the information is in the sqlalchemy query, but the execution performance a development time was much faster when I stopped analyzing the query and just consulted the dict.

[Andrew Martin]

That's very interesting, Jonathan. Could you show me a quick example of that approach? I'm not sure I *need* to do that, but I think I would learn about SQLAlchemy from such an example and trying to understand it.

On Wed, Jun 26, 2019 at 11:00 AM Jonathan Vanasco <jona...@findmeon.com> wrote:

FWIW, I found a better approach to a similar problem was to create a dict/object I used to log metadata about the query I wanted... then build the query or analyze it based on that metadata.  All the information is in the sqlalchemy query, but the execution performance a development time was much faster when I stopped analyzing the query and just consulted the dict.

--

SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
 
http://www.sqlalchemy.org/
 
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.
---

You received this message because you are subscribed to a topic in the Google Groups "sqlalchemy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sqlalchemy/chGVkNwmKyQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sqlalchemy+...@googlegroups.com.

To post to this group, send email to sqlal...@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/18bed722-c8c3-42a3-97de-a5f58987fdcf%40googlegroups.com.

[Jonathan Vanasco]

On Wednesday, June 26, 2019 at 2:43:44 PM UTC-4, Andrew Martin wrote:

That's very interesting, Jonathan. Could you show me a quick example of that approach? I'm not sure I *need* to do that, but I think I would learn about SQLAlchemy from such an example and trying to understand it.

One large project has an 'internal api' that tries to centralize the sqlalchemy interface.  

Let's say we're searching for a "user".  I would create a dict for the data like this:

    query_metadata = {'requester': 'site-admin',  # admin interface, user interface, user api, etc
                      'filters': {'Username=': 'foo',   # just an internal notation
                                  },
                       'query_data': {},  # information related to what is in the query as it is built

                      }

This payload is basically the same stuff you'd pass to as queryargs to one of your functions above.  We essentially pass it to our version of your CRUD service which then acts on it to generate the query. 

We don't implement this approach on every query - just on a handful of queries that have intense logic with dozens of if/else statements and that connect to multiple "filters".  Stashing this in an easily accessible manner has just been much easier than constantly trying to examine the query to act on it.  

[Andrew Martin]

Oh that's really interesting. Thank you for that. I'll definitely tuck that away in my back pocket. My background is really heavy in raw SQL, and meta-programming raw SQL is *awful.* Debugging sql that writes sql and execs it is not fun. I'm not allowed to use sqlalchemy at work because no one else on the team uses python, and we can't go around implementing stuff in a way that only one person knows how to work on. But I really want to get away from the SQL-only approach in my personal/side projects. For some reason I often find myself really blocked when it comes to sqlalchemy. Every time I approach my databases I just flip to sql mode and totally forget that everything in sqla is just plain python, and I can treat it that way. I see the obvious-level mapping between the two and just kind of hit a block beyond that. I should probably sit down and read the source code to try and get past the block, that way I'm not so surprised by answer that Mike and people like you give me. Anyway, probably TMI. Cheers and thanks!

--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
 
http://www.sqlalchemy.org/
 
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to a topic in the Google Groups "sqlalchemy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sqlalchemy/chGVkNwmKyQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sqlalchemy+...@googlegroups.com.
To post to this group, send email to sqlal...@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy/39a4a326-c384-4c91-909a-40d9c3acb323%40googlegroups.com.