[Proposal][DAL] Making DAL a separate project and a pip package

[Giovanni Barillari]

Hi all.

It's quite a while I see discussions here about converting DAL to a separate project. Since I started working with weppy I've seen a big opportunity to that. I can write down some pro and cons on having DAL on a separate repository on github and maybe installable as a python package:

- Pros:

    - increase the project popularity

    - convoy contributions on a single repository, instead of having many different projects integrating the library and push backs fixes and features from one to another and so on

    - making use of DAL easier for projects external to web2py

    - testing suite directly into the project would allow to power it up

    - relieve the issues from web2py moving the DAL related ones on a separate platform, allowing web2py's developers to focus on the project, and DAL ones to do the same

- Cons:

    - we need to redesign DAL a bit

    - any other?

Talking about the re-design, I've faced some troubles porting DAL into weppy, in particular:

    - DAL calls a lot of libraries from gluon/contrib

    - DAL uses some features which are in web2py, like validators, serializers, HTTP class, sqlhtml, reserved keywords

So, to make DAL a separate project, and preferably a pip installable package, I think we had to:

    - integrate in DAL code the modules you have in contrib (ordered ditct and some adapters)

    - integrate validators in DAL code or make it possible to pass validators to DAL when inited or via subclassing

    - allow the possibility to use custom serializers, avoid expectance of the web2py's ones

    - not raising HTTP exceptions but custom ones, so other projects can catch them as they prefer

    - move out of DAL the rendering with sqlhtml or allow to use custom render functions

    - move testing inside DAL

I know this list is quite large, but it wouldn't be hard to do that.

What do you think?

/Giovanni

[Niphlod]

didn't we try that with gluino and - basically - didn't receive any attention from the community ?!
as for the "steps" ....

    - integrate in DAL code the modules you have in contrib (ordered ditct and some adapters)

not a problem

    - integrate validators in DAL code or make it possible to pass validators to DAL when inited or via subclassing

validators have a pretty specific machinery to let them work with forms, rendering, widgets, etc. I'm +1 on rationalizing "allow this field to be an integer from 0 to 10" but it'll be hard to do it maintaining backward compatibility AND requiring any framework using DAL to provide a full set of their own validators....

    - allow the possibility to use custom serializers, avoid expectance of the web2py's ones

not an issue too, but AFAIK there are some pretty specific serialization gotchas that will basically lock down the implementation to what is already in there

    - not raising HTTP exceptions but custom ones, so other projects can catch them as they prefer

see "validators"

    - move out of DAL the rendering with sqlhtml or allow to use custom render functions

see "validators"

    - move testing inside DAL

last of the problems: test suites are already pretty isolated

[Giovanni Barillari]

Il giorno domenica 26 ottobre 2014 19:03:07 UTC+1, Niphlod ha scritto:

didn't we try that with gluino and - basically - didn't receive any attention from the community ?!
as for the "steps" ....

    - integrate in DAL code the modules you have in contrib (ordered ditct and some adapters)

not a problem

    - integrate validators in DAL code or make it possible to pass validators to DAL when inited or via subclassing

validators have a pretty specific machinery to let them work with forms, rendering, widgets, etc. I'm +1 on rationalizing "allow this field to be an integer from 0 to 10" but it'll be hard to do it maintaining backward compatibility AND requiring any framework using DAL to provide a full set of their own validators....

Ok, maybe moving the validators inside of the DAL is not the right choice, but IMHO to make DAL an 'independent' project I think the flow of dependency has to be changed, from the actual scheme where DAL tries to import web2py validators to a scheme where web2py tells to DAL "use these validators", so that another project can do the same.

Maybe this solution is more accepted?

 

    - allow the possibility to use custom serializers, avoid expectance of the web2py's ones

not an issue too, but AFAIK there are some pretty specific serialization gotchas that will basically lock down the implementation to what is already in there

AFAIK, DAL tries to import web2py's serializers, in fact in _load.py you have an 'have_serializers' variable. The only changes I see as important, as for the validators, is to change the flow: DAL doesn't import serializers, projects using it adds serializers to it.

What do you think about that?

 

    - not raising HTTP exceptions but custom ones, so other projects can catch them as they prefer

see "validators"

I think it's quite different from validators as HTTP is used only in Field code:

def retrieve(self, name, path=None, nameonly=False):
        """
        If `nameonly==True` return (filename, fullfilename) instead of
        (filename, stream)
        """
        self_uploadfield = self.uploadfield
        if self.custom_retrieve:
            return self.custom_retrieve(name, path)
        import gluon.http as http
        if self.authorize or isinstance(self_uploadfield, str):
            row = self.db(self == name).select().first()
            if not row:
                raise http.HTTP(404)
        if self.authorize and not self.authorize(row):
            raise http.HTTP(403)

I think that changing this to 2 custom exception so other projects can catch them as they prefers and web2py can catch them with HTTP 404 and 403 it wouldn't hurt anyone.

 

    - move out of DAL the rendering with sqlhtml or allow to use custom render functions

see "validators"

As for HTTP exception there are only 2 methods using web2py's sqlhtml inside DAL.

1) The render() method of Rows:

 

def render(self, i=None, fields=None):
        """
        Takes an index and returns a copy of the indexed row with values
        transformed via the "represent" attributes of the associated fields.


        Args:
            i: index. If not specified, a generator is returned for iteration
                over all the rows.
            fields: a list of fields to transform (if None, all fields with
                "represent" attributes will be transformed)
        """


        if i is None:
            return (self.render(i, fields=fields) for i in range(len(self)))
        import gluon.sqlhtml as sqlhtml
        row = copy.deepcopy(self.records[i])
        keys = row.keys()
        tables = [f.tablename for f in fields] if fields \
            else [k for k in keys if k != '_extra']
        for table in tables:
            repr_fields = [f.name for f in fields if f.tablename == table] \
                if fields else [k for k in row[table].keys()
                                if (hasattr(self.db[table], k) and
                                    isinstance(self.db[table][k], Field)
                                    and self.db[table][k].represent)]
            for field in repr_fields:
                row[table][field] = sqlhtml.represent(
                    self.db[table][field], row[table][field], row[table])
        if self.compact and len(keys) == 1 and keys[0] != '_extra':
            return row[keys[0]]
        return row

2) the xml() method of Rows:

def xml(self,strict=False,row_name='row',rows_name='rows'):
        """
        Serializes the table using sqlhtml.SQLTABLE (if present)
        """


        if strict:
            return '<%s>\n%s\n</%s>' % (rows_name,
                '\n'.join(row.as_xml(row_name=row_name,
                                     colnames=self.colnames) for
                          row in self), rows_name)


        import gluon.sqlhtml as sqlhtml
        return sqlhtml.SQLTABLE(self).xml()

I'm not saying to cut off functionalities, but since these are web2py's features, if we want to make DAL a separate project they should be moved back to web2py. IMHO having try/excepts blocks inside DAL that tries to import web2py's and raises *you don't have X or Y* wont produce any benefit for an user/project using DAL.

 

    - move testing inside DAL

last of the problems: test suites are already pretty isolated

glad to know ^^

/Giovanni 

[Giovanni Barillari]

Hi all.

I've just pushed to github a temporary repository with a pip version of DAL: https://github.com/gi0baro/pyDAL

The updates I did are the ones posted in my first message of the discussion.

To use it, I personally suggest to clone the repo, create a virtualenv and run 'python setup.py develop'

I've also pushed a special branch of web2py with updated imports, using pydal:

https://github.com/gi0baro/web2py/tree/pydal

The relevant part of pyDAL usage by subclassing is here:

https://github.com/gi0baro/web2py/blob/pydal/gluon/dal/base.py

You can download this version of web2py via the github zip link:

https://github.com/gi0baro/web2py/archive/pydal.zip

and run web2py in the prior activated virtualenv.

Some relevant notes about this web2py version:

- tests are not updated to use pydal

- documentation is untouched

- I haven't converted any contrib file that uses DAL, something can be broken inside this folder

- I've changed the flow about DRIVERS array initialization, so right now this version of web2py BREAKS backward compatibility in gluon/sql.py. If developers will approve pyDAL, we should fix this in someway.

Also, if we want to keep the 'zero setup' philosophy of web2py, pyDAL can be added to gluon or contrib and imported from there instead of using the package.

Please, post your thoughts and feedbacks. 

/Giovanni

[Leonel Câmara]

Can you make a repository for the DAL where the first commit is the unmodified DAL and the second is your changes so it's easier to see what we would need to change in web2py?

[Paolo Valleri]

The best would be to start a new repository that contains ALL commits of dal since the beginning. By doing that you will keep track of all contributions, contributors and so on.

 Paolo

2014-11-21 10:57 GMT+01:00 Leonel Câmara <leonel...@gmail.com>:

Can you make a repository for the DAL where the first commit is the unmodified DAL and the second is your changes so it's easier to see what we would need to change in web2py?

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Giovanni Barillari]

You would like to see the DAL modifications or the web2py ones?

In the second case you can look at the web2py repository I've linked, inspecting the last commit: https://github.com/gi0baro/web2py/commit/b4b25948b3d2c20ded82cf94360c2f00afbc4cc4

Otherwise I can produce a diff from the original DAL.

[Giovanni Barillari]

I think we can do that creating the new repository from the web2py's one.

If the developers approves the current implementation, Massimo can create a new repository on github, under web2py/pydal with the latest web2py master, and then I can make a commit removing everything but DAL, and updating it to this format. This way will keep all the commits from web2py.

/Giovanni

[Massimo DiPierro]

I promise to do this on Sunday. can you wait?

Massimo

[Giovanni Barillari]

Yep, still waiting :)

[Massimo DiPierro]

It is done!

https://github.com/web2py/database-abstraction-layer

[Massimo DiPierro]

Please send me any correction/suggestions. i think we should make it into a PyPI package.

[Massimo DiPierro]

Problem: how do we handle the git history? Now basically this code lives in two repos.

[Giovanni Barillari]

Hi Massimo,
As we said before, I think you should base the repo from the original web2py one, to keep the original commit history and contributors. Basically this will make the job:
1) delete current DAL repo
2) clone the current web2py master
3) change the cloned repo origin to the DAL one
4) create a commit where you delete everything but DAL, move to the upper level and add readme
5) push this to the DAL origin

/Giovanni

You received this message because you are subscribed to a topic in the Google Groups "web2py-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py-developers/ch6ZRfW0AVk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py-develop...@googlegroups.com.

[Giovanni Barillari]

PS: if this would be a pip package I suggest to name it pydal ;-)

[Massimo DiPierro]

OK. Will do after lunch.

[Massimo DiPierro]

I tried but I treats the files in dal/ as new files and not as gluon/dal files. suggestions? Is this ok? Anyway, the histories will diverge from web2py eventually.

https://github.com/web2py/pydal/

On 12/5/14 1:07 PM, Giovanni Barillari wrote:

[Giovanni Barillari]

Seems ok to me.

Should I submit a PR with the edits I did here https://github.com/gi0baro/pyDAL? We still need something about drivers initialization and backward compatibility..

/Giovanni

[Massimo DiPierro]

Here is the problem. Do we submit PRs to pydal or web2py? Do you envision the two to diverge? For now send a PR so I can see it.

Massimo

[Giovanni Barillari]

I think we should separate pyDAL from web2py (obviously stating is from web2py) with this approach:

- update pyDAL to my modifications so we have a working pip package (can be installed without web2py and works without web2py)

- remove old dal from web2py and integrate in one of these ways:

    - as a git submodule

    - as a git subtree

    - copying the pydal module under contrib, updating it with the releases of pyDAL

The pros of this scheme are (my point of view):

    - separate the issues between web2py and DAL (I suggest to use github issues for pydal)

    - allow DAL developers to push fix/new features quickly

    - manage releases of DAL and web2py separately

    - allow us to guarantee pyDAL python3 compatibility easier (I would like to do this in the next months)

To achieve these I think next steps would be:

1) update pyDAL to pip structure

2) move web2py's DAL tests to pyDAL and create a travis CI dedicated to pydal

3) release pyDAL 1.0 when everything works from point 2

4) release a web2py new version without old DAL and using pyDAL 1.0 (obviously backward compatible)

What do you think?

/Giovanni

[Paolo Valleri]

@Massimo there are few PRs against 'old' dal, please consider them before

 Paolo

[Giovanni Barillari]

Doh. I'm making the merge right now for the new structure. Would be such a big deal to make new PRs on pyDAL after the new structure changes?

/Giovanni

[Giovanni Barillari]

Btw, here is the PR: https://github.com/web2py/pydal/pull/1

[Paolo Valleri]

frankly yes

 Paolo

[Giovanni Barillari]

I'm sorry about that.

I supposed wouldn't be hard as the real diff between pydal and PR are: 

* 2 lines in this https://github.com/web2py/web2py/pull/552/files

* 2 lines in this https://github.com/web2py/web2py/pull/550/files

Anyway I can wait the merge of these, and make (again) a new complete PR.

Let me know.

/Giovanni

[Massimo DiPierro]

Flying today. Will process later or tomorrow.

[Niphlod]

@giovanni: made this PR https://github.com/web2py/web2py/pull/566 .... make sure it gets in on pydal before merging/submoduling/etc please

[Giovanni Barillari]

Please, make the PR on pydal repo so we can merge it directly.

/Giovanni

Il 19/dic/2014 00:05 "Niphlod" <nip...@gmail.com> ha scritto:

@giovanni: made this PR https://github.com/web2py/web2py/pull/566 .... make sure it gets in on pydal before merging/submoduling/etc please

[Niphlod]

will do.

[Niphlod]

BTW: has the integration been considered on docs on readthedocs and travis-ci ?
e.g.:
- will the tests be executed on on both web2py and pydal ? Will web2py run pydal's tests too ?
- will the docs be separated? is pydal going to have its own docs ? is web2py going to include docs from pydal ?

[Niphlod]

BTW2: how will this affect the googlecode repo ? I was one of the first stating that github should be "the one and only" but kept getting "we'll maintain googlecode as "backup"" ... there are a few scripts around pointing to googlecode that will need to be adapted if googlecode gets shut down, and frankly I don't see how it can be maintained easily in sync with github and the new structure.........

[Giovanni Barillari]

I would like to hear @massimo on this.

My considerations:

- in the PR I've made on web2py to move from DAL to pyDAL, I've removed old DAL tests, which are implemented directly on pyDAL, and added some basic tests (https://github.com/gi0baro/web2py/blob/pydal-pip/gluon/tests/test_dal.py) on the implementation (should be completed, see the TODO in the file). So IMHO dal tests should be only on pyDAL, 'cause pydal will be a git submodule in web2py and testing it on web2py's repo will means repeat the same tests on the same code.

- I think the DAL documentation inside web2py is written quite well. I don't see any valuable reason to move the documentation on a different location. But, would be good if we can provide just the API documentation (build with readthedocs) for pyDAL in a dedicated website.

BTW2: how will this affect the googlecode repo ? I was one of the first stating that github should be "the one and only" but kept getting "we'll maintain googlecode as "backup"" ... there are a few scripts around pointing to googlecode that will need to be adapted if googlecode gets shut down, and frankly I don't see how it can be maintained easily in sync with github and the new structure.........

 I agree on the "github move", I don't see any trouble about maintaining a googlecode backup, as we can update it just when we do a new release, we just need to update the scripts.

/Giovanni

[Niphlod]

I was merely pointing out some things that should be clear before the merge of the codebases.
All of those are valid ways to implement pydal separation...
I'm +1 on CI-ing pydal and web2py separately (we'll loose just the "total coverage" stats but it's not a major issue) but I fear it'll be more complicated for regular users to test web2py as a full entity (not that I've never seen anyone doing that :-P)
The only "minor" point is that I don't know how readthedocs will behave on web2py with pydal as a submodule: one thing for sure is that if pydal needs its own "separate" docstring documentation, it'll need a docs/ folder with what is in gluon/docs/dal.rst

 

 I agree on the "github move", I don't see any trouble about maintaining a googlecode backup, as we can update it just when we do a new release, we just need to update the scripts.

 

in theory there shouldn't be but there were several occasions in the past where googlecode was left behind. I'm just "advocating" that with the not-so-far-in-the-future migration of issues from googlecode to github, there is yet another one reason to ditch googlecode in its entirety.

[Giovanni Barillari]

I've created a milestone on github: https://github.com/web2py/pydal/milestones/1.0

This should contains all the issues and TODOs we need for releasing the first public release (to be pushed on pypi), which I supposed will be the 1.0, but feel free to propose anything else.

Actually I've put issues for the caching tests and the docs. If you think we need everything else, please add an issue on github.

@niphlod would you like to port the docs files?

/Giovanni

[Niphlod]

it's not a big deal but I don't know what will happen exactly with the merge: do we have to keep some "fixed" directory structure ?

[Paolo Valleri]

I‘d write more tests

--

-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---

You received this message because you are subscribed to the Google Groups "web2py-developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.

[Niphlod]

wait a sec, now we're shipping pg8000 inside contrib inside pydal ?

It's going to be a nightmare the directory structure of the merged web2py........... what about pymysql and pypyodbc ?!?!

are we sure we have an idea of what the result should be before pushing forward ?

[Giovanni Barillari]

pyDAL contains every contrib module used by DAL inside web2py: https://github.com/web2py/pydal/tree/master/pydal/contrib
I've just kept everything like before, I don't now why you ship these drivers inside web2py, and I had no reasons to change this.

Obviously after the merge these contrib module should be removed from web2py to avoid code duplication and confusion.

I think the directory structure is not so important to the users since pyDAL is merged into the 'contrib' folder.

Also remember that with pydal, you don't have to write:

from gluon.contrib.pydal.contrib.module import foo

but just

from pydal.contrib.module import foo

Don't be scared :)

/Giovanni

[Niphlod]

does this mean that we'll have

web2py

     gluon

         contrib

              pydal

                 contrib 

?

[Giovanni Barillari]

Here is the new tree: https://github.com/gi0baro/web2py/tree/pydal-pip

Since the pydal repository is included as a git submodule, you will actually have:

gluon

   contrib

       pydal -> repository

           pydal -> python module

               contrib

What's wrong with that?

/Giovanni

[Paolo Valleri]

I'am not convinced either, the point is that Web2py is 'batteries included' but what about pydal?

We could have pydal with/without contrib, we have to take a decision.

 Paolo

--

[Giovanni Barillari]

We cannot have pyDAL without contrib: some of the modules included are required (simplejson and ordereddict for py2.5, mockimaplib for testing...)

We can revisit the decision about the drivers included which are the same of web2py:

- pg8000

- pymsql

- pypyodbc

As I said before, I don't know why you chosen to integrate these 3 drivers into web2py, and I've just kept the same logic.

I think we have 3 options about this:

1) keep the actual scheme, motivating why pyDAL includes these 3 drivers and why doesn't include the others

2) remove these 3 drivers from pyDAL and keep them into web2py, which can add the drivers to pyDAL context easily:

    from gluon.contrib import pg8000

    from pydal.drivers import DRIVERS

    DRIVERS['pg8000'] = pg8000

3) keep the drivers in web2py and add drivers in pyDAL with a pypi requirement (a motivation is needed as for the point 1 about which drivers we include)

I personally vote for 1 or 2.

/Giovanni

[Giovanni Barillari]

A further consideration:

If we remove included web2py drivers from pyDAL and you want to tests the ones into web2py, this will require a dedicated tests in web2py.

OT:

I got the difference between psycopg2 and pg8000 testings, but what's the difference between postgres and postgres2 adapter testings? They both runs on psycopg2, aren't them?

/Giovanni

[Paolo Valleri]

postgres2 uses NewPostgreSQLAdapter which provides a 'native' data type for list:*

postgres uses PostgreSQLAdapter and list:* are mapped to TEXT

 Paolo

[Massimo DiPierro]

I have a strong preference for 2.

[Niphlod]

nothing is stopping us but I'm just raising "concern points" on the overall results.
before merging there are always some things to consider, and now is the time to do it.

What I don't like of the "current" status on the merge:
- no coverage
- no docs
- weird folder structure for contribs
- test separation ('cause pydal won't ever be able to run tests depending on something related to web2py strictly, e.g. validate_and_insert() tests)

coverage and docs are merely implementation issues.... weird folder structure and test separation needs a "clear path" to follow before going further on with the implementation.

Having
    gluon
        contrib
             pydal
                pydal
                   contrib

is a PITA. Nothing will make me consider this as a "sane" scheme, a) for the "double" pydal folder and b) for the two "contrib" folder.
As for the "rules" of modules in contribs (web2py-side) the point was to make available any pure-python driver module available because of the "batteries included" nature of web2py.
Now, if pyDAL wants to follow the same "batteries included" mode, we should avoid having drivers in two places: if instead pyDAL remains a module who can optionally use compatible drivers installed on the system, it'll be more of a python module like any others, with a fixed number of - optional - dependencies (and, to be fair, I'd prefer it).
At that point, though, the imports at the top would have to be rewritten to use the available system-wide modules (to use pyDAL as a standalone package) and web2py would have to use system-wide modules before the ones in web2py-contrib. Don't know about the reliability of such system.

[Massimo DiPierro]

I agree.

[Massimo DiPierro]

I agree.

On Dec 26, 2014, at 9:05 AM, Niphlod <nip...@gmail.com> wrote:

[Giovanni Barillari]

Il giorno venerdì 26 dicembre 2014 16:05:00 UTC+1, Niphlod ha scritto:

nothing is stopping us but I'm just raising "concern points" on the overall results.
before merging there are always some things to consider, and now is the time to do it.

I agree. You raised a good point.

 

What I don't like of the "current" status on the merge:
- no coverage
- no docs
- weird folder structure for contribs
- test separation ('cause pydal won't ever be able to run tests depending on something related to web2py strictly, e.g. validate_and_insert() tests)

coverage and docs are merely implementation issues....

I agree on this. 

Docs are actually quite ready, since they're still inside the code and can be produced via sphinx and stored on readthedocs. We just need to create a 'docs' folder with the new tree and the sphinx configuration.

Coverage is the same: we just need to properly configure it.

These also means that the 'dal' part of the sphinx docs should be removed from web2py because points to the older implementation.

 

weird folder structure and test separation needs a "clear path" to follow before going further on with the implementation.

Having
    gluon
        contrib
             pydal
                pydal
                   contrib

is a PITA. Nothing will make me consider this as a "sane" scheme, a) for the "double" pydal folder and b) for the two "contrib" folder.

I don't agree on that.

I mean, the actual merge scheme is "awful", yes.

But actually I can't see any main reason to consider it as a PITA. My reasons are:

1) the meaning of having a 'contrib' folder is actually to store in it external modules/packages/features which are not involved into web2py "main development". I can object with the same logic that I don't like to see modules in that folder that have a dedicated 'tests' folder, because can make confusion with the gluon/tests folder. But as I said, here we have "external" modules, so why a module cannot have its own tests/docs/libraries?

2) users' side: I'm quite sure a web2py user doesn't care about "contrib" folder substructure

3) developers' side: the actual implementation via a git-submodule is actually very "clear" to a developer, as it points DIRECTLY to the pydal repository, meaning that if I'm a developer I can understand on what I'm working on in a snap. Moreover, the git sub-module allow web2py to track a particular release of pyDAL. Which means you can release web2py 2.10.x with pydal 1.x and web2py 2.10.y with pydal 1.y simply running "git pull" and "git checkout v1.y" inside the contrib/pydal folder.

The only way I see to implement a different directory structure is to "craft" a copy of pyDAL code on a new release, which leads to:

1) code duplication

2) a hard issues management

3) to rebuild a copy of pyDAL code every time you want to update it.

Now, I consider this 2nd option a PITA, not the first. If we're talking just about the "directory" structure, does the game worth the effort of choose this 2nd option?

 

As for the "rules" of modules in contribs (web2py-side) the point was to make available any pure-python driver module available because of the "batteries included" nature of web2py.
Now, if pyDAL wants to follow the same "batteries included" mode, we should avoid having drivers in two places: if instead pyDAL remains a module who can optionally use compatible drivers installed on the system, it'll be more of a python module like any others, with a fixed number of - optional - dependencies (and, to be fair, I'd prefer it).
At that point, though, the imports at the top would have to be rewritten to use the available system-wide modules (to use pyDAL as a standalone package) and web2py would have to use system-wide modules before the ones in web2py-contrib. Don't know about the reliability of such system.

I agree on Massimo about the "batteries included" talk.

I also prefer a pyDAL without included adapters, and without dependencies, as let anyone install the adapter needed/preferred.

This wouldn't be hard on web2py's side: as I stated above, pyDAL allows adapters injection, so web2py can check the availability of system installed adapters and if not available inject the ones in the contrib folder.

At this point I suggest @paolo updates the pg8000 adapter also on web2py, so we can then remove it from pyDAL.

/Giovanni

[Giovanni Barillari]

Another consideration I forgot, about tests:

Actually pyDAL has the validation test, which is Implemented thanks to one validator taken from web2py (but can be any class written to validate an input, I've chosen one from web2py for simplicity).

I think pyDAL should be tested on every feature, because the actual scheme allows everyone to implement it's own validators/caching/ecc..

So IMHO, DAL's tests should be on pyDAL. web2py should contains tests on pyDAL particular implementation into web2py.

Right now I think we should implement:

- on pyDAL:

    - re-implement caching test (I just need to write a simple caching class)

    - tests other adapters (compatibly with travis services)

- on web2py:

    - test web2py default validators for fields

    - test web2py caching on pyDAL

    - test web2py serialization with DAL

    - test what I called "representers" (SQLTABLE and sqlhtml.represent)

I think this would be the best testing we can set up for pyDAL and web2py with pyDAL.

/Giovanni

Il giorno venerdì 26 dicembre 2014 16:05:00 UTC+1, Niphlod ha scritto:

[Paolo Valleri]

@giovanni the new pg8000 involves an update in the postgres adapter. As a result, I can

- update pg8000 in web2py after the integration of pydal

- update pg8000 in web2py and update the current (old) dal

What do you think?

 Paolo

--

[Giovanni Barillari]

Ok, let's wait for pydal integration.

You received this message because you are subscribed to a topic in the Google Groups "web2py-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py-developers/ch6ZRfW0AVk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py-develop...@googlegroups.com.

[Niphlod]

leaving aside coverage and docs (working on that)

 

I mean, the actual merge scheme is "awful", yes.

But actually I can't see any main reason to consider it as a PITA. My reasons are:

1) the meaning of having a 'contrib' folder is actually to store in it external modules/packages/features which are not involved into web2py "main development". I can object with the same logic that I don't like to see modules in that folder that have a dedicated 'tests' folder, because can make confusion with the gluon/tests folder. But as I said, here we have "external" modules, so why a module cannot have its own tests/docs/libraries?

2) users' side: I'm quite sure a web2py user doesn't care about "contrib" folder substructure

3) developers' side: the actual implementation via a git-submodule is actually very "clear" to a developer, as it points DIRECTLY to the pydal repository, meaning that if I'm a developer I can understand on what I'm working on in a snap. Moreover, the git sub-module allow web2py to track a particular release of pyDAL. Which means you can release web2py 2.10.x with pydal 1.x and web2py 2.10.y with pydal 1.y simply running "git pull" and "git checkout v1.y" inside the contrib/pydal folder.

good points ...
Agreed on the "pyDAL should not include contribs" matter, it remains "weird" because of the "double" pydal folder.
As - especially the "old ones" - developers know, I'm the last person to reach when in need of a solid and clean naming scheme (we all know my brain isn't cut for naming things), but the fact that I actually don't like it raises an alert ;-P

Can't we make a

gluon
     packages <-- instead of pydal
          pydal
     contrib

"kinda" structure ?

[Niphlod]

BTW: docs and coverage have been ironed out... on the coverage PR there are a few considerations to make, but it's a start nonetheless

[Niphlod]

BTW2: docs fail to build the adapter from google due to import issues:

pydal/adapters/google.py", line 8, in <module>
    from .._gae import classobj, gae, ndb, NDBDecimalProperty, GAEDecimalProperty, \
ImportError: cannot import name ndb

Frankly, this happens also on web2py, but I really don't know how to fix it either locally or in readthedocs (wherever appengine is not on pythonpath)

[Giovanni Barillari]

Yes, we can make the structure you proposed, but the "weirdness" would be quite the same, looking like this:

Gluon
Packages
Pydal -> repository
Pydal -> pydal module inside the repository

Actually, the double "pydal" naming is because the "pydal" repo is linked inside web2py as a git submodule, and the repository has the "pydal" module inside it.

So, +1 for the packages directory for pydal as it clarify the "contrib" issue, but I don't know how to avoid the "double pydal folder".

/Giovanni

[Niphlod]

shoot, I'm not a proficient submoduler but I thought we could achieve

packages
     pydal

is this "double weirdness" a common thing (i.e. a requirement) in git repositories that have a submodule ?

[Giovanni Barillari]

Well that's actually a consequence of using a submodule. A git submodule is basically an inclusion of a repository into another one using a folder.
The "duplication" of the "pydal" directory comes from the fact I named "pydal" the folder into I linked the repository, and the pydal is structured with a "package design" so the code is under a "pydal" directory.

The main issue here is that web2py has the "zero setup" design, so we cannot use pypi to use pydal.

Considering the purpose of the 2 projects, and to avoid code duplication, I think a submodule is the best solution we have. Even using a git subtree will cause a code duplication and, consequentially, an harder management of the development flow.

I'm actually out of ideas on how to better design the structure of the inclusion.
If it's just a naming factor, we can do something like that:
gluon
   Packages
       Dal -> repository
           Pydal -> repository pydal folder

But I think is only a "masquerading" thing..

/Giovanni

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---

You received this message because you are subscribed to a topic in the Google Groups "web2py-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py-developers/ch6ZRfW0AVk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py-develop...@googlegroups.com.

[Massimo DiPierro]

This looks better to me but I would go with lower case.

web2py/

   packages/

      dal/

        pydal/

You received this message because you are subscribed to the Google Groups "web2py-developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.

[Giovanni Barillari]

@massimo @niphlod

Is this directory structure better? https://github.com/gi0baro/web2py/tree/pydal-pip/gluon

/Giovanni

[Massimo DiPierro]

very much

[Giovanni Barillari]

Ok, will wait for this to be merged, then will update codes to remove adapters from pydal and use back the ones from web2py.

/Giovanni

[Niphlod]

the new structure seems more legit, +1 on that.

BTW: sorry for being so nitpicky with the naming, but it just felt ugly.