ahhhhhh!

[Massimo DiPierro]

I merged the pydal pull request.

Everything broke and I am not sure how to fix it, other than reverting….

The code now imports pydal but there i no pydal. There is a gluon/pacakges/dal and it is empty.
I tried to clone the latest web2py and still there is no dal or pydal.

I am missing something. I upgraded git from 1.9.4 to 2.2.1 but no change.

Any advice?

Massimo

[Marin Pranjić]

Hi Massimo,

is it submodule now?

Can you try git submodule init && git submodule update ?

Marin

Massimo

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details    : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official    : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Massimo DiPierro]

Thank you that worked but when somebody does

git clone https://github.com/web2py/web2py

they should get gluon/packages/dal and they should not need to run git submodule commands. they do not know. I did not know. Any advice?

Massimo

On Jan 23, 2015, at 2:47 PM, Marin Pranjić <marin....@gmail.com> wrote:

Hi Massimo,

is it submodule now?

Can you try git c ?

Marin

[Marin Pranjić]

Instead you can do git clone --recursive

Alternative is to use subtree instead of submodule but I never used it.

Others might know more about it :)

Marin

[Massimo DiPierro]

I had to revert this. It is not acceptable that users have to type ‘

git clone —recursive

instead of

git clone 

to get web2py. Users do not know and they will think web2y does not work.

I read a little more on the topic and I have concluded a subtree is better than a submodule.

http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/

I agree with what is said in that article.

For now I have reverted the patch.

I think pydal should be a subtree under gluon.packages and not a submodule.

Massimo

[Giovanni Barillari]

First of all, I'm sorry about the mess. I clearly forgot to make accurate specs on new solution.

Now, we have a problem.

This is how it looks like with a git subtree: https://github.com/gi0baro/web2py/tree/pydal-pip/gluon/packages/dal

Which is exactly the same thing as copy-paste the code of pydal inside web2py when you want to.

Is awful.

Code duplication, merging is a mess, if someone make a code change into pydal folder and someone merge it unintentionally, you loose the reference from the pydal code.

I think we should take a decision.

It's basically 'coding and contribution organisation' vs '1 more command for advanced users who use the master cloned from git'.

Honestly, if duplicating the code is the chosen solution for the 'integration', I think it would be really hard to keep everything together.

It would make more sense to craft a 'monolithic' version of pydal into web2py's contrib. But also quite awful.

@massimo: how many users use the 'master'? how do they download it? I think the zip/tar.gz generated for releases and/or download button of github should work fine with submodules.

OT: @everybody is the 'master' branch of web2py considered 'usable' or 'experimental'? I'm probably missing something here.

/Giovanni

[Massimo DiPierro]

Don’t be sorry. This is my fault more than anybody else. I should have been on top of things.

As a general rule, It is ok to make life a little more difficult to developers. It is not ok to make like more difficult to users. There are more and more user who use git as a means of delivery. Yes it is an experimental branch but people can use tags to got back to a previous stable version.

Massimo

More and more people use 

For more options, visithttps://groups.google.com/d/optout.

-- 
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
--- 
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.

For more options, visithttps://groups.google.com/d/optout.

[Niphlod]

uhm...we discussed "the shift" in over a month and it seemed that I was the only one raising hands... we settled for a few shortcomings in favour of "promoting" yet another dal-exported trial (remember gluino?!). now that we're finally ready to go....

github is a repository... developers shouldn't have issues with a submodule.
newbies in git world need to master a SINGLE argument in addition to the usual, and we planned beforehand all the steps involved (announcements and a specific section on the book).
Are we really going to toss everything in favour of git clone instead of git clone --recursive ?

[Giovanni Barillari]

Hi all.

I've made another branch with pyDAL integration using neither git submodules or subtrees: https://github.com/gi0baro/web2py/tree/pydal-fksub

The result is similar to subtree, the difference is that I cloned pyDAL repository inside the web2py's one.

The usage is simpler compared to subtree (assuming you are inside web2py repo):

git clone https://github.com/web2py/pydal.git gluon/packages/dal
git add gluon/packages/dal/
cd gluon/packages/dal
git checkout v0.12.25

Then commit the result. (which duplicate the code, yes)

Now, you now what I think about code duplication, I've already expressed myself about it.

I think this is quite awful too.

But if we really want to avoid git submodules 'problems', I think this is a bit easier than git subtree. Under this scenario we should add something to the repository warning developers about packages/dal folder, like "DON'T EVEN THINK TO TOUCH THIS -> go to pydal", which is awful too.

I think we should really think about all this before making a decision.

I still see too many problems deriving from code duplication, and only one on submodule (and this is why I picked submodule from the beginning).

[Massimo DiPierro]

I do not recall it ever came up that the process to “git clone” would change. This was not discussed. This is a new fact that came up and I consider very important.

Nothing else has changed. The fact that we would distribute pydal. The fact that the book would discuss it. Etc.

Now I want to understand the implication of the solution that Giovanni just proposed. What is your opinion on that?

Massimo

[Massimo DiPierro]

Why not a normal subtree? This seems to be exactly what we want?

https://medium.com/@v/git-subtrees-a-tutorial-6ff568381844

It works as if we had a copy but commits in the subtree got to the subproject.

[Niphlod]

I though the main deal about having pydal as a separate project (docs, tests, separate issues, the whole "new folder structure naming with me being nitpicking") was enough to "enlighten" that the repo structure would change...

By experience accumulated so far, it seems that I'm the only one always rambling about something in web2py that I don't like. I'm kinda scared I'm going to be "that guy" just because I speak about boring things like tests, integrations, rainy days and so on.
Unfortunately my humble opinion, if I'm still allowed to explain my POV, is that every - great - project outside web2py eventually dies (gluino, templating, web2py-celery, whoosh/haystack, mvvm integration, etc) and new ones are REALLY hard to include (or nobody wants it, like the long-standing issue of using python-social-auth).
Web2py is "scared" of depending on external modules, but it's slowly choking because maintainers can't keep up with the pressure of a world of other developers doing wonderful and reusable things.
Plugins are more and more discontinued, recipes on web2pyslices.com are unmaintained and eventually we see people leaving web2py just because you can't really hook up in it unless you know the full picture....
Yes, it's feature-packed with parts that are getting more and more integrated ........ but, tl;dr you basically can't hook your new idea outside webp2y core: it has to stay inside.

The problem we faced so far with lots of wonderful contributions is that we're eager to pull those in, but we're bad at maintaining it, because eventually the original contributor just made it up for himself and never cared to add docs or tests for it, or think to the broader usecase.
Heck, even seeing a bs3 working scaffolding app has been pushed back for months, just because we don't have a clean way (or idea) to hook up in form generation and we're scared of overriding default widgets. bs3 is out for 1,5 years and other frameworks had clean scaffolds one month later (in addition to other css framework but hey, just one would suffice). I had a working POC in cs_monitor_plugin and it took me, that I know web2py inside-out, a couple of weeks to figure it out partially. But again, nobody cared because it's outside core, and things outside core are never considered.

The idea of submodules is (IMHO) a perfect fit (or Yet Another Way) to keep things linked to the main project and attract more devs, hopefully speeding up development and contributions.
I'd like to see DAL as a submodule and thrive with its own pack of devs faster than what it'll take it under web2py. I envision pydal always being rather stable and "tag" monhtly releases, with web2py pulling in every month a new "tagged" release of pydal.
Then I'd like to see web2py structuring its code to accept hooks with "plugins" that are maintained as separate projects, ala wheezy, but still shipped as submodules (that is not a great deal of pain for a developer)
If the whole thing won't work, we would still have to maintain the exact same amount of code, but hopefully it'll be organized better and the clutter among modules would be less, so touching one thing wouldn't disrupt others as frequently as now.

[Massimo DiPierro]

I agree. Perhaps there is semething I do not understand about subtree but,as far I understand, making it a subtree does not change the status of pydal. It only changes the status relative to web2py. Web2py knows it is a subproject and makes commits there. When it pulls, it pulls the subtree form the subtree repo. 

the pydal would continue to have its own independent repo. Its commits would go there. Am I getting this wrong?

Massimo

--

[Niphlod]

with a subtree we'd still accepting changes in web2py repo.
subtree is substantially a copy, and copies are double the work to maintain.
changes made to web2py subtree would have to be pushed back to pydal, doubling the commit efforts and tracking, and eventually painful merges.
same goes for changes to pydal.
git submodule is about separation

What we - really ! - want to leverage is pydal making big steps forward incorporating devs outside of web2py and leverage those great ideas in a small,not-so-tiny, but completely unrelated repo.
Commits go only there, issues are hopefully raised only there, tests gets added, pydal shines.
pydal then "tags" a "golden release" (hopefully faster than web2py) when things are stable enough (I like the idea of "ubuntu" numbering, such as YYYY.MM), and web2py pull those changes in - let's say - once a month, with a proper single well-done PR affecting one file. web2py maintainers needs to learn about doing git submodule update once a month. git newbies wanting to pull in the whole thing needs to append a --recursive to the git clone command. is really a big deal ?

[Michele Comitini]

I think niphlod got very important points here.

It is good to have things like pydal that force web2py to keep up with their pace, not only the other way.

Well maintained components give little burden on a full distribution, because their ultimate goal is aiming for broader usage, so they make more effort to behave well with anyone. Also because of this they are more receptive to needed changes and innovation imposed by dependent projects.  More the projects dependent dependent on a common component, better is the component in the long run.

DAL (aka pyDAL) was driven only by the needs of one project, now let's hope many others will chime in.

I'd expect to see a bit more integration effort in the short term, but evolution at much faster pace in the mid term.

I really do not think that the --recursive will be a huge problem for developers.

If such problems exists for non-developers, we could think of a clone that has already all dependencies resolved. 
Think of the linux kernel tree, if you want you can integrate newer drivers from other sources, or stick with the "official" clone and keep drivers that come with it.

--

[Massimo DiPierro]

OK. I chatted with Simone and he convinced that I have been dumb. he always does that and he is always right!

The solution is the one already agreed upon but we will perform a check in web2py so that, when it starts, it checks that pydal is there and complains gracefully.

Also as web2py is committed to backward compatibility we have to make it clear that pydal is also committed to the same.

I will revert the original pull request from Giovanni.

Massimo

[Niphlod]

that being said, if we're still going for submodule, I think that @giovanni (he made all the effort, one step more won't hurt) can explain here what web2py will do to:
- pull in tagged releases of pydal
- update the submodule once the new tagged release has been pulled in
- clone and fork the first time

in a "quick developers guide" that we can put on the readme and on the book.


EDIT: just because I'm dwelling in submodules for a month now, while it seems that others have a more clean picture.

[Niphlod]

btw, but don't freak out: github.com doesn't include submodules in their "automatic" archives (i.e. https://github.com/web2py/web2py/archive/master.zip won't include pydal contents). I was always eager to see github.com being our "reference" for source and binaries archives (as opposed to the current http://www.web2py.com/examples/static/web2py_*.* scheme) but EVEN if we shift to github we can post attachment to the releases.

[Massimo DiPierro]

We should email github and complain.

On Jan 24, 2015, at 4:16 PM, Niphlod <nip...@gmail.com> wrote:

btw, but don't freak out: github.com doesn't include submodules in their "automatic" archives (i.e. https://github.com/web2py/web2py/archive/master.zip won't include pydal contents). I was always eager to see github.com being our "reference" for source and binaries archives (as opposed to the current http://www.web2py.com/examples/static/web2py_*.* scheme) but EVEN if we shift to github we can post attachment to the releases.

[Niphlod]

unfortunately, they already stated that they use git archive.
once git archive will support submodules they'll start supporting it.
they don't plan to change the current behaviour until then.

[Massimo DiPierro]

then we need to submit a pull request to git. Will it make git self conscious?

[Niphlod]

it'll still be a long process. but one can always hope.

[Giovanni Barillari]

Wow, it was a long reading.

Here is what I think for the next steps:

- I will post a new PR (always tracking pydal 0.12.25) with an edit on gluon's __init__.py to produce something like this:

try:
   sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), "packages", "dal"))

   import pydal
except:
   raise RuntimeError(
       "Seems that you haven't pydal inside your web2py's copy.\nIf you cloned" +
       "the repository without '--recursive' option, please run 'git submodule" +
       " update' inside your web2py copy.\nIf you downloaded the package from" +
       " github with the download button, please download a released copy from" +
       " http://www.web2py.com or one attached to the releases on github."
    )

- @massimo will merge the new PR

- @simone can update the pg8000 adapter

- with the new pg8000 adapter we can track the current pyDAL master

- as soon as we release the first 'stable' version of pyDAL we can track it inside web2py

Is that ok to you? Wanna another message?

that being said, if we're still going for submodule, I think that @giovanni (he made all the effort, one step more won't hurt) can explain here what web2py will do to:
- pull in tagged releases of pydal
- update the submodule once the new tagged release has been pulled in
- clone and fork the first time

1) clone first time

    git clone <address> --recursive

2) to update the submodule to track a new tag is quite simple

    - enter in the 'gluon/packages/dal' directory

    - git checkout master

    - git pull

    - git checkout vX.Y.Z

    - back to the root web2py's folder and commit

3) what do you mean with fork? you can always do whatever you want inside the submodule as it's the same as having a cloned pydal repo. Until you don't commit on web2py's repo, what you do in dal folder is actually on pydal. When you checkout somewhen and committing it to web2py, then you change the tracking of one repo to the another :)

/Giovanni

[Massimo DiPierro]

Fine with everything.

One more thing. Since we are not importing from pydal and not gluon.dal we could re-instate gluon/dal.py.

It would simply “from pydal import *”

Why? so if people skip the 2.9.12 automatic upgrade everything will work fine, even if they are doing from gluon.dal import whatever.

Do you agree.

Massimo

[Giovanni Barillari]

Except that we would have a conflict due to the subclassing:

from pydal import DAL as pyDAL

and it won't change anything since pyDAL doesn't have an __all__ defined in it's init.

My gluon/dal.py makes available these imports:

- DAL

- Field

- SQLCustomType

- geoPoint

- geoLine

- geoPoligon

For the other classes/functions of DAL it's the same for 2.9.x -> 2.9.12 where all imports are changed due to 'modular' DAL.

I would be more happy with accurate 'Release notes' or upgrading guides/scripts.

btw, I will post the new PR tomorrow.

/Giovanni

[Massimo DiPierro]

The main issue is that in upgrading from 2.9.11 to 2.9.12 dal.py is missing and on upgrade it does not get deleted. So any import dal fails because looks for the old dal.py instead of the new /dal/__init__.py. So I think any dal.py will do not because people have to really import form it but because we do not want the upgrade to fail.

This until we have a better mechanism where scripts are run on upgrade.

[Giovanni Barillari]

Ok, the best way I can think about is to manually import inside new gluon/dal.py the old imports we want to keep available.

This is because it would be non-sense writing an __all__ array inside pydal, and also because some stuffs SHOULD NOT be imported unless you know what you're doing quite good.

I will post the PR without more imports, leaving you and others the responsibility to pick the right ones :)

/Giovanni

To unsubscribe from this group and stop receiving emails from it, send an email to web2py-developers+unsub...@googlegroups.com.

[Massimo DiPierro]

Ok

[Niphlod]

watch out for checking statically a folder because in binaries I don't think it'll work.

[Giovanni Barillari]

Suggestions?

Il 25/gen/2015 12:15 "Niphlod" <nip...@gmail.com> ha scritto:

watch out for checking statically a folder because in binaries I don't think it'll work.

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---

You received this message because you are subscribed to a topic in the Google Groups "web2py-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py-developers/Ado7IS-l1mo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py-develop...@googlegroups.com.

[Giovanni Barillari]

Here is the new PR: https://github.com/web2py/web2py/pull/599

[Massimo DiPierro]

It is done!

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---

You received this message because you are subscribed to the Google Groups "web2py-developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.

[Niphlod]

Am I the only one with current trunk broken ?


 File "./web2py.py", line 18, in <module>
    import gluon.widget
  File "/home/niphlod/Scrivania/web2py_source/web2py/gluon/__init__.py", line 39, in <module>
    from sqlhtml import SQLFORM, SQLTABLE
  File "/home/niphlod/Scrivania/web2py_source/web2py/gluon/sqlhtml.py", line 34, in <module>
    from gluon.dal import _default_validators
ImportError: cannot import name _default_validators

from all the import-turnaround, it seems that the import on gluon.widget triggers gluon/__init__.py that adds pydal.... but when sqlhtml is imported (and depends on gluon.dal), gluon.dal is not there yet

[niphlod]

ok, perfect. Was starting to doubt my git-fu ^_^'

take a look also into https://github.com/web2py/web2py/pull/601 , could be originated by the same issue

PS: don't forget to keep web2py-developers on cc ... I'm afraid we'd loose track/view of what's going on....

2015-01-27 0:56 GMT+01:00 Giovanni Barillari <giovanni....@gmail.com>:

I will fix this in the next hours..

[Giovanni Barillari]

Ok.. Something strange around this. I've just cloned the master and it works like a charm.

Still investigating this..

[Giovanni Barillari]

@niphlod I think you have some conflicts on your repo. 

Tested right now with a "--recursive" cloned repo and a 'git submodule init && git submodule update'

working on both.

Maybe you still have a 'dal' directory inside gluon?

[Niphlod]

I was definitely drained yesterday. seemed strange to me also that travis-ci reported no issue but I called the night and left for bed. As soon as I get home I'll recheck.

[Paolo Valleri]

I'd the same problem by running on my local repository the usual 'git merge upstream/master'

After the merge all files in the former dal/*.pyc were preserved, which cause the issue.

After removing all dal/*.pyc everything starts working

 Paolo

[Paolo Valleri]

As far as travis-ci is concerned, all dal tests have been removed from web2py. 

As a result, we are running web2py without any test or at least "integration test" against the new pydal.

I would test pydal even inside web2py, namely running pydal tests in web2py too.

 Paolo

[Giovanni Barillari]

Why we should duplicate all the tests of DAL?

I've opened this: https://github.com/web2py/web2py/issues/759

My idea is to only re-do pyDAL tests for contrib adapters. This would make sense.

I don't see any reason to repeat tests we do on pyDAL also on web2py.

What do you think about this?

/Giovanni

[Paolo Valleri]

I like your idea.

My worry is that: web2py has it own environment, which it is different from pydal env.

Are we really sure pydal will always work in web2py env?

In addition, pydal's contrib are going to keep updated in line with web2py contrib or not?

 Paolo

[Giovanni Barillari]

Since pydal is included into web2py as a package, it's not influenced by web2py's end, so i'm confident when I say "pydal will always work inside web2py where pydal works itself".

About contrib: pydal has its own contrib folder and will never be in conflict with web2py's one as no one of the two project imports stuffs from the other one contrib. So pydal's contrib will be updated when we do it on pydal and vice versa on web2py.

You received this message because you are subscribed to a topic in the Google Groups "web2py-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py-developers/Ado7IS-l1mo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py-develop...@googlegroups.com.

[Niphlod]

two little notes: 

1) keep sys.path clean. 

It's always been [ web2py_root, web2py/gluon, root/site-packages] . I kinda get that web2py/gluon/packages/dal needs to be added (not inserted) because we don't have a packages/__init__.py (should we ?) . Everything else should be reachable without issues and without further altering sys.path...

2) @gi0baro: we settled for testing adapters in pydal and I'm ok with that, but I think there are / there will be things in web2py tied to pyDAL that we need to test in web2py (uhm, like db validators ....). Relying on pydal's developers to "vouch" for web2py compatibility isn't going to cut the deal... I would feel better (I consider myself both a web2py dev and pydal dev)  if web2py-intertwined-pydal tests are left on web2py test-suite, even if part of them is on pydal-specific things. we've been bitten in the past already.

[Massimo DiPierro]

1) I agree. I don’t even thing dal should be in sys.path. web2py knows where it is.

2) I think web2py should also run all of the dal tests

3) pydal requires portalocker. I think we should move it in pydal/dal/contrib and web2py should import it from there

[Richard Vézina]

Niphold,

I really appreciate your guidance and vision regarding web2py developpement...

I don't want to talk about the submodule or subtree thing, but about the other points you raise (un-maintained web2py stuffs)... 

First, I am very curious to know more about our community and how it uses web2py. As far as I can see, it seems to be that web2py is pretty much use in intranet as easy way to build data centric webapp and databases (it what I am doing)... I think about your different concern, and try to identify reasons for such situation. 

Why are thing un-maintained? 

Possibles reasons :

- Backward compatibilty make them reusable during a long time

- There is too many differents locations where to put stuff and no one is good enough (git/bitbuchet repo, web2pyslices, mailing-list, blog, SO, etc.)

- App base on web2py don't evolve as fast as web2py itself (I can't follow the rythm since a year for some personnal reason and because app get more and more complex, so time is the limit in my case)...

- People has differents skill (still search for time and how it the best way to integrate TDD in my app dev pratices).

- People come and go

- Angularjs and orther js framework emerge since web2py inception

- ...

- I would suggest also that there is no project until now that has regroup us... I see so many question/needs on the list that never really address. Many of mine, many recurrent...

- People start from web2py existing demo stuff build there app and improve them overtime share some of their work but not all (that ok, it how it is)

- There is no "real" product made with web2py (here I just say that, I don't know, what I mean by real product is something wildly used by client, like some Django projects that we beleive and are used wildly, I don't know Django ecosystem    but I know it is used by many online company).

Now I will develop base on some of my last points...

If we were more informed of the projects that use web2py (app contest, board of apps that use web2py, ???), maybe it could catch more attention and interest more poeple. We maybe demonstrate that web2py is a really good solution for developping big web app. Maybe we can make a poll and ask web2py user what they needs, and how we can improve their productivity with features they need, but have to build custom solution and reinvented the wheel (losing time, but at the same time it lets us create differents ways to solve the same issue with the possibility to create new better solutions, though we need to share the found solutions). Maybe by knowing what users are building with web2py we can focus on the needs of the majority of us... I see more and more situation where people build a framework for building their app... What lead me to think that real app (company backed app) don't try to select the best framework but build framework answering their needs... Finally, and speaking for my by the way, I think that datacentric app with reusable sophisticated widget, packed well thinked database model and including analytics (graph display, data analysis routine linear regression for instance) and compliance in mind could be such an app project where we can share our best code where it counts and get rewarded by others best contributions.

What do you think?

Richard

On Sat, Jan 24, 2015 at 3:29 PM, Niphlod <nip...@gmail.com> wrote:

I though the main deal about having pydal as a separate project (docs, tests, separate issues, the whole "new folder structure naming with me being nitpicking") was enough to "enlighten" that the repo structure would change...

By experience accumulated so far, it seems that I'm the only one always rambling about something in web2py that I don't like. I'm kinda scared I'm going to be "that guy" just because I speak about boring things like tests, integrations, rainy days and so on.
Unfortunately my humble opinion, if I'm still allowed to explain my POV, is that every - great - project outside web2py eventually dies (gluino, templating, web2py-celery, whoosh/haystack, mvvm integration, etc) and new ones are REALLY hard to include (or nobody wants it, like the long-standing issue of using python-social-auth).
Web2py is "scared" of depending on external modules, but it's slowly choking because maintainers can't keep up with the pressure of a world of other developers doing wonderful and reusable things.
Plugins are more and more discontinued, recipes on web2pyslices.com are unmaintained and eventually we see people leaving web2py just because you can't really hook up in it unless you know the full picture....
Yes, it's feature-packed with parts that are getting more and more integrated ........ but, tl;dr you basically can't hook your new idea outside webp2y core: it has to stay inside.

The problem we faced so far with lots of wonderful contributions is that we're eager to pull those in, but we're bad at maintaining it, because eventually the original contributor just made it up for himself and never cared to add docs or tests for it, or think to the broader usecase.
Heck, even seeing a bs3 working scaffolding app has been pushed back for months, just because we don't have a clean way (or idea) to hook up in form generation and we're scared of overriding default widgets. bs3 is out for 1,5 years and other frameworks had clean scaffolds one month later (in addition to other css framework but hey, just one would suffice). I had a working POC in cs_monitor_plugin and it took me, that I know web2py inside-out, a couple of weeks to figure it out partially. But again, nobody cared because it's outside core, and things outside core are never considered.

The idea of submodules is (IMHO) a perfect fit (or Yet Another Way) to keep things linked to the main project and attract more devs, hopefully speeding up development and contributions.
I'd like to see DAL as a submodule and thrive with its own pack of devs faster than what it'll take it under web2py. I envision pydal always being rather stable and "tag" monhtly releases, with web2py pulling in every month a new "tagged" release of pydal.
Then I'd like to see web2py structuring its code to accept hooks with "plugins" that are maintained as separate projects, ala wheezy, but still shipped as submodules (that is not a great deal of pain for a developer)
If the whole thing won't work, we would still have to maintain the exact same amount of code, but hopefully it'll be organized better and the clutter among modules would be less, so touching one thing wouldn't disrupt others as frequently as now.

[Giovanni Barillari]

Il giorno martedì 27 gennaio 2015 14:41:31 UTC+1, Niphlod ha scritto:

two little notes: 

1) keep sys.path clean. 

It's always been [ web2py_root, web2py/gluon, root/site-packages] . I kinda get that web2py/gluon/packages/dal needs to be added (not inserted) because we don't have a packages/__init__.py (should we ?) . Everything else should be reachable without issues and without further altering sys.path...

Actually the deal with pydal is that the whole repo is added under gluon/packages/dal, but actually the real python package is under gluon/packages/dal/pydal, this is why we need to use sys.path. Adding an __init__.py into packages won't change anything.

I don't see any real trouble with the actual solution, but if something more "elegant" comes to the mind we can certainly change this.

 

2) @gi0baro: we settled for testing adapters in pydal and I'm ok with that, but I think there are / there will be things in web2py tied to pyDAL that we need to test in web2py (uhm, like db validators ....). Relying on pydal's developers to "vouch" for web2py compatibility isn't going to cut the deal... I would feel better (I consider myself both a web2py dev and pydal dev)  if web2py-intertwined-pydal tests are left on web2py test-suite, even if part of them is on pydal-specific things. we've been bitten in the past already.

My argumentation is just based on the same principle of the git submodule: avoid code duplication and keep the development process easier.

Running the pyDAL tests under web2py means to slower the testing process and require services that are not really needed to test web2py.

We need to test pydal implementation on web2py, I'm not arguing about this. But testing the 'implementation' under the above principle would means:

1) Run pydal tests with web2py's contrib adapters

2) validators: we just need a test that verify Fields.requires of a created table contains web2py validators

3) rendering: we can create a small dedicated test

4) serialization: see 3

I can't see any reason for considering the above points not enough as tests on pyDAL.

Do you see my point? :)

/Giovanni

[Niphlod]

@richard: My very-concise summary on your points is that web2py is a framework with lots of wonderful ideas labelled as "ready-to-use" but "never-tested-that-much". There won't ever be a single app that can show the whole feature set of web2py, and being a framework rather than an app kinda defeat the case of making an app that really groups together developers. Everything in web2py is kinda-of-baked for an original usecase: what has been engineered correctly is useful to many, what has been engineered poorly is left on the shelf. I welcome the advent of rationalizing big feature-sets in external libs as pydal 'cause it can be the way to attract more (i.e. jump on board with VALID pieces of code even if discarding web2py in its entirety)

@gi0baro: my point on sys.path was originated more from https://github.com/web2py/web2py/pull/762 rather than pydal "sys.path" manipulations. Given that we know the definitive location in web2py, altering sys.path, albeit a solution-per-se, kinda makes me a little bit scared of doing unnecessary sys.path "thingy" (we've been stung before, and still carry around a few shortcomings of populating sys.path unappropriately (see no relative imports in core code, "from gluon import x" and "import x" screwed conventions, circular imports, etc)). tl;dr: the cleaner the better.

on tests: I agree on most of the things.
Testing time isn't really an issue (1min vs 10min on travis-ci are 9min "wasted" well, on my POV): a single user testing its own backend won't feel hurted, and in the end we don't pay travis-ci time. I have a PR ready to speed things up consistently that I'll send to both web2py and pydal repo as soon as I polish things up (probably in a few hours at most)
you're correct on "separating tests".... Testing if a table gets created on the backend doesn't really need both pydal and web2py checking on CI, but as I said there are / there will be things to cope with, that don't/won't entirely rely on pydal side... just don't exclude the possibility right from the start.
Right now (we just "adopted" the new structure) I'd be relieved to know for sure that yet EXISTING precious tests are verifying functionalities (on both sides). Going further we'll work crossing off the "redundant list" the ones we clearly don't need from each-other, and add what we don't have.

[Niphlod]

BTW (while testing the new travis features....).. latest merge killed coveralls integration. I know I could be more angry than others in this regard - because poor me was the one to introduce it on web2py - but ............ GROAN!!!!!

Anyway, PR is on the way.