https://github.com/web2py/web2py/issues/1203

[Niphlod]

IMHO this is the perfect poster for missing tests.

see https://codecov.io/github/web2py/web2py/gluon/validators.py?ref=3808b1f6aed7e6f30cff29214cfbced82a28cab3#l-490 to make yourself accustomed to "web2py is not tested".

the bug report is perfectly valid..... a new and undocumented - and uncommented, for that matter - feature crept in completely breaking validation for list:reference. The offending commit is https://github.com/web2py/web2py/commit/999f235b756b690c62f22a5f29c0e3b05d404d93 .

to sum up, values are converted to a list of ints from a list of string values, i.e. from

['1', '2'] to [1,2]

but the calculated set of IS_IN_DB was since the beginning of times a list of strings

https://github.com/web2py/web2py/blob/master/gluon/validators.py#L594

I'm completely thrown off by that piece of code, as I reaaaally don't see why it got added in the first place.

Now, auto_add undocumented and breaking feature apart, can we all agree that web2py needs more coverage BEFORE pushing any new feature (or trying to fix existing)? 

Pending PRs that stay there months, vulnerabilities that don't get fixed, long-standing missing features discussed over and over never even tackled, experiments that don't get traction, endless time wasted on trying to get a more "we're not immatures" vibe .... no new contributors, no new features... and some backward compatibility broken here and there (not on purpose, but still, I'm afraid of upgrading web2py myself).

I'm done ranting. Frankly, I'm a step away from abandoning web2py, as I can't push it as a serious env to code apps into, no matter the productivity achieved after having read the entire codebase over and over in the last years.

[Leonel Câmara]

I agree. Lack of tests makes it impossible to catch this kind of errors and new features need more discussion. I would hate to see you go. So, I'm going to commit myself to only writing tests for a while, until we're at least in the 70% coverage range. I will try do add at least one test (even if it's a simple one) a week. Starting this week.

[Kiran Subbaraman]

Niphlod,
Please consider staying back. Your contribution to web2py in the form of quality code & interactions are very useful, whenever I have had the need to use web2py.
I hear your call to "Improve the quality of web2py, and start with tests first".

I am not very conversant with testing web2py or writing testcases for it. I will look up the web2py book and existing code to figure out how to contribute.
In terms of  deciding which areas require testcases first, is it worth looking at the code-coverage results, and go after the ones which are below 50% coverage? Am looking at this link: https://codecov.io/github/web2py/web2py?view=all&ref=3808b1f6aed7e6f30cff29214cfbced82a28cab3#sort=coverage&dir=asc

[Massimo Di Pierro]

I second the sentiment. We do not want to lose Simone (niphlod). He is invaluable to this community. In fact what he says and complains about it right. We need to make a better job. As I said I cannot promise to have more time available.

I am open to any suggestion on how to improve things.

I agree we need more tests. i also think we should first make a detailed list of which modules and functions have the least coverage. For me some modules are more important than others. In particular some modules contain code that I want to reused in web3py and others I want to ditch. Specifically there is a lot of code in sqlhtml that I think should die so we should not waste too much on that one. validators are really good and they should be tested better.

I also think we should consider moving the scheduler into pydal.

Perhaps validators too should be moved into pydal.

What do people think?

Massimo

[Leonel Câmara]

While I agree that much of sqlhtml should die (I mostly don't use it). I think it's important to have something like it well tested because sqlhtml tests a bunch of web2py features working together so if sqlhtml works well you can be pretty sure that there's a lot of stuff you haven't broken.

[Kiran Subbaraman]

There are a few things that keep coming up... and also mentioned in this thread. Some of those are:

1. web2py - maintain backward-compatibility, and create quality releases (tests, with a process for enhancements / code changes)
2. refactoring/evolution of web2py - move parts to pydal, create web3py
3. decision making / moderation - Massimo / others? (accept PRs, answer user questions, set directions, etc)

For me, point 1 is the most important one... and that anchors on point 3. This also means that we get as much as coverage as possible for 'web2py', in its current form ... because I see myself using this for quite some time now. This is what I will use for business.
So, the next step is to get coverage in areas which are important first, and then to cover the code base as much as possible.

I hear Massimo suggesting that we get the 'validators' as priority 1. Can we list the rest in order of importance? Again, am referring to the list from here: https://codecov.io/github/web2py/web2py?view=all&ref=3808b1f6aed7e6f30cff29214cfbced82a28cab3#sort=coverage&dir=asc

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

[Michael M]

I would be really bummed to lose you Niphlod.  I have found untold number of answers to my questions from reading posts by you in the past.  I am entry level developer and hope to be a more valuable asset in the future to the web2py team.  I just got hired into a Unix OS team and given the role of automation programming.  So hopefully that will help push me faster into a skillset that will be able to help assist the framework.  But I want to reiterate it would be a great loss if you move on. 

Regards,

Michael Messmer.

[Massimo DiPierro]

Simone. We all do mistakes. I will review that offending commit. It is my responsibility and it was done to fix a bug.

I do not see pending PR stay there for months, unless it is something truly contested.

Anyway, I cannot promise I will have more time, but I if there is something specific that you want me to do please let me know in detail and I will do it.

massimo

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Massimo DiPierro]

Thanking a second look. That commit was a mistake. Was supposed to be committed to a different branch. It is clearly not finished as it includes print statements.

I apologize for the mistake and will revert it until I have time to figure out what is that I was doing.

Massimo

On Mar 8, 2016, at 3:33 PM, Niphlod <nip...@gmail.com> wrote:

[Massimo DiPierro]

I reverted that change of behavior. Not the entire commit. In fact the print statements are no longer there.

I remembered what I was doing and there is a reason for that code (although the int(…) was a bug).

The idea is to allow IS_IN_DB to validate fields that come from autocomplete tag libraries. they can contain a list of references as long as text separated by a delimiter. 

It would allow to add a new tag and automatically add it to the database and create a reference for it.

If no auto_add then it should work as before and that was the mistake.

Massimo

On Mar 8, 2016, at 3:33 PM, Niphlod <nip...@gmail.com> wrote:

[Massimo DiPierro]

Thank you Leonel,

I think we should take one step back. Let’s start by making a list of coverage by module and by function. Then we decide where to concentrate our efforts. There are some modules for which I would like 100% test coverage sooner since I can use to port web2py functionality to web3py (although it does not exist it will). Some modules like sqlhtml are terrible and we should not spend too much time on them. We should replace it by the new Form.py. There are design flaws in sqlhtml are not worth our time. This bug is because of the weird interaction of validators with sqlhtml.

Massimo

On Mar 8, 2016, at 5:51 PM, Leonel Câmara <leonel...@gmail.com> wrote:

I agree. Lack of tests makes it impossible to catch this kind of errors and new features need more discussion. I would hate to see you go. So, I'm going to commit myself to only writing tests for a while, until we're at least in the 70% coverage range. I will try do add at least one test (even if it's a simple one) a week. Starting this week.

[Massimo DiPierro]

It will be replaced by Form.py. So I think we should spend time testing new code rather then code we may deprecated and unfixable.

On Mar 9, 2016, at 6:37 AM, Leonel Câmara <leonel...@gmail.com> wrote:

While I agree that much of sqlhtml should die (I mostly don't use it). I think it's important to have something like it well tested because sqlhtml tests a bunch of web2py features working together so if sqlhtml works well you can be pretty sure that there's a lot of stuff you haven't broken.

[Massimo DiPierro]

For me this is the order of importance:

1) dal

2) validators

3) scheduler

4) rocket

5) templates

6) session logic

7) helpers

8) auth (should be broken into multiple files)

9) sqlform and grid should be replaced by new logic (for form it partially exists) we would not break existing code but we would not fix bugs. we would ask users to move to form.

I also think dal+validators+scheduler should be merged

Rocket and template should become their own separate packages.

Rocket kind of is but is not maintained and we changed it.

The JS that ships with welcome should also become its own package that does not require web2py.

web3py will replace some internal logic and default to new auth (no wiki) and new form logic. It will work with python 2 and 3. web3py will be able to run 99% of web2py code provided one does: from past import * 

I think this is an achievable goal by the summer.

Massimo

[Richard Vézina]

More tests is definitely required and is for sure a flaw of web2py that start to show up in the last years... For sure speaking for all, we can't as a community lost any member and and particuarly someone who is so often rigth as Simone :D

Tests is good, but could requirements help us even more? I mean, the book details most of the feature, but I feel (since I am not involve enough in web2py dev) that sometimes once in wild "undocumented" feature get broken... I believe that this happen because un clear requirements... Could more testing (or at all) with Behave test framework help even more?

Also, unit test is one thing, but functional test with an app that implement all or most of the web2py feature could help also... Each time change is apply we should try (manually at least and preferably programmatically) the app that implement the web2py features??

Richard

[Niphlod]

thanks for the appreciation showed but I kinda shed some tears over the same ideas popping up now. summing up rants (some coinciding with this threads ideas).

- "should we have an app that tests functionalities in addition to unittests?" --> check --> https://groups.google.com/d/msg/web2py-developers/ED33zavy_mE/R_lY2rQ7150J

- "can we have a clear path to where web2py is going?", "should we refactor web2py in small usable bits that could be used somewhere else?" --> web3py --> check --> https://github.com/mdipierro/gluino , https://github.com/mdipierro/web3py, https://github.com/web2py/web3py (dead, dead, probably dead ?!)

- "what are we going to do to address css framework lock-in?" --> formstyle callbacks, extrapolate widgets to put in the app and not in core, forms.py, validate_and_insert, vue.js, html5 widgets, ractive.js ..... all evaluated and dead. practically zero availability for a scaffolding app that is not the one shipped with web2py

- "admin is becoming too complex" --> result https://groups.google.com/d/msg/web2py-developers/MsLVi1gPaFo/1nOrNHFgCQAJ

- signals, callback, a way to hook in with custom and/or third party libraries --> psa, p2a, janrain, recaptcha, social auth, contrib.login_methods nightmares, execution flow, response.custom_commit(), check check check seen attempted dead

- various screams about not being ever able to do a requirements.txt file up to date with useful libraries, aka "reinventing the wheel" --> large and unmaintained codebase

- integrating PRs without tests, resorting in all kinds of misbehaviours and/or braking backward compatibility, and/or even worse introducing a bug that has to stay because "we missed it for a long time"

- websocket, greenifying code, blablabla, got there before most of all with tornado_messaging --> never really integrated --> django channels is truly a better fix

- documentation, docstrings, coverage, bounties for improving code quality --> tim on the book (until recently), 31 issues on the book opened, 9 PR pending. CI up and running (after endless trial and errors due to web2py's unique workflow), tests in docstrings moved to unittests, no significant coverage added except for yours truly. API docs moved from ugly epydoc to readthedocs (another source of pain) and style moved from "whatever" to "google style". +1 for the effort (again, yours truly), no docs ever added.

- pip installable package --> source of endless laughs for most pythonista's colleagues, ungratious attempt freezed web2py on pypi at 4 years ago. several proposal over the years, never took place

- smartgrid refactor --> discussed over and over, features added, complexity added, again dead in the water

- releases followed by a stream of "bugfix" in the next days

- bug triaging --> 160 opened issues migrated from google repo to github. 82 closed, 80 open. meh. 85 issues created in the last 6 months, 58 closed, 27 opened. shiny new pydal repo (big part responsible for "closed issues" on web2py that  have been migrated) 44 open issues. 

- endless recurrences of "we don't like web2pyslices.com" . nobody with a valid alternative, 5 "slices" on the last 6 months.

- roadmap (trello board, agile/kanban/scrum/userbase interaction on actual features .... never took up): 8 todo since creation, 8 wishes, 6 months ago last activity, 14 "done" (again, most of it is spearheading removal of ugly and unmaintained code)

- milestones : only on pydal, and even there not always respected (although a zillion kudos for giobaro that is spearheading practically by himself)

- last but not least: "manpower"... recent contributors on both repos: 10

ranted over and over in the past years. all rants are freely available (some rash words included, sorry). 

heard the stories already, some discussed in fine-grained details, seen them not happening mostly because lack of manpower (or real interest, or real investments).

heard stories of big webapps with web2py under the hood. maybe only sahana eden helped contributing back some features. zillion years ago.

proud of the work done, not so much on the "marketing" side to promote contributions. something is clearly wrong with the disparity of the community. 

spearheading not so fun after a lot. productivity and "pushability" impaired by all of the above. 

features and beautiful implementations popping up here and there, better fit for the job but discarded up until now for "proudness" of web2py. 

kinda tired of porting them to web2py (or even proposing).

I'll dial back from contributions and see what happens.

[Dave S]

On Tuesday, March 8, 2016 at 3:51:39 PM UTC-8, Leonel Câmara wrote:

I agree. Lack of tests makes it impossible to catch this kind of errors and new features need more discussion. I would hate to see you go. So, I'm going to commit myself to only writing tests for a while, until we're at least in the 70% coverage range. I will try do add at least one test (even if it's a simple one) a week. Starting this week.

I'd be interested in helping with tests, but I'm kinda slow in understanding "hands free" testing.  The Mercurial project uses both doctests and scripts that run in a shell that understands expected-versus-actual results (and buildbot slaves to run them).  I gather Web2Py has some tests running in a Continuous Integration environments?

/dps

[Niphlod]

muhahahah, sorry, I couldn't stop myself:

"helping others help with web2py?" --> check --> http://web2py.com/books/default/chapter/29/15/helping-web2py 

[Massimo DiPierro]

On Mar 9, 2016, at 3:11 PM, Niphlod <nip...@gmail.com> wrote:

thanks for the appreciation showed but I kinda shed some tears over the same ideas popping up now. summing up rants (some coinciding with this threads ideas).

- "should we have an app that tests functionalities in addition to unittests?" --> check --> https://groups.google.com/d/msg/web2py-developers/ED33zavy_mE/R_lY2rQ7150J

YES

- "can we have a clear path to where web2py is going?", "should we refactor web2py in small usable bits that could be used somewhere else?" --> web3py --> check --> https://github.com/mdipierro/gluino , https://github.com/mdipierro/web3py, https://github.com/web2py/web3py (dead, dead, probably dead ?!)

all I can say is that we should take it apart so pieces can be reusable.

- "what are we going to do to address css framework lock-in?" --> formstyle callbacks, extrapolate widgets to put in the app and not in core, forms.py, validate_and_insert, vue.js, html5 widgets, ractive.js ..... all evaluated and dead. practically zero availability for a scaffolding app that is not the one shipped with web2py

open for discussion

- "admin is becoming too complex" --> result https://groups.google.com/d/msg/web2py-developers/MsLVi1gPaFo/1nOrNHFgCQAJ

agreed. admin should be simplified and all based on ractive.js I can do that. I am kind of already doing it.

- signals, callback, a way to hook in with custom and/or third party libraries --> psa, p2a, janrain, recaptcha, social auth, contrib.login_methods nightmares, execution flow, response.custom_commit(), check check check seen attempted dead

what do you recommend?

- various screams about not being ever able to do a requirements.txt file up to date with useful libraries, aka "reinventing the wheel" --> large and unmaintained codebase

- integrating PRs without tests, resorting in all kinds of misbehaviours and/or braking backward compatibility, and/or even worse introducing a bug that has to stay because "we missed it for a long time”

I do not think we should slow down development because some PR do not have tests. There are some PR that have been waiting for months because you asked for tests, I agreed, and tests were never submitted. This is not acceptable. Some times it is better to accept the PR and see if something breaks. 

- websocket, greenifying code, blablabla, got there before most of all with tornado_messaging --> never really integrated --> django channels is truly a better fix

how do you suggest we do this.

- documentation, docstrings, coverage, bounties for improving code quality --> tim on the book (until recently), 31 issues on the book opened, 9 PR pending. CI up and running (after endless trial and errors due to web2py's unique workflow), tests in docstrings moved to unittests, no significant coverage added except for yours truly. API docs moved from ugly epydoc to readthedocs (another source of pain) and style moved from "whatever" to "google style". +1 for the effort (again, yours truly), no docs ever added.

I will crunch them shortly. My apologies to Tim. I did not notice because I did not get notifications.

- pip installable package --> source of endless laughs for most pythonista's colleagues, ungratious attempt freezed web2py on pypi at 4 years ago. several proposal over the years, never took place

+1. If somebody helps with setup.py I can integrate this in the build process.

- smartgrid refactor --> discussed over and over, features added, complexity added, again dead in the water

should be moved to client side.

- releases followed by a stream of "bugfix" in the next days

The only way to avoid this for me is to release more often. We do not have enough testers for a different approach.

- bug triaging --> 160 opened issues migrated from google repo to github. 82 closed, 80 open. meh. 85 issues created in the last 6 months, 58 closed, 27 opened. shiny new pydal repo (big part responsible for "closed issues" on web2py that  have been migrated) 44 open issues. 

- endless recurrences of "we don't like web2pyslices.com" . nobody with a valid alternative, 5 "slices" on the last 6 months.

That is not our concern. That is a community maintained project. we are responsible for development.

- roadmap (trello board, agile/kanban/scrum/userbase interaction on actual features .... never took up): 8 todo since creation, 8 wishes, 6 months ago last activity, 14 "done" (again, most of it is spearheading removal of ugly and unmaintained code)

Because people do not do what they are told to do. people work on what they need. I think this is ok and unavoidable. I do it too.

- milestones : only on pydal, and even there not always respected (although a zillion kudos for giobaro that is spearheading practically by himself)

kudos to Gi0baro.

- last but not least: "manpower"... recent contributors on both repos: 10

ranted over and over in the past years. all rants are freely available (some rash words included, sorry). 

heard the stories already, some discussed in fine-grained details, seen them not happening mostly because lack of manpower (or real interest, or real investments).

heard stories of big webapps with web2py under the hood. maybe only sahana eden helped contributing back some features. zillion years ago.

proud of the work done, not so much on the "marketing" side to promote contributions. something is clearly wrong with the disparity of the community. 

spearheading not so fun after a lot. productivity and "pushability" impaired by all of the above. 

features and beautiful implementations popping up here and there, better fit for the job but discarded up until now for "proudness" of web2py. 

Are you referring to JWT? I disagree on your conclusions. No work has ever been rejected or discarded form you. If there is a complete PR that does not break anything and add functionality, it has always been included in web2py. If something has not been included it is because of the extra work required and no obvious push from the community. I - like others - work on things mostly because I need them.

kinda tired of porting them to web2py (or even proposing).

I'll dial back from contributions and see what happens.

I think the approach of taking out and taking better care of what consider the good piece of web2py is a good thing and a good place to start. 

[Dave S]

On Wednesday, March 9, 2016 at 1:20:36 PM UTC-8, Niphlod wrote:

muhahahah, sorry, I couldn't stop myself:

"helping others help with web2py?" --> check --> http://web2py.com/books/default/chapter/29/15/helping-web2py 

A fair shot, and I've read that passage at least once (long ago), but I did say "kinda slow".    :-}

/dps

[Niphlod]

I've ranted my fair share of proposals. And pushed and solved and yelled some. and then others.

This IS NOT niphlod vs mdipierro, not intended to be and never going to be. 

niphlod builds constantly small shrines here and there praying to mdipierro. 

niphlod learned python mostly because of mdipierro. 

niphlod also faces the issue of not seeing mdipierro around so much and web2py falling behind and behind, and community not picking up the slack.

hoping that the "point on point debate" is not for niphlod eyes but for every other developers, I'm quite confident (and happy) that no of my PRs have ever been rejected.

most of the "point on point" you probably missed, because they've been discussed already (some of them on multiple reincarnations).

I'll reply to the most obvious ATM.

I do not think we should slow down development because some PR do not have tests. There are some PR that have been waiting for months because you asked for tests, I agreed, and tests were never submitted. This is not acceptable. Some times it is better to accept the PR and see if something breaks. 

development IS SLOW (or became so). My POV is that I can't contribute/refactor/integrate/reiterate with the current level of coverage. I can add new features or blindly accept PRs, but with no tests and no committments on developer's side, we ended up incorporating LARGE parts of "personal projects" that become crystalized because no sane mind can even think to poke around them. And with no tests, and no testers, who sees "if something breaks" ?

 

- websocket, greenifying code, blablabla, got there before most of all with tornado_messaging --> never really integrated --> django channels is truly a better fix 

how do you suggest we do this.

don't even know. I'm bitter about this because of being the perfect poster of "ideas we had but never grasped minds". 

- documentation, docstrings, coverage, bounties for improving code quality --> tim on the book (until recently), 31 issues on the book opened, 9 PR pending. CI up and running (after endless trial and errors due to web2py's unique workflow), tests in docstrings moved to unittests, no significant coverage added except for yours truly. API docs moved from ugly epydoc to readthedocs (another source of pain) and style moved from "whatever" to "google style". +1 for the effort (again, yours truly), no docs ever added.

I will crunch them shortly. My apologies to Tim. I did not notice because I did not get notifications.

- pip installable package --> source of endless laughs for most pythonista's colleagues, ungratious attempt freezed web2py on pypi at 4 years ago. several proposal over the years, never took place

+1. If somebody helps with setup.py I can integrate this in the build process.

- smartgrid refactor --> discussed over and over, features added, complexity added, again dead in the water

should be moved to client side.

Hoping this won't be a tackle to "our own" datatables, slickgrid, jqgrid...because they're clearly smarter than us at javascript, it's really not a solution. Moving client-side is pushing a new (and probably less "argument-filled") feature, with probably a lot less # of possible contributors especially on the web2py community. 

 

- releases followed by a stream of "bugfix" in the next days

The only way to avoid this for me is to release more often. We do not have enough testers for a different approach.

- bug triaging --> 160 opened issues migrated from google repo to github. 82 closed, 80 open. meh. 85 issues created in the last 6 months, 58 closed, 27 opened. shiny new pydal repo (big part responsible for "closed issues" on web2py that  have been migrated) 44 open issues. 

- endless recurrences of "we don't like web2pyslices.com" . nobody with a valid alternative, 5 "slices" on the last 6 months.

That is not our concern. That is a community maintained project. we are responsible for development.

- roadmap (trello board, agile/kanban/scrum/userbase interaction on actual features .... never took up): 8 todo since creation, 8 wishes, 6 months ago last activity, 14 "done" (again, most of it is spearheading removal of ugly and unmaintained code)

Because people do not do what they are told to do. people work on what they need. I think this is ok and unavoidable. I do it too.

debatable. a roadmap is there because there is no roadmap anywhere on web2py.com . that was up to show userbase what we're working on and track progress among ourselves, and/or proposing new features. everyone code for themselves, and hopefully the 10 contributors are pushing back what they coded themselves. that doesn't mean a complete drop in new features, nor avoid polls on web2py-users, just having a handle on "todo/done/nevergoingtohappen". we can use github issues and milestones too, but they need to be managed either way.

[Dave S]

On Wednesday, March 9, 2016 at 1:50:16 PM UTC-8, Dave S wrote:

On Wednesday, March 9, 2016 at 1:20:36 PM UTC-8, Niphlod wrote:

muhahahah, sorry, I couldn't stop myself:

"helping others help with web2py?" --> check --> http://web2py.com/books/default/chapter/29/15/helping-web2py 

 

A fair shot, and I've read that passage at least once (long ago), but I did say "kinda slow".    :-}

And now I've read the slice article, as well.

Coveralls is down-ish?  ("Working with AWS engineers to resolve a database issue")

[Niphlod]

coveralls.io has been ditched in favour of codecov.io, because it can aggregate coverage coming from TravisCI (Unix-based enviroment) and AppVeyor (Windows-based environment).

It's on the README badge for quite some time now, in addition to have been quoted in this very thread, but the coverage reports are the same. "green" lines are covered, "red" not.

[Dave S]

On Wednesday, March 9, 2016 at 2:47:40 PM UTC-8, Niphlod wrote:

coveralls.io has been ditched in favour of codecov.io, because it can aggregate coverage coming from TravisCI (Unix-based enviroment) and AppVeyor (Windows-based environment).

It's on the README badge for quite some time now, in addition to have been quoted in this very thread, but the coverage reports are the same. "green" lines are covered, "red" not.

Got it.

/dps

 

[Massimo DiPierro]

I will say this again. If there is something you me to do just ask. If there is something you want to do (like take more responsibility) also ask.

Can we have a chat session some time next week and agree on a roadmap?

Meanwhile let’s think what should be on that roadmap. I do not think the goal should be test everything in web2py. The goal should how to build something better. How to move users over. How to capitalize on what we have built. To me the way to start the discussion is not what is broken in web2py but where does web2py falls short in terms of features? What tool do I (as a developed) would like to have?

I can start by telling what I want.

- I want a solid DAL + validators + scheduler as a self standing package

- I want rocket as a self standing package

- I want web2py templates as a self standing package

- I want a form system where I can model widgets in HTML, not python

- I want web2py.js replaced by a JS toolkit that is server side framework agnostic

- I want admin (rebuilt with ractive) but with a better file browser (as a reusable widget)

- I want a new grid that works with ajax and doubles as appadmin

- I want a way to build apps in a more modular way.

- I want a more modular tools with Auth and Mail and JWT but Auth relying on third party auth libraries.

I have been putting lots of work into:

https://github.com/web2py/web3py

http://experts4solutions.com/w3

http://mdipierro.github.io/stupid.css/themes/random.html

and other JS libs that I am not sharing quite yet.

I would like some help in making web2py tickets work with https://github.com/web2py/web3py and making sure it works with python3 and latest DAL.

To me the future is in the confluence of those tools. 

Massimo

[Michele Comitini]

Too much form my head, I just take two from the pack at this time and merge them.

- I want a form system where I can model widgets in HTML, not python

- I want web2py.js replaced by a JS toolkit that is server side framework agnostic

That is the goal of projects like polymer, not easy... I have been playing with ractive.js components, not easy but smarter and they come close to what I usually need.   

The tradeoff is having something maintained out of web2py's scope but full fledged, vs something lean but in need of a maintainer.

That could bring a lot of JS + HTML code in and shifting from a python centric framework promise, which is what web2py is compared to others that have for instance a specialized templating language.  

Solution would be to move to adopt an event API that decouples completely this frontend framework from the backend framework.

BTW Semantic-UI has this kind of API but it is not easy to extend and uses jquery, which in my opinion is rarely needed anymore since the window.fetch inception and reactive frameworks appearance.

 

[Massimo DiPierro]

Not planning to compete with polymer but like we already have web2py.js I think it is ok to require some JS library when handling forms and grid, as long as they be somewhat customized. Those js libraries should be framework neutral, so that we get broader adoption, and should work out of the box with web2py, without need for JS programming. I like handssontable but it turning into a commercial product. 

I want a system that replaces sqlhtml where the html is described by example, not in python, the html can be generated either serverside or clientside. and we have some client side widget that are applied by the js. We do this already for integer, double, date, datetime, time, list:string, autocomplete. Our widgets are css agnostic but can be improved. We could have a more complete set and ship it as a its own widgets.js thus decoupling this problem from sqlhtml, the source of our nightmares.

Imagine the grid being a combination of this http://mistic100.github.io/jQuery-QueryBuilder/ and https://handsontable.com/ and working out of the box with appadmin. Anyway, this is just an example, I am still looking for options and perhaps you can help look for options.

Massimo

[Massimo DiPierro]

For example, I would like to replace web2py.js with this: http://mdipierro.github.io/stupid.css/widgets/index.html.  Notice it is 6Kb including the source of the calendar.

Need to add: autocomplete, tags, a grid, markmin.js, and form trapping from web2py.js.

On Mar 10, 2016, at 4:08 AM, Michele Comitini <michele....@gmail.com> wrote:

[Richard Vézina]

I see differents direction in Simone wants of improvement and Massimo's... Massimo seems to seek for a way to get beyond web2py (that a good thing to me)...

Both are rigth in their expectation... I mean, it's not because unfruitful Massimo's experimentations with web2py refactoring that he should stop trying...

For sure more constraints should be put in place to make sure new PRs doesn't break web2py in difficult to solve issues... Should we stop accepting PR with no proper test case before their author provide them. Stop merge PRs over piece of code that are not corvered by test and apply them once proper testing is in place - actually before trying to refactor web2py in substancial manner, I had in the past add proper tests to make sure I wasn't causing regression, I consider this a best practice...

Considering the lack of ressources, maybe we should identify milestone improvement and set task force to resolve these identified issues and prioritise them??

I think Massimo's brainstorm about where priorities should be allowed is a good start (also Michele's proposal).

Richard

[Leonel Câmara]

Frankly I would prefer to move some of the functionality to their own applications and plugins. I don't think web2py should include a grid at all for example. There could be multiple grid plugins people would use. There can be an official one which we support, but we can stop supporting it anytime we want (since we maintain backwards compatibility).

[Michele Comitini]

Yes things like the grid should go in plugins or example apps.

Back to forms the framework I would like should talk json back and forth in a reactive fashion.  The client page is tied to a model, that model needs a single simple way to be synchronized between client and server, when synchronization happens is up to the programmer, but there should be a simple way to transfer data without wasting bandwidth and CPU cycles...

Moving away from the classical multipart or formencoded stuff would be healthy, except  for blobs/files.  It would simplify coupling with client frameworks.  The idea would be to have some js "adapters" for working with different third party frameworks. Yet it would also be nice to have a set of ready made widgets or a preferred client framework to allow to fast prototyping.

2016-03-10 20:32 GMT+01:00 Leonel Câmara <leonel...@gmail.com>:

Frankly I would prefer to move some of the functionality to their own applications and plugins. I don't think web2py should include a grid at all for example. There could be multiple grid plugins people would use. There can be an official one which we support, but we can stop supporting it anytime we want (since we maintain backwards compatibility).

--

[villas]

I never imagined that my posting issue 1203 would lead to this discussion,  but many good points are being made.

I think there are two separate matters being confused.  Here is my take on it...

1. Massimo's view:  Let's be ambitious and make web2py evolve quickly into what we would like it to be.  

2. Simone's view:  I fully support Massimo,  but hey guys,  stuff is getting broken,  things aren't being tested and fixed.  If we really want backwards compatibility and quality code,  we need to plan, allocate resources, have good tests and finish what we start.

So can we really have the best of both worlds?  I think only with a slightly bigger team.  IMO the solution is this:  more delegation.  

Thanks to Massimo,  Simone and all other contributors!

[Richard Vézina]

Good resume villas... That what I want to standout but with lack of clarity...

--

[Leonel Câmara]

I think web2py has just reached a point in its development where some refactoring is in order. Tests are always important, but when you need to refactor something that has a lot of moving parts like web2py they become indispensable so people can make changes with a modicum of confidence that they won't break things because of some unknown interaction. This sort of has to do with Python as well, a less dynamic language would warn you of a lot of the breakage that we need tests to detect.  

[Paolo Valleri]

+1 for moving scheduler in pydal or in a separated project. is there something blocking it?

+1 for moving validators in pydal.

+1 for making example snippets using modern client side framework. For example, is web2py ready for receiving a post from a client side SQLFORM? If yes, lets publish an example either in the book or in the github wiki or better pack an app in web2py/applications.

 Paolo

2016-03-11 3:38 GMT+01:00 Leonel Câmara <leonel...@gmail.com>:

I think web2py has just reached a point in its development where some refactoring is in order. Tests are always important, but when you need to refactor something that has a lot of moving parts like web2py they become indispensable so people can make changes with a modicum of confidence that they won't break things because of some unknown interaction. This sort of has to do with Python as well, a less dynamic language would warn you of a lot of the breakage that we need tests to detect.  

[Giovanni Barillari]

I can't see any good reason to move both scheduler and validators inside pydal.

And I can't understand how moving code between packages solves the problems stated in this topic. 

The principle of creating separated packages is to use those packages in the first instance, but web2py doesn't since we don't have a pip package. And, additionally, when you make a package you have to invest more about it in terms of documentation, testing and marketing, otherwise people won't use it at all and it makes all the operation useless.

pyDAL is an example: we have just descriptive documentation splitted and repeated between web2py and weppy websites but we don't have any interface documentation. We haven't increased the adoption or the number of contributors in a valuable number. And frankly I don't think this is due to the fact pyDAL hasn't included validators or the scheduler.

We just have a lack of organisation and communication.

/Giovanni

[Massimo DiPierro]

The reason I am proposing it is that the scheduler is a single file and it only needs pydal and other frameworks (weppy?) may want to use it.  Same thing for validators.

It does not solve our immediate problem but if we increase the user based of pydal, it may.

Massimo

[Giovanni Barillari]

Still, I can't see why including validators in pyDAL would be a good move.

The idea of having pyDAL as a separate package is, above all the others, that I can use it without any web interface. If I don't have a web interface I don't need, practically, validators too (at least in the 99% of cases). If I use pyDAL from console behind my web application then I would use the web2py's shell or the weppy's one, and this is another story.

Moreover, the code behind web2py validators has a lot of dependencies and logic that are parts of web2py, not applicable in pydal. And this is why, for example, in weppy validators were totally rewritten, and today have very few code and logic in common with the web2py's ones.

Making web2py's validators parts of pyDAL makes a clear statement: this is how validators have to be done. But frankly, this is just how web2py does the validation over pyDAL. And, additionally, this also means including in pyDAL untested and not python3 compatible code, so it will degrade the code quality over pyDAL. You may refactor web2py's validator and make them python3 compatible, but the first thing I've said is still valid.

Personally I would like to see an interface documentation on how people should write validators in order to use them over pyDAL. That would make sense to me. Not making pyDAL a "cauldron" of mixed stuffs that are not necessarily related to database operations. pyDAL is a "Database abstraction layer", not a small framework to "do things on databases from a web application". If we want to do that, then we're talking about a new project, and is a totally different story.

/Giovanni

[Anthony]

- websocket, greenifying code, blablabla, got there before most of all with tornado_messaging --> never really integrated --> django channels is truly a better fix 

how do you suggest we do this.

don't even know. I'm bitter about this because of being the perfect poster of "ideas we had but never grasped minds". 

Should we attempt to replicate something like Django channels? Seems like a lot of work and adds a fair bit of complexity. Maybe one option would be to create a simple integration with Pushpin (for example, see http://blog.fanout.io/2014/11/06/building-a-realtime-api-in-django/). We could even take it a step further and add an integration with RethinkDB changefeeds (see https://www.rethinkdb.com/blog/building-a-realtime-api/), giving us something similar to Meteor's livequery functionality.

Anthony

[Massimo DiPierro]

How about scheduler?

[Anthony]

On Saturday, March 12, 2016 at 2:07:49 PM UTC-5, Massimo Di Pierro wrote:

How about scheduler?

To me, the scheduler is even less relevant to pyDAL than validators (at least validators are related to data models) -- the scheduler is just a tool that happens to have a dependency on pyDAL. What's the advantage of packaging the scheduler with pyDAL as opposed to making it a separate package?

Anthony

[villas]

+1 Anthony

I had the impression that the Scheduler was taken out of contrib simply so it could be on the batteries-included list for web2py.  In any case it was designed to be 'self-standing' and that was consistent with web2py decoupling modules rather than integrating stuff (which I thought was the plan).

I think the Scheduler and pyDAL are good examples of sub-modules that were being looked after by contributors (Simone and Giovanni).  Surely that's a model which should be encouraged for other parts of web2py.  Massimo could be an even greater conductor if the orchestra had a few more good musicians looking after the instruments.

[Anthony]

On Thursday, March 10, 2016 at 2:32:27 PM UTC-5, Leonel Câmara wrote:

Frankly I would prefer to move some of the functionality to their own applications and plugins. I don't think web2py should include a grid at all for example. There could be multiple grid plugins people would use. There can be an official one which we support, but we can stop supporting it anytime we want (since we maintain backwards compatibility).

I think the grid can be improved, but I wouldn't necessarily relegate it to a plugin (that we can stop supporting at any time). It's a very useful piece of functionality, and the kind of thing that distinguishes web2py from other frameworks (i.e., lots of functionality out of the box).

Anthony

[Massimo DiPierro]

Can we all have an IRC chat some time this week? For me 8am morning Pacific time is a good time almost any day

[Anthony]

On Monday, March 14, 2016 at 1:54:42 PM UTC-4, Massimo Di Pierro wrote:

Can we all have an IRC chat some time this week? For me 8am morning Pacific time is a good time almost any day

That works for me most days as well.

Anthony

[Leonel Câmara]

That hour is fine by me. I think I don't have any meetings this week so you're free to pick.

[Massimo DiPierro]

How about Thursday morning at 8am PST? Simone?

On Mar 14, 2016, at 1:19 PM, Leonel Câmara <leonel...@gmail.com> wrote:

That hour is fine by me. I think I don't have any meetings this week so you're free to pick.

[Richard Vézina]

Hello Massimo,

In the related post you mention rocket in order of importance... 

I found that there is no test for rocket.py

Since it based on a external library I am not sure if we want to cover it by tests or not... And if so, maybe we should reuse the original project tests : http://bazaar.launchpad.net/~tdfarrell/rocket/trunk/files/head:/tests/

??

Richard

On Wed, Mar 9, 2016 at 10:17 AM, Massimo DiPierro <massimo....@gmail.com> wrote:

For me this is the order of importance:

1) dal

2) validators

3) scheduler

4) rocket

5) templates

6) session logic

7) helpers

8) auth (should be broken into multiple files)

9) sqlform and grid should be replaced by new logic (for form it partially exists) we would not break existing code but we would not fix bugs. we would ask users to move to form.

I also think dal+validators+scheduler should be merged

Rocket and template should become their own separate packages.

Rocket kind of is but is not maintained and we changed it.

The JS that ships with welcome should also become its own package that does not require web2py.

web3py will replace some internal logic and default to new auth (no wiki) and new form logic. It will work with python 2 and 3. web3py will be able to run 99% of web2py code provided one does: from past import * 

I think this is an achievable goal by the summer.

Massimo

On Mar 9, 2016, at 9:09 AM, Kiran Subbaraman <subbaram...@gmail.com> wrote:

There are a few things that keep coming up... and also mentioned in this thread. Some of those are:

1. web2py - maintain backward-compatibility, and create quality releases (tests, with a process for enhancements / code changes)
2. refactoring/evolution of web2py - move parts to pydal, create web3py
3. decision making / moderation - Massimo / others? (accept PRs, answer user questions, set directions, etc)

For me, point 1 is the most important one... and that anchors on point 3. This also means that we get as much as coverage as possible for 'web2py', in its current form ... because I see myself using this for quite some time now. This is what I will use for business.
So, the next step is to get coverage in areas which are important first, and then to cover the code base as much as possible.

I hear Massimo suggesting that we get the 'validators' as priority 1. Can we list the rest in order of importance? Again, am referring to the list from here: https://codecov.io/github/web2py/web2py?view=all&ref=3808b1f6aed7e6f30cff29214cfbced82a28cab3#sort=coverage&dir=asc

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Massimo DiPierro]

rocket has a perverse story. I rewrote cherrypy.wsgiserver. Timothy adopted the project and rewrite mine with lots of improvement. He stopped maintaining it. We forked and added features such as logic to pass the pathoc tests. At some time it passed them all but I have not run them in 2 years. I think we should continue to use pathoc (http://pathod.net/docs/pathoc) for tests but integrate the call in gluon/tests so they run automatically. No need to make our own tests.

Massimo

[Richard Vézina]

Here little improvements over helper tests that should be merge quickly to not impact possible other contributors commits : https://github.com/web2py/web2py/pull/1215

Sorry for silly PR with PEP8 improvements...

Richard

[Richard Vézina]

About pathoc, I am not sure about how to do that... The doc is limited...

Richard

On Mon, Mar 14, 2016 at 2:49 PM, Massimo DiPierro <massimo....@gmail.com> wrote:

[Massimo DiPierro]

Try this first:

http://corte.si/posts/code/pathod/pythonservers/index.html

we should pass those tests. Than we can automate with these:

http://pathod.net/docs/test

[Niphlod]

How about Thursday morning at 8am PST? Simone?

almost 24 hour from "now"...too late to the party or my pst--gmt+1 conversion sucked ?

 

[Massimo DiPierro]

Thursday, not Tuesday. It is in 3 days.

[Niphlod]

lol, right, will be there.

[Niphlod]

correction. is pst or pdt ? according to worldtimeanddate it's pdt in california ;-(

if it's

http://www.timeanddate.com/worldclock/fixedtime.html?msg=web2py-developers+meeting&iso=20160317T08&p1=3922&am=30

I'll try my best

if instead it's

http://www.timeanddate.com/worldclock/fixedtime.html?msg=web2py-developers+meeting&iso=20160317T08&p1=224&am=30

I won't be there

[Michele Comitini]

Please send an invitation through gcalendar or any ical.  That would take care of clocks too. ;-)

[Massimo DiPierro]

done

[Massimo DiPierro]

can you do 8:30am PDT? If not we will do 9am PDT.

[Niphlod]

9 is better, everything between 8 and 9 will translate to myself organizing at work.

[Massimo DiPierro]

ok changed to 9am PDT

[Leonel Câmara]

Where is the chat going to be, #web2py in freenode? Google hangouts?

[Massimo DiPierro]

I originally thought of hangout but looks like there may be more people than anticipated. Do let’s use #web2py freenode.

On Mar 15, 2016, at 7:48 AM, Leonel Câmara <leonel...@gmail.com> wrote:

Where is the chat going to be, #web2py in freenode? Google hangouts?

[Richard Vézina]

Hello Massimo,

Do you have any base pathoc code implementation ?

I am confused on how to use pathoc inside unittest case... It needs a rocket instance to be up and running (not sure if rocket can run stand alone without web2py) been able to launch web2py instance from python with subprocess... Also, there is example of pathod unittest, but to my understanding we want to simulate client request with pathoc and the only documentation relate to pathoc command lines call.

Thanks

Richard

On Mon, Mar 14, 2016 at 2:49 PM, Massimo DiPierro <massimo....@gmail.com> wrote:

[Massimo DiPierro]

yes. it is possible. we do it already. Look into the gluon/tests/test_web.py tests

[Massimo DiPierro]

looks like I am in a minority here. So we will leave the scheduler where it is for now.

Massimo