bootstrap3

[Massimo DiPierro]

I anybody doing any bs3 work or any dal work that I should be waiting for before pushing 2.10.1?

[Niphlod]

did anyone test extensively the new scaffolding we're so in a rush to push ?

[Massimo DiPierro]

We need more testing. we are not pushing without more testing.

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paolo Valleri]

I'd suggest running pydal tests on web2py for at least the contrib drivers (pg8000/pymysql)

Today I discovered that mongo adapter doesn't support common_filter for methods such as update and delete, tomorrow I'll post a patch along with few tests

 Paolo

[Niphlod]

I know, I'm being stressfull...........Am I the only fan of dropping 3-level menus ? mobile-wise the experience is terrible.

notes (I visited only default/index.html and the login page, how can everybody skip over these ??? ):
- footer is sticky, yep, but not fixed size. the less tall the page, the taller the footer.
- navbar is in a container, all the rest in a container-fluid
- response.flash is way too wide
- web2py.js should probably add a btn-default to buttons
- form.add_button() behaves wrongly (see the "remember me" on the login page)
- auth.settings.formstyle doesn't inherit response.formstyle. Either we set it explicitely in db.py or we change auth logic to honor response.formstyle
- menu links point to web2py.css, that is still there but unused.
- web2py-bootstrap3.js comes from a side-project. apart from documentation pointing to the side-project, can be dropped safely once 3rd level menus are dropped. Even if we kept those, needs revisitations
- docs are pointing to side-project also for web2py-bootstrap3.css. Several styles are unused and should be dropped. Some are plain wrong (form max-width: 600px ... .hidden replacing bs3 behaviour .... ?)

auth versioning is still enabled....... I'd disable it. Did anybody use janrain lately ? there's a bug opened on it: should we really keep it on the default scaffolding ?


PS: is the time to drop the examples app ? ATM I'm reaaaally eager to drop it: too much things to be revised and frankly the only link pointing to it is embedded in an (old)copy of web2py.com and reachable by a not-so-visible link....

PS2: the scaffolding app carries the hypermedia module : apart from it being largely untested (and tearing the app down with large tables by default), it requires python 2.7. Nice POC, but maybe scaffolding isn't the right place to advertise.

[Niphlod]

https://github.com/web2py/web2py/blob/master/applications/welcome/views/layout.html#L30

the script tag isn't closed, so web2py_ajax injected variables are unavailable.

[Massimo DiPierro]

On Mar 17, 2015, at 3:52 PM, Niphlod <nip...@gmail.com> wrote:

I know, I'm being stressfull...........Am I the only fan of dropping 3-level menus ? mobile-wise the experience is terrible.

notes (I visited only default/index.html and the login page, how can everybody skip over these ??? ):
- footer is sticky, yep, but not fixed size. the less tall the page, the taller the footer.

That was by design. I like it that way. I am ok to change it but how?

- navbar is in a container, all the rest in a container-fluid

fixed

- response.toolbar doesn't behave as usual

- response.flash is way too wide

fixed

- web2py.js should probably add a btn-default to buttons

it does it

- form.add_button() behaves wrongly (see the "remember me" on the login page)

not sure how. can you help?

- auth.settings.formstyle doesn't inherit response.formstyle. Either we set it explicitely in db.py or we change auth logic to honor response.formstyle

It does here: https://github.com/web2py/web2py/blob/master/gluon/tools.py#L1361 is it not working?

- menu links point to web2py.css, that is still there but unused.

fixed

- web2py-bootstrap3.js comes from a side-project. apart from documentation pointing to the side-project, can be dropped safely once 3rd level menus are dropped. Even if we kept those, needs revisitations

I agree they should not be used but why remove them since we went thought the table of implementing them?

- docs are pointing to side-project also for web2py-bootstrap3.css. Several styles are unused and should be dropped. Some are plain wrong (form max-width: 600px ... .hidden replacing bs3 behaviour .... ?)

is there a way to remove them automatically?

auth versioning is still enabled....... I'd disable it. Did anybody use janrain lately ? there's a bug opened on it: should we really keep it on the default scaffolding ?

PS: is the time to drop the examples app ? ATM I'm reaaaally eager to drop it: too much things to be revised and frankly the only link pointing to it is embedded in an (old)copy of web2py.com and reachable by a not-so-visible link….

I agree but not in this version. That requires changing the documentation as well as my deployment process.

PS2: the scaffolding app carries the hypermedia module : apart from it being largely untested (and tearing the app down with large tables by default), it requires python 2.7. Nice POC, but maybe scaffolding isn't the right place to advertise.

What do others think about this? I do not have strong opinions but it is something I want to encourage people to try and use.

[Niphlod]

On Tuesday, March 17, 2015 at 11:00:20 PM UTC+1, Massimo Di Pierro wrote:

That was by design. I like it that way. I am ok to change it but how?

repetita juvant....Do you have ANY application that has a taddle bit of animations that change the container height ? If yes, then you can't like it.  In order to replicate the behaviour .... put in a controller

def test():
     return dict()

navigate to it, press any response.toolbar button. Ugly.

This is how, BTW, http://getbootstrap.com/examples/sticky-footer-navbar/sticky-footer-navbar.css

- web2py.js should probably add a btn-default to buttons

it does it

nope. https://github.com/web2py/web2py/blob/master/applications/welcome/static/js/web2py.js#L79

- form.add_button() behaves wrongly (see the "remember me" on the login page)

not sure how. can you help?

IMHO a revision of extra_fields for SQLFORM is needed, in addition to add_button(). the same thing happens with the "check to delete" buttons. the label (including separator) is after the checkbox, and doesn't seem to be styled as dictated by the formstyle.

- auth.settings.formstyle doesn't inherit response.formstyle. Either we set it explicitely in db.py or we change auth logic to honor response.formstyle

It does here: https://github.com/web2py/web2py/blob/master/gluon/tools.py#L1361 is it not working?

whoops. It's working. Need to raise another point then: there is no management of error-related classes in core-shipped bs3-related formstyles.

I agree they should not be used but why remove them since we went thought the table of implementing them?

try it on a narrow screen, you'll see that a vertical scrollbar is added. Try to navigate it on a touch-screen and you'll quickly realize it's impossible. I'm not very keen of "add if someone went through the trouble of implementing". We dropped far more "sweaty implementations": if they don't work, they don't. Moreover, it's not our sweat, it's copied off bootsnipp. In addition to that, there's a reason why it's a snippet and it's not included in bs3. Mobile navigation.
There are other subtle small bits on web2py-bootstrap3.js that are plain wrong (or, to better explain, they were useful in the original process but got left unused in the current state, which is is a copy-paste of portions of the original project).

- docs are pointing to side-project also for web2py-bootstrap3.css. Several styles are unused and should be dropped. Some are plain wrong (form max-width: 600px ... .hidden replacing bs3 behaviour .... ?)

is there a way to remove them automatically?

There's a distinction to make....unused, added for no reason and overriding defaults.
unused could, theoretically. Code a page that includes all kinds of possible web2py-supplied pieces, then pass it through uncss or some chrome plugins. there are tons.
Added for no reason........well. There's no pre or post processor smarter than a brain.
overriding defaults is a bit of a conceptual nightmare.

<totally_personal_opinion coming_from="a DBA" not_so_skilled_in="web design">
If you embrace a css framework, you shouldn't override its rules.
To the previous statement, I append a part... "unless you are at least as badass as bootstrap developers", that in my case is obviously true.
I was discomforted from the previous set of overrides (still in web2py.css) that screwed my design in some occasions for no apparent reasons. If someone can comment/justify/show test cases I'd be more inclined to accept them, but as they stand, e.g. the max-width fixed limitation on EVERY form, no matter of media-queries, well, I really don't see the reason behind.
</totally_personal_opinion>

auth versioning is still enabled....... I'd disable it. Did anybody use janrain lately ? there's a bug opened on it: should we really keep it on the default scaffolding ?


PS: is the time to drop the examples app ? ATM I'm reaaaally eager to drop it: too much things to be revised and frankly the only link pointing to it is embedded in an (old)copy of web2py.com and reachable by a not-so-visible link….

I agree but not in this version. That requires changing the documentation as well as my deployment process.

Bummer. At least keep examples in sync with what web2py.com is now then.

PS2: the scaffolding app carries the hypermedia module : apart from it being largely untested (and tearing the app down with large tables by default), it requires python 2.7. Nice POC, but maybe scaffolding isn't the right place to advertise.

What do others think about this? I do not have strong opinions but it is something I want to encourage people to try and use.

well. from "we shouldn't drop 2.5" to "we force newbies to 2.7" in less than a week, you're a welcome addition to 2.7 supporters :-P .
I just drop it as soon as I see it because it'll try to serialize 10k rows without even blinking.

[Massimo DiPierro]

On Mar 17, 2015, at 5:46 PM, Niphlod <nip...@gmail.com> wrote:

On Tuesday, March 17, 2015 at 11:00:20 PM UTC+1, Massimo Di Pierro wrote:

That was by design. I like it that way. I am ok to change it but how?

repetita juvant....Do you have ANY application that has a taddle bit of animations that change the container height ? If yes, then you can't like it.  In order to replicate the behaviour .... put in a controller

def test():
     return dict()

navigate to it, press any response.toolbar button. Ugly.

The problem with the sticky footer that I get is that when the page has low height, the footer appears in the middle of the page. Also I like tall footers with sitemaps. I do not know how to make those with a sticky footer. Perhaps I do not understand your proposal. How would you like it

This is how, BTW, http://getbootstrap.com/examples/sticky-footer-navbar/sticky-footer-navbar.css

- web2py.js should probably add a btn-default to buttons

it does it

nope. https://github.com/web2py/web2py/blob/master/applications/welcome/static/js/web2py.js#L79

You are right. It is done here:

https://github.com/web2py/web2py/blob/master/applications/welcome/static/js/web2py-bootstrap3.js#L106

because it is a bs feature.

- form.add_button() behaves wrongly (see the "remember me" on the login page)

not sure how. can you help?

IMHO a revision of extra_fields for SQLFORM is needed, in addition to add_button(). the same thing happens with the "check to delete" buttons. the label (including separator) is after the checkbox, and doesn't seem to be styled as dictated by the formstyle.

Can you fix it? Else I can do it later or tomorrow.

- auth.settings.formstyle doesn't inherit response.formstyle. Either we set it explicitely in db.py or we change auth logic to honor response.formstyle

It does here: https://github.com/web2py/web2py/blob/master/gluon/tools.py#L1361 is it not working?

whoops. It's working. Need to raise another point then: there is no management of error-related classes in core-shipped bs3-related formstyles.

I agree they should not be used but why remove them since we went thought the table of implementing them?

try it on a narrow screen, you'll see that a vertical scrollbar is added. Try to navigate it on a touch-screen and you'll quickly realize it's impossible. I'm not very keen of "add if someone went through the trouble of implementing". We dropped far more "sweaty implementations": if they don't work, they don't. Moreover, it's not our sweat, it's copied off bootsnipp. In addition to that, there's a reason why it's a snippet and it's not included in bs3. Mobile navigation.

There are other subtle small bits on web2py-bootstrap3.js that are plain wrong (or, to better explain, they were useful in the original process but got left unused in the current state, which is is a copy-paste of portions of the original project).

I do not disagree but since you know what they are (the wrong bits) and I do not, can you be more explicit or remove them?

- docs are pointing to side-project also for web2py-bootstrap3.css. Several styles are unused and should be dropped. Some are plain wrong (form max-width: 600px ... .hidden replacing bs3 behaviour .... ?)

is there a way to remove them automatically?

There's a distinction to make....unused, added for no reason and overriding defaults.
unused could, theoretically. Code a page that includes all kinds of possible web2py-supplied pieces, then pass it through uncss or some chrome plugins. there are tons.
Added for no reason........well. There's no pre or post processor smarter than a brain.
overriding defaults is a bit of a conceptual nightmare.

<totally_personal_opinion coming_from="a DBA" not_so_skilled_in="web design">
If you embrace a css framework, you shouldn't override its rules.
To the previous statement, I append a part... "unless you are at least as badass as bootstrap developers", that in my case is obviously true.
I was discomforted from the previous set of overrides (still in web2py.css) that screwed my design in some occasions for no apparent reasons. If someone can comment/justify/show test cases I'd be more inclined to accept them, but as they stand, e.g. the max-width fixed limitation on EVERY form, no matter of media-queries, well, I really don't see the reason behind.
</totally_personal_opinion>

I agree. But we web2py.css is not imported anymore in welcome app. we only have web2py-bootstrap3.css. If there are rules that you think should not be overidden, I am happy to remove them. Which ones?

auth versioning is still enabled....... I'd disable it. Did anybody use janrain lately ? there's a bug opened on it: should we really keep it on the default scaffolding ?


PS: is the time to drop the examples app ? ATM I'm reaaaally eager to drop it: too much things to be revised and frankly the only link pointing to it is embedded in an (old)copy of web2py.com and reachable by a not-so-visible link….

I agree but not in this version. That requires changing the documentation as well as my deployment process.

Bummer. At least keep examples in sync with what web2py.com is now then.

web2py.com is the examples app from the stable branch. they get in sync when I push stable.

PS2: the scaffolding app carries the hypermedia module : apart from it being largely untested (and tearing the app down with large tables by default), it requires python 2.7. Nice POC, but maybe scaffolding isn't the right place to advertise.

What do others think about this? I do not have strong opinions but it is something I want to encourage people to try and use.

well. from "we shouldn't drop 2.5" to "we force newbies to 2.7" in less than a week, you're a welcome addition to 2.7 supporters :-P .

you are right. I will remove it.

I just drop it as soon as I see it because it'll try to serialize 10k rows without even blinking.

[Niphlod]

At the end, it seems I'm forced to edit the bs3 even if I don't like it.....

The problem with the sticky footer that I get is that when the page has low height, the footer appears in the middle of the page. Also I like tall footers with sitemaps. I do not know how to make those with a sticky footer. Perhaps I do not understand your proposal. How would you like it

as the example it has been taken for http://getbootstrap.com/examples/sticky-footer-navbar/ .

 

This is how, BTW, http://getbootstrap.com/examples/sticky-footer-navbar/sticky-footer-navbar.css

- web2py.js should probably add a btn-default to buttons

it does it

nope. https://github.com/web2py/web2py/blob/master/applications/welcome/static/js/web2py.js#L79

You are right. It is done here:

https://github.com/web2py/web2py/blob/master/applications/welcome/static/js/web2py-bootstrap3.js#L106

because it is a bs feature.

that's definitely not the place to have it. that script is not applied to components (LOAD(..., ajax=True), for instance)

 

- form.add_button() behaves wrongly (see the "remember me" on the login page)

not sure how. can you help?

IMHO a revision of extra_fields for SQLFORM is needed, in addition to add_button(). the same thing happens with the "check to delete" buttons. the label (including separator) is after the checkbox, and doesn't seem to be styled as dictated by the formstyle.

Can you fix it? Else I can do it later or tomorrow.

- auth.settings.formstyle doesn't inherit response.formstyle. Either we set it explicitely in db.py or we change auth logic to honor response.formstyle

It does here: https://github.com/web2py/web2py/blob/master/gluon/tools.py#L1361 is it not working?

whoops. It's working. Need to raise another point then: there is no management of error-related classes in core-shipped bs3-related formstyles.

I agree they should not be used but why remove them since we went thought the table of implementing them?

try it on a narrow screen, you'll see that a vertical scrollbar is added. Try to navigate it on a touch-screen and you'll quickly realize it's impossible. I'm not very keen of "add if someone went through the trouble of implementing". We dropped far more "sweaty implementations": if they don't work, they don't. Moreover, it's not our sweat, it's copied off bootsnipp. In addition to that, there's a reason why it's a snippet and it's not included in bs3. Mobile navigation.

There are other subtle small bits on web2py-bootstrap3.js that are plain wrong (or, to better explain, they were useful in the original process but got left unused in the current state, which is is a copy-paste of portions of the original project).

I do not disagree but since you know what they are (the wrong bits) and I do not, can you be more explicit or remove them?

once done, I don't want to hear anyone saying "why did you remove them"...............

 

- docs are pointing to side-project also for web2py-bootstrap3.css. Several styles are unused and should be dropped. Some are plain wrong (form max-width: 600px ... .hidden replacing bs3 behaviour .... ?)

is there a way to remove them automatically?

There's a distinction to make....unused, added for no reason and overriding defaults.
unused could, theoretically. Code a page that includes all kinds of possible web2py-supplied pieces, then pass it through uncss or some chrome plugins. there are tons.
Added for no reason........well. There's no pre or post processor smarter than a brain.
overriding defaults is a bit of a conceptual nightmare.

<totally_personal_opinion coming_from="a DBA" not_so_skilled_in="web design">
If you embrace a css framework, you shouldn't override its rules.
To the previous statement, I append a part... "unless you are at least as badass as bootstrap developers", that in my case is obviously true.
I was discomforted from the previous set of overrides (still in web2py.css) that screwed my design in some occasions for no apparent reasons. If someone can comment/justify/show test cases I'd be more inclined to accept them, but as they stand, e.g. the max-width fixed limitation on EVERY form, no matter of media-queries, well, I really don't see the reason behind.
</totally_personal_opinion>

I agree. But we web2py.css is not imported anymore in welcome app. we only have web2py-bootstrap3.css. If there are rules that you think should not be overidden, I am happy to remove them. Which ones?

same as before: once done, I don't want to hear anyone saying "why did you remove them"...............

 

auth versioning is still enabled....... I'd disable it. Did anybody use janrain lately ? there's a bug opened on it: should we really keep it on the default scaffolding ?


PS: is the time to drop the examples app ? ATM I'm reaaaally eager to drop it: too much things to be revised and frankly the only link pointing to it is embedded in an (old)copy of web2py.com and reachable by a not-so-visible link….

I agree but not in this version. That requires changing the documentation as well as my deployment process.

Bummer. At least keep examples in sync with what web2py.com is now then.

web2py.com is the examples app from the stable branch. they get in sync when I push stable.

uhm. I compared visually them and there's something wrong. 

 

[Massimo DiPierro]

On Mar 18, 2015, at 4:06 AM, Niphlod <nip...@gmail.com> wrote:

At the end, it seems I'm forced to edit the bs3 even if I don't like it…..

and I appreciate you doing that a lot. :-)

The problem with the sticky footer that I get is that when the page has low height, the footer appears in the middle of the page. Also I like tall footers with sitemaps. I do not know how to make those with a sticky footer. Perhaps I do not understand your proposal. How would you like it

as the example it has been taken for http://getbootstrap.com/examples/sticky-footer-navbar/ .

 

This is how, BTW, http://getbootstrap.com/examples/sticky-footer-navbar/sticky-footer-navbar.css

- web2py.js should probably add a btn-default to buttons

it does it

nope. https://github.com/web2py/web2py/blob/master/applications/welcome/static/js/web2py.js#L79

You are right. It is done here:

https://github.com/web2py/web2py/blob/master/applications/welcome/static/js/web2py-bootstrap3.js#L106

because it is a bs feature.

that's definitely not the place to have it. that script is not applied to components (LOAD(..., ajax=True), for instance)

But it refers to BS specific classes. How would you do this?

 

- form.add_button() behaves wrongly (see the "remember me" on the login page)

not sure how. can you help?

IMHO a revision of extra_fields for SQLFORM is needed, in addition to add_button(). the same thing happens with the "check to delete" buttons. the label (including separator) is after the checkbox, and doesn't seem to be styled as dictated by the formstyle.

Can you fix it? Else I can do it later or tomorrow.

- auth.settings.formstyle doesn't inherit response.formstyle. Either we set it explicitely in db.py or we change auth logic to honor response.formstyle

It does here: https://github.com/web2py/web2py/blob/master/gluon/tools.py#L1361 is it not working?

whoops. It's working. Need to raise another point then: there is no management of error-related classes in core-shipped bs3-related formstyles.

I agree they should not be used but why remove them since we went thought the table of implementing them?

try it on a narrow screen, you'll see that a vertical scrollbar is added. Try to navigate it on a touch-screen and you'll quickly realize it's impossible. I'm not very keen of "add if someone went through the trouble of implementing". We dropped far more "sweaty implementations": if they don't work, they don't. Moreover, it's not our sweat, it's copied off bootsnipp. In addition to that, there's a reason why it's a snippet and it's not included in bs3. Mobile navigation.

There are other subtle small bits on web2py-bootstrap3.js that are plain wrong (or, to better explain, they were useful in the original process but got left unused in the current state, which is is a copy-paste of portions of the original project).

I do not disagree but since you know what they are (the wrong bits) and I do not, can you be more explicit or remove them?

once done, I don't want to hear anyone saying "why did you remove them"……………

I cannot promise but at least I will know what bits you refer too. ;-)

 

- docs are pointing to side-project also for web2py-bootstrap3.css. Several styles are unused and should be dropped. Some are plain wrong (form max-width: 600px ... .hidden replacing bs3 behaviour .... ?)

is there a way to remove them automatically?

There's a distinction to make....unused, added for no reason and overriding defaults.
unused could, theoretically. Code a page that includes all kinds of possible web2py-supplied pieces, then pass it through uncss or some chrome plugins. there are tons.
Added for no reason........well. There's no pre or post processor smarter than a brain.
overriding defaults is a bit of a conceptual nightmare.

<totally_personal_opinion coming_from="a DBA" not_so_skilled_in="web design">
If you embrace a css framework, you shouldn't override its rules.
To the previous statement, I append a part... "unless you are at least as badass as bootstrap developers", that in my case is obviously true.
I was discomforted from the previous set of overrides (still in web2py.css) that screwed my design in some occasions for no apparent reasons. If someone can comment/justify/show test cases I'd be more inclined to accept them, but as they stand, e.g. the max-width fixed limitation on EVERY form, no matter of media-queries, well, I really don't see the reason behind.
</totally_personal_opinion>

I agree. But we web2py.css is not imported anymore in welcome app. we only have web2py-bootstrap3.css. If there are rules that you think should not be overidden, I am happy to remove them. Which ones?

same as before: once done, I don't want to hear anyone saying "why did you remove them"……………

OK. about this I promise.

 

auth versioning is still enabled....... I'd disable it. Did anybody use janrain lately ? there's a bug opened on it: should we really keep it on the default scaffolding ?


PS: is the time to drop the examples app ? ATM I'm reaaaally eager to drop it: too much things to be revised and frankly the only link pointing to it is embedded in an (old)copy of web2py.com and reachable by a not-so-visible link….

I agree but not in this version. That requires changing the documentation as well as my deployment process.

Bummer. At least keep examples in sync with what web2py.com is now then.

web2py.com is the examples app from the stable branch. they get in sync when I push stable.

uhm. I compared visually them and there's something wrong. 

 

[Niphlod]

ok, will work on it right about now. It'll be covering ALL things I brought up that are not fixed, even if anyone didn't care to discuss it here (and there).

As for bs3 related js code, which IMHO should be only related to:
- btn btn-default to all buttons (just because it doesn't hurt, even if it has e.g. btn-warning on it)
- list:* widgets
and possibly a different calendar widget for date(time) fields, there's another "discussion"

is web2py.js a "worldwide" thing that needs to be updated within every application at the same time a new web2py version comes up, or is specific to the scaffolding app someone choose ?
As it stands right now, albeit "less specific", there are a few things that are tailored only for bs2 template.
There has been a time when we recommended to overwrite any web2py.js with the latest web2py.js to make an app work, because we changed some implementations that were done in core code previously by mistake.
This isn't the case (we didn't change any logic), but, if we want to keep web2py.js just about "the js logic needed by web2py as a framework, regardless the scaffolding used", then the method can only be to ship a web2py-bootstrap3.js that overrides the relevant bits on web2py.js .
If instead web2py.js is always specific to the scaffolding that is shipped, than the accomodations can be done in web2py.js directly.
I'll work with the former assumption, in order to demonstrate what a scaffolding based on another css framework could hook up the existing web2py.js code.

[Massimo DiPierro]

On Mar 18, 2015, at 2:42 PM, Niphlod <nip...@gmail.com> wrote:

ok, will work on it right about now. It'll be covering ALL things I brought up that are not fixed, even if anyone didn't care to discuss it here (and there).

Thanks Simone

As for bs3 related js code, which IMHO should be only related to:
- btn btn-default to all buttons (just because it doesn't hurt, even if it has e.g. btn-warning on it)
- list:* widgets

This works for me. 

and possibly a different calendar widget for date(time) fields, there's another “discussion"

I would not change the calender widget. I think there is value in shipping with a css neutral calendar.

is web2py.js a "worldwide" thing that needs to be updated within every application at the same time a new web2py version comes up, or is specific to the scaffolding app someone choose ?

I consider is a “worldwide” thing.

As it stands right now, albeit "less specific", there are a few things that are tailored only for bs2 template.
There has been a time when we recommended to overwrite any web2py.js with the latest web2py.js to make an app work, because we changed some implementations that were done in core code previously by mistake.
This isn't the case (we didn't change any logic), but, if we want to keep web2py.js just about "the js logic needed by web2py as a framework, regardless the scaffolding used", then the method can only be to ship a web2py-bootstrap3.js that overrides the relevant bits on web2py.js .
If instead web2py.js is always specific to the scaffolding that is shipped, than the accomodations can be done in web2py.js directly.
I'll work with the former assumption, in order to demonstrate what a scaffolding based on another css framework could hook up the existing web2py.js code.

I agree.

[Niphlod]

First iteration is here https://github.com/web2py/web2py/pull/854
Needed to change gluon/tools.py too, to make it honor response.form_label_separator, that incidentally was risen as an issue exactly for bs3.

- cruft removal
- web2py-bootstrap3.js behaving like it should, overriding web2py.js where needed
- styles sanitization
- AppConfig examples

For sure there are still many (probably too many) "ie hacks" (ie7, ie8, ie9) I'm not sure are needed.
Will check in the next days thanks to modern.ie images.
Given that IE7 is not supported by bs3 I'd be inclined to drop all IE7 related ones without even blinking.


Didn't test anything past usual forms, and grids.
Todo for everybody... test, test, test... on the top of my head, some things that is often left in a corner untested are:
- wiki
- crud (?!)
- all kinds of widgets
- form.add_button()
- IS_IN_SET with checkboxes and radios (multiple too)
-   



PS: If we are going towards response.ui, now is definitely the time to chime in.
 

and possibly a different calendar widget for date(time) fields, there's another “discussion"

I would not change the calender widget. I think there is value in shipping with a css neutral calendar.

If only the whole app wasn't based on a specific css-framework, I'd agree with you. Given the actual solution doesn't fit the bill for lots of usecases (i.e. mobile, touch) (and, frankly, I find myself most of the times to drop it), and it's unsupported, and it's discontinued, and can't be tuned, and can't be customized... why don't treat calendar.css + calendar.js as web2py.css ? Still there, but unused.

[Massimo DiPierro]

On Mar 18, 2015, at 5:29 PM, Niphlod <nip...@gmail.com> wrote:

First iteration is here https://github.com/web2py/web2py/pull/854
Needed to change gluon/tools.py too, to make it honor response.form_label_separator, that incidentally was risen as an issue exactly for bs3.

- cruft removal
- web2py-bootstrap3.js behaving like it should, overriding web2py.js where needed
- styles sanitization
- AppConfig examples

For sure there are still many (probably too many) "ie hacks" (ie7, ie8, ie9) I'm not sure are needed.
Will check in the next days thanks to modern.ie images.
Given that IE7 is not supported by bs3 I'd be inclined to drop all IE7 related ones without even blinking.

I agree.

Didn't test anything past usual forms, and grids.
Todo for everybody... test, test, test... on the top of my head, some things that is often left in a corner untested are:
- wiki
- crud (?!)

I think we have been clear they are legacy. I stopped supporting those eons ago.

- all kinds of widgets

- form.add_button()
- IS_IN_SET with checkboxes and radios (multiple too)
-   

I will test the latter.

PS: If we are going towards response.ui, now is definitely the time to chime in.

The problem is response.formstyle has been there for 1 year. Even if we move to reponse.ui I do not need response.formstyle going away. Anyway, @Anthony, any comments?

 

and possibly a different calendar widget for date(time) fields, there's another “discussion"

I would not change the calender widget. I think there is value in shipping with a css neutral calendar.

If only the whole app wasn't based on a specific css-framework, I'd agree with you. Given the actual solution doesn't fit the bill for lots of usecases (i.e. mobile, touch) (and, frankly, I find myself most of the times to drop it), and it's unsupported, and it's discontinued, and can't be tuned, and can't be customized... why don't treat calendar.css + calendar.js as web2py.css ? Still there, but unused.

[Niphlod]

Didn't test anything past usual forms, and grids.
Todo for everybody... test, test, test... on the top of my head, some things that is often left in a corner untested are:
- wiki
- crud (?!)

I think we have been clear they are legacy. I stopped supporting those eons ago.

meant auth.wiki ....
 

The problem is response.formstyle has been there for 1 year. Even if we move to reponse.ui I do not need response.formstyle going away. Anyway, @Anthony, any comments?

Being there and being documented as stable and used within applications is a totally different thing. I didn't know about it until 2 weeks ago.
There's no mention of response.formstyle anywhere in the book, and, TBF, auth.settings.formstyle is the one documented.

[Massimo DiPierro]

looks great. I made some new minor changes to the css. I also I added something you may not like:

input[type=checkbox], input[type=radio] {

    margin: 4px 4px 0 0;

}

.btn {

    margin-right: 4px;

}

The former is required to add space between checkboxes and their right labels. According to the BS3 docs the input is supposed to be in the label but our widgets do not do it and I think it is crazy for the input to be inside the label.

The second is so that multiple buttons can can near each other and do not touch each other. I am opposed to buttons touching each other. It is gross.

Massimo

On Mar 18, 2015, at 5:29 PM, Niphlod <nip...@gmail.com> wrote:

[Niphlod]

On Thursday, March 19, 2015 at 12:34:35 AM UTC+1, Massimo Di Pierro wrote:

looks great. I made some new minor changes to the css. I also I added something you may not like:

input[type=checkbox], input[type=radio] {

    margin: 4px 4px 0 0;

}

.btn {

    margin-right: 4px;

}

The former is required to add space between checkboxes and their right labels. According to the BS3 docs the input is supposed to be in the label but our widgets do not do it and I think it is crazy for the input to be inside the label.

The second is so that multiple buttons can can near each other and do not touch each other. I am opposed to buttons touching each other. It is gross.

Massimo

the input inside the label allows you to click on the label and have the input checked. It's User Interaction 101. 

As for buttons not touching, if that rule won't let me use button groups, then it's an issue.

[Massimo DiPierro]

On Mar 19, 2015, at 2:21 AM, Niphlod <nip...@gmail.com> wrote:

On Thursday, March 19, 2015 at 12:34:35 AM UTC+1, Massimo Di Pierro wrote:

looks great. I made some new minor changes to the css. I also I added something you may not like:

input[type=checkbox], input[type=radio] {

    margin: 4px 4px 0 0;

}

.btn {

    margin-right: 4px;

}

The former is required to add space between checkboxes and their right labels. According to the BS3 docs the input is supposed to be in the label but our widgets do not do it and I think it is crazy for the input to be inside the label.

The second is so that multiple buttons can can near each other and do not touch each other. I am opposed to buttons touching each other. It is gross.

Massimo

the input inside the label allows you to click on the label and have the input checked. It's User Interaction 101. 

That UI still works. It is not related to the input being (or not) in the label but it requires the <label for> attribute which we do set.

As for buttons not touching, if that rule won't let me use button groups, then it's an issue.

I tried and it works unaltered.

[Massimo DiPierro]

There is a repost that web2py_win.zip does not start. Anybody tried it? I do not have a windows machine?

[Giovanni Barillari]

Except for bootstrap3 related stuffs, which I haven't followed, before 2.10.1 release I suggest:

- [pydal] clean out a bit the google related code (I can see some stuffs are now unused since @massimo refactor)

- [pydal] double check connection issues (if any), I think we should set travis to be verbose on testing (If I remember correctly, I've seen something strange about pg8000 sockets warnings)

- [web2py] review scripts/packages (windows?) due to the big change from dal.py -> gluon/dal -> dal.py + packages/dal

/Giovanni

[Giovanni Barillari]

Also, I think something in contrib is actually broken after the pydal change.

[Massimo DiPierro]

Are you looking into these?  I can look into scripts and contrib, unless you already know what to look for.

[Giovanni Barillari]

I will look at web2py's contrib and investigate a bit more pydal code tonight. I will post a status update (and maybe some PRs) tomorrow .

/Giovanni

[Massimo DiPierro]

I only see dal imported in heroku.py and the path was wrong. I fixed it. 

[Richard Vézina]

What about auth_ldap.py?

I test and report here...

Richard

[Richard Vézina]

Work fine, at least with AD which is my LDAP directory available.

Richard

[Niphlod]

I downloaded the "yesterday" version and it starts. Need to confirm if it's working on postgresql, which I'll do in a minute. wait for it....

[Niphlod]

there's a bug in caching selects with cache.disk. It happens on both linux and windows, so it's related to "core" code, nothing version or build specific.
Guess it's time to resurface the discussion about not having tests on web2py related to pydal code ^_^'

to replicate, just create a table with whatever backend, then try to
db(db.table.id>0).select(cache=(cache.disk, 60))

works
db(db.table.id>0).select(cache=(cache.disk, 60), cacheable=True)

doesn't.

[Massimo DiPierro]

I agree web2py should test pydal. Let me know if you are already working this else I can fix it tonight.

[Giovanni Barillari]

I will try to implement them in the next hours.

You received this message because you are subscribed to a topic in the Google Groups "web2py-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py-developers/hv2tn5b562Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py-develop...@googlegroups.com.

[Giovanni Barillari]

@niphlod: I'm not an expert about the cache code inside pydal. Can someone explain me in short words what 'cacheable' parameter does?

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---

You received this message because you are subscribed to a topic in the Google Groups "web2py-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py-developers/hv2tn5b562Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py-develop...@googlegroups.com.

[Massimo DiPierro]

Good question. 

So when you make a select the Db sends back a list of tuples. Web2py parses them into Row(s) objects. This parsing takes time. A Row object is not cachable because has update_record and delete_record methods.

So when you do (…).select(…cache=….) weh2py caches the list of tuples before parsing

When you do (…).select(… cache=…,cachable=True) web2py cache omits the Row methods and caches the Rows object instead.

You received this message because you are subscribed to the Google Groups "web2py-developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.

[Massimo DiPierro]

I should add I am not sure cachable=True on cache.disk ever worked. Do we know it did?

[Niphlod]

On Thursday, March 19, 2015 at 10:59:49 PM UTC+1, Massimo Di Pierro wrote:

I should add I am not sure cachable=True on cache.disk ever worked. Do we know it did?

lol........ at this point I'm burn out.

I find out (trial and errors) that the issue seems lying in

rows.db.__dict__

as soon as you remove it, e.g. using delattr(therowsobject.db, '__dict__'), pickle doesn't complain. Of course it fails badly afterwards, when trying to reconstruct the rows object, but that's not really the point


BTW: @leonel, the author of the new cache.disk (that doesn't seem really to be the issue at hand ) ... inspecting the code, I found out that at this point https://github.com/web2py/web2py/blob/master/gluon/cache.py#L325 "value" holds already the tuple containing time.time() and the actual value.... later (line 328), time.time() is added another time....
When fetching https://github.com/web2py/web2py/blob/master/gluon/cache.py#L341 , you correctly return value, but timestamp is not used.......... why the need to store time.time() two times ?
 

[Niphlod]

however, I'm still not that burnout that I can't test............

web2py 2.9.12 works fine.

TBF, the whole point of cacheable=True IS to be able to store the Rows object in any cache backend that accepts a pickled string.

code to test

./web2py.py -M -S welcome
>>> db.auth_user.insert(first_name='homer')
1L
>>> db.commit()
>>> db(db.auth_user.id>0).select(cacheable=True, cache=(cache.disk, 60))
<Rows (1)>
>>> exit()

guess that at pydal level, a - rather specific - test can be done doing

import pickle
db.define_table('t_a', Field('f_a'))
db.commit()
db.t_a.insert(f_a='homer')
db.commit()
rtn = db(db.t_a.id>0).select(cacheable=True)
pickle.dumps(rtn)

for all matters and purposes, a smaller (and maybe helpful test) is to do in a controller

import pickle
def test():
    rtn = db(db.auth_user.id>0).select(cacheable=True)
    a = pickle.dumps(rtn)
    return locals()

2.9.12 doesn't break executing the code......... trunk instead fails reporting

Can't pickle <function repr_ref at 0x7f8b3008f488>: it's not found as gluon.dal.repr_ref

I'm calling the day, though. Too many emotions this evening : I'll leave this up to you. Sorry I couldn't see it through.

[Massimo DiPierro]

Thanks Simone. You helped a lot. I will check it as soon as I get back home.

[Giovanni Barillari]

Ok, I have some bad news about pydal testing inside web2py.

I tried to figure out a way to import the tests file of pydal in web2py while running web2py tests, but I think this is not achievable (at least considering how the pydal tests are structured now). I'll try to explain it.

Here are the facts:

1) we don't want to duplicate tests code, because it will be a PITA

2) the better way to run pydal tests inside web2py is to add another test command inside tox, so it can load the correct tests

3) we need to set the ENV parameter (tox or travis) for the database before running the tests to use the 3 contrib drivers (and only them), so pymsql, pyodb and pg8000

4) the injection of the contrib drivers is performed only when the DAL class is imported from gluon

So, basically, even if we read the tests, and add the required ENV parameters, the pydal tests use the pydal's DAL class, not the gluon one.

I don't know any way to inject the correct DAL class into the unittest run (and I don't know if it's even possible).

Eventually, the way COULD be (meaning that is only a theory at the moment) creating a new TestCase class inside gluon tests, manually import all the classes inside pydal sql.py test file, inject them into the new class and try to get it working. I can try this but I think it will require at least a couple of days.

/Giovanni

--

[Giovanni Barillari]

As far for the connection issues, everything seems fine, I only had warning on pg8000 on py3 test branch. Python 2.6 and 2.7 seems ok.

Working on gae cleanup right now.

[Giovanni Barillari]

Regarding the pickle issue, seems that the problem is around the default validators injection: https://github.com/web2py/web2py/blob/master/gluon/dal.py#L64

I'm trying to investigate it..

[Giovanni Barillari]

@massimo I've send some PRs.

This is for gae cleanup: https://github.com/web2py/pydal/pull/91 read the description carefully ;)

https://github.com/web2py/pydal/pull/92 and https://github.com/web2py/web2py/pull/857 should fix the issue with cache.disk and cacheable. Please, merge the pydal PR first, then update the pydal tracking inside web2py repo and then merge the web2py PR.

Now I should sleep a bit ^^"

[Massimo DiPierro]

This is fixed now. I do not like the solution but I could not find anything better. I do not like having a pydal/DAL and dal.py/DAL. There should really be a single object. This is calling for trouble. 

Anyway, we can leave with it for now. If you have any solution, let me know.

On Mar 19, 2015, at 5:40 PM, Niphlod <nip...@gmail.com> wrote:

[Paolo Valleri]

Beside bt3, we've +160 open issues, before any new release it should be nice to make a pass to either fix or close them.

 Paolo

[Giovanni Barillari]

I think this is not changeable. It's the only way to have the same APIs both on pyDAL and web2py (and weppy).

Let me try to hack a bit unittest ;)

/Giovanni

2015-03-20 7:29 GMT+01:00 Massimo DiPierro <massimo....@gmail.com>:

You received this message because you are subscribed to a topic in the Google Groups "web2py-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py-developers/hv2tn5b562Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py-develop...@googlegroups.com.

[Massimo DiPierro]

I have a pushed a better solution.

[Giovanni Barillari]

This not solves the tests issue.

[Massimo DiPierro]

? This solves the pickle.dumps(rtn) issue. No?

[Giovanni Barillari]

That was already solved by my commits..

The argue about double DAL class was about running pydal tests inside web2py, not the pickling issue.

[Niphlod]

even if I don't use GAE (nor plan to spend any time on it) ... did anyone report back on the - seemingly - broken examples/app.example.yaml that is shipped with web2py ?
I recall spending a lot of time on it with @lpg but the final response was never reported back nor an update has been made to that file since a year ago.

[Massimo DiPierro]

I have been using it without trouble. What was the problem?

On Mar 20, 2015, at 3:52 PM, Niphlod <nip...@gmail.com> wrote:

even if I don't use GAE (nor plan to spend any time on it) ... did anyone report back on the - seemingly - broken examples/app.example.yaml that is shipped with web2py ?
I recall spending a lot of time on it with @lpg but the final response was never reported back nor an update has been made to that file since a year ago.

[Niphlod]

Don't really know what you installed on GAE.
regexes for static_files were not working, and neither some of the patterns on skip_files.
It made appearance when we shipped 2.10.x , because then the "admin" interface "broke". Upon inspection it was found out that the problem didn't appeared in 2.9.x just because 9 is 1 "digit long", while "10" is 2 "digits long".

BTW: this regex https://github.com/web2py/web2py/blob/master/examples/app.example.yaml#L24 works only for "single digits".

see https://groups.google.com/forum/#!topic/web2py-developers/CYE9E7sL7QI , https://groups.google.com/d/msg/web2py/sN8mWOiCqgI/HPtppCDvXrQJ and https://groups.google.com/d/msg/web2py-developers/CYE9E7sL7QI/R5GIuEgdoXgJ

[Massimo DiPierro]

:-) Fixed.

[Niphlod]

last time I tried (lpg provided an account), summary was (as reported in one in the linked threads)

this was plain wrong
/(.+?)/static/_\d.\d.\d\/(.+)
as in regex terms wouldn't had allowed you to use (using x.y.z as the example) to have a x.y.zz.

/(.+?)/static/_[\d]+\.[\d]+\.[\d]+/(.+)
is more correct from a regex POV, because sintactically it allows you to have AT LEAST one digit for every x, y, or z, ("+" is "one or more"). Unfortunately, GAE seems to NOT respect regex syntax and maps correcly x.y.z but not xx.yy.zz nor xx.y.z nor x.y.zz

/(.+?)/static/_(\d+\.\d+\.\d+)

/(.+)
this is exactly as before, but the _1.2.3 part is now part of a grouping. This means that the backreferences must be shifted (static_files: applications/\1/static/\3 instead of static_files: applications/\1/static/\2). But in now the "rewrite" works, while the previous didn't


in regex terms, this means that _1.2.3 NEEDS to be in a group to allow both _1.2.3 and _123.123.123 to work fine. Strangely enough the regex syntax (that is documented to work fine) doesn't work if not inside a grouping.

If you are confirming that it's all working fine (that means someone reported the bug to GAE developers and got fixed), I'm happy.

[Massimo DiPierro]

On Mar 20, 2015, at 6:27 PM, Niphlod <nip...@gmail.com> wrote:

last time I tried (lpg provided an account), summary was (as reported in one in the linked threads)

this was plain wrong
/(.+?)/static/_\d.\d.\d\/(.+)
as in regex terms wouldn't had allowed you to use (using x.y.z as the example) to have a x.y.zz.

/(.+?)/static/_[\d]+\.[\d]+\.[\d]+/(.+)
is more correct from a regex POV, because sintactically it allows you to have AT LEAST one digit for every x, y, or z, ("+" is "one or more"). Unfortunately, GAE seems to NOT respect regex syntax and maps correcly x.y.z but not xx.yy.zz nor xx.y.z nor x.y.zz

I am using it. I have Google App Engine Launcher, 1.9.17 and it works for me. I have not tried deploy it. Are you saying it works locally (with dev_appserver) but not when deployed? I will try deploy over the week-end.

/(.+?)/static/_(\d+\.\d+\.\d+)

/(.+)
this is exactly as before, but the _1.2.3 part is now part of a grouping. This means that the backreferences must be shifted (static_files: applications/\1/static/\3 instead of static_files: applications/\1/static/\2). But in now the "rewrite" works, while the previous didn't


in regex terms, this means that _1.2.3 NEEDS to be in a group to allow both _1.2.3 and _123.123.123 to work fine. Strangely enough the regex syntax (that is documented to work fine) doesn't work if not inside a grouping.

If you are confirming that it's all working fine (that means someone reported the bug to GAE developers and got fixed), I'm happy.

On Saturday, March 21, 2015 at 12:13:26 AM UTC+1, Massimo Di Pierro wrote:

:-) Fixed.

On Mar 20, 2015, at 5:30 PM, Niphlod <nip...@gmail.com> wrote:

[Niphlod]

apparently dev did not behave as when deployed, and, TBF, wasn't coherent with the docs, albeit they're not specifying which kind of regex syntax is allowed.

[Massimo DiPierro]

very weird. OK will check.

On Mar 20, 2015, at 6:36 PM, Niphlod <nip...@gmail.com> wrote:

apparently dev did not behave as when deployed, and, TBF, wasn't coherent with the docs, albeit they're not specifying which kind of regex syntax is allowed.

[Niphlod]

BTW: @leonel, the author of the new cache.disk (that doesn't seem really to be the issue at hand ) ... inspecting the code, I found out that at this point https://github.com/web2py/web2py/blob/master/gluon/cache.py#L325 "value" holds already the tuple containing time.time() and the actual value.... later (line 328), time.time() is added another time....
When fetching https://github.com/web2py/web2py/blob/master/gluon/cache.py#L341 , you correctly return value, but timestamp is not used.......... why the need to store time.time() two times ?

if noone works it out tomorrow I'll tackle this too.

[Niphlod]

BTW 115: someone can please remind me why we switched to this implementation of cache.disk ?
ATM I can't really find out why the current one should be preferred to the "old" one based on shelve....

[Massimo DiPierro]

What I remember is that people reported locking issues. So the redesign consist of having each key/value pair in its own file. The file is not locked but never overwritten, only replaced, so it does not need to be locked.

[Niphlod]

uhm, ok, I'll test it.
That assumption would be valid only in multiprocess environment, since in threaded there's really no benefit because evey access still triggers the threading lock.
IMHO there's a small hiccup even in multiprocess environments: all operations need to insist on the file representing the "ephemeral" statistics dict, irregardless of the choosen key.
Did the "original inventor" have a script proving the benefits ?

[Niphlod]

oh, my. Ok, it doesn't "block" on accessing statistics but..................
I have a few things to say about cache.disk.....ok, lack of tests can be blamed but....... inclusion into gluon was either too "trusty" or simply wrong.

A) cache statistics are initialized (if you set a SINGLE cached value, it creates 2 files) but it's actually NEVER updated. There's no way to see what are the stats about hits and misses.
Goes without saying that either one of these action should be pursued:
           1) all code related to miss/hits needs to be pruned
           2) code should take care of updating statistics
B) what I originally spotted was right. there's a time.time() unusefully stored. the cached pickle is (time, (time, value)) instead of just (time, value)
C) even stripped out of the cache miss/hits machinery, the current code accesses the disk (and recalculates "dummy_hash") 2 times for every cache call. One for "exists", the other for actually reading or writing the file.
D) there's something terribly wrong in the code, on my system, or whatever.......can anyone test the following snippet ?

from gluon.storage import Storage
from gluon.cache import CacheOnDisk, CacheInRam
import time
import threading


s = Storage({'application': 'admin',
                     'folder': 'applications/admin'})

cache = CacheOnDisk(s)
#cache = CacheInRam(s)
   
def add_to_it():
    for a in range(1000):
        cache.increment('a')

def start_two_threads():
    threads = []
    for i in range(5):
        t = threading.Thread(target=add_to_it)
        threads.append(t)
        t.start()
    for t in threads:
        t.join()
    return 'finished'

if __name__ == '__main__':
    rtn = start_two_threads()
    print 'should be 5000', cache('a', lambda : 1, time_expire=10000)

CacheOnDisk NEVER returns 5000 .

[Leonel Câmara]

A) cache statistics are initialized (if you set a SINGLE cached value, it creates 2 files) but it's actually NEVER updated. There's no way to see what are the stats about hits and misses. 

They're updated in the __call__ method like in the other cache implementations. I'm all for just pruning it.

CacheOnDisk NEVER returns 5000 

Well that's to be expected, cache disk is not thread safe as it doesn't have any locks in it so if you do operations that aren't atomic that's what will happen.

The same thing can happen in cache ram if instead of using the provided increment which has a lock on the entire cache you make one yourself where you get the value from the cache and then you set it to +1 yourself.

Basically if you're going to use cached values for computations and you're writing the value of the result in the same key you're using for the computations, stuff will break unless you do your own concurrency management.

I think shelve has its own set of problems as it isn't "safe" either and you had to lock the entire cache.

I could use portalocker to make a lock for the file or the dir of the key being accessed I guess.

If we were running the cache on disk in its own process key-value store it would still have the same problem because you would be asking the cache for a value and then making another operation to store it.

The only solution to this is to lock the entire cache like cache ram and the old cache disk that used shelve did, what this effectively does is forbid any computation that uses the same key in the cache twice which removes the problem by simply not allowing it.

I'm open to suggestions.

[Massimo DiPierro]

I am not released 2.10.1 until Leonel and Niphlod reach and agreement about cache.disk. Sooner is better the latter. Tabling this is also OK with me as a temporary solution but I want to your opinion first.

[Niphlod]

On Wednesday, March 25, 2015 at 11:57:41 PM UTC+1, Leonel Câmara wrote:

A) cache statistics are initialized (if you set a SINGLE cached value, it creates 2 files) but it's actually NEVER updated. There's no way to see what are the stats about hits and misses. 

They're updated in the __call__ method like in the other cache implementations. I'm all for just pruning it.

if you update it but not store it somewhere, there's really not much of a point to update it. Try that out: if you can squeeze anything else than 0 from those counters, I'd like to know how. I couldn't.

 

CacheOnDisk NEVER returns 5000 

Well that's to be expected, cache disk is not thread safe as it doesn't have any locks in it so if you do operations that aren't atomic that's what will happen.

please don't get it in a wrong way, but WHEN anyone agreed in having a cache.disk that ISN'T thread-safe ?! Everything in web2py is, and for a lot of good reasons.

 

The same thing can happen in cache ram if instead of using the provided increment which has a lock on the entire cache you make one yourself where you get the value from the cache and then you set it to +1 yourself.

Except that the published API works around the issues you describe for the exact reason that a not-threadsafe cache is basically unusable for newbies. web2py is all about enforcing sane defaults, and the current implementation is NOT safe at all.

 

Basically if you're going to use cached values for computations and you're writing the value of the result in the same key you're using for the computations, stuff will break unless you do your own concurrency management.

I don't want to do that in my code, heck, I'm not even sure I could do it. 

If I stash something in the cache, I want it stashed, period. If I'm running concurrently the same computation (ever head of the "thundering herd problem" ?) which happens kinda "by default" in a web application, I want a consistent result out of it, not a behaviour that isn't the same if events triggering that computation are coming in differently disposed in a timeline.......the second computation gets blocked instead of running concurrently? good. it wins the second over the first? good. it depends? bad.

granted, we're not happy with locking the entire cache because it's the easiest - albeit slowest - implementation, and cache isn't really as smart as it could be, but thread-safety (at least for me) is a deal breaker.

 

[Leonel Câmara]

please don't get it in a wrong way, but WHEN anyone agreed in having a cache.disk that ISN'T thread-safe ?! Everything in web2py is, and for a lot of good reasons.

I have zero ego when it comes to code so no need to fear me taking offense. My rationale was that I explained how I was going to remove the locks and I showed everyone the code implementation, I can see how there should have been more discussion.  

  

Can I get until the end of the day to implement a fine grained lock cache disk? Frankly I think going back to the old shelve implementation is not good enough, the old cache disk could only be accessed by 1 thread of 1 process at a time which totally defeats the purpose of using a cache to speedup your website.

Then we could carefully review it/test it and hopefully by tomorrow afternoon have this issue sorted out.

[Niphlod]

No problems, blame on me that didn't look at the implementation enough. for everybody else chiming in, a little resume:

"how should it behave", at least IMHO:
Let's assume I have something that needs to be cached. it takes 30 seconds to create the "objectA" and 30 to create "objectB", and I'm in the need of storing into cache for speedup

- client1 sends the request that needs objectA
- app starts calculating the cached value, as it's not in the cache yet
- client2 sends the request that needs objectA
- IMHO there's no point in doing the same thing another time in parallel: 30 seconds of cpu wasted. client2 waits for the cached value to be readily available
- app finishes the creation, cache stores it
- client1 gets the response
- client2 gets the response

What happened until "recent times":
- client1 sends the request for objectA
- app starts calculating objectA
- client2 sends the request for objectA
- app finishes calculating objectA
- client1 gets the response
- client2 gets the response
Perfect! But.........indeed there was a "but"

What was wrong in the previous implementation:
- client1 sends request for objectA
- app starts calculating objectA
- client2 sends request for objectB
- app waits calculating objectB, because the entire "cache" is locked, not only the part relative to objectA
- app finishes calculating objectA
- client1 gets objectA
- app starts calculating objectB
- app finishes calculating objectB
- client2 gets objectB

What is wrong with the current implementation of cache.disk (and, closely related, why the cache.increment test shines in identifying a bad behaviour...)?
- client1 sends request for objectA
- app starts calculating objectA for the 1st time
- client2 sends request for objectA
- app starts calculating objectA for the 2nd time
- app finishes calculating objectA
- client1 gets the response
- app finishes calculating objectB
- client2 gets the response

if, like cache.increment, you're relying on the cache operations to be atomic on a single key, you're screwed. even if you didn't rely on atomic operations, you app burned double the cpu needed to serve client1 and client2, and neither got the page faster, because when app has finished the 1st round of calculation, there's no way to "signal" the second thread calculating the same thing to discard whatever it's doing and serve the just-computed value.


So, +1 for removing "entire cache lock", -1 for removing all kind of locks ^_^ cache should still avoid the "thundering herd" issue, locking (if possible) a single key in order to NOT have "two calculations of the same thing in parallel".

[Leonel Câmara]

As promised here's what I currently have:

https://github.com/leonelcamara/web2py/tree/lock_disk

It passes the tests in trunk and the parallel test you provided earlier.

It now also tracks statistics correctly but I still think they should be removed, or at least made optional somehow, as they constitute a bottleneck for the entire cache since every access must lock it.