Re: Multiples ajax forms (with LOAD). It is possible?

[Anthony]

The problem is that we made a change so that no session file is created at all if this is a new session and the session remains empty. So, when the index page is requested, no session file is created. Next, the form1 Ajax request comes in, and because there is no session file, there is nothing to read or lock. Meanwhile, the form2 Ajax request comes in, and again, there is no file to read or lock. So, both requests start with an empty session and add their respective formkeys to it. Next, the form1 request creates a session file and writes its version of the session to it. Finally, the form2 request opens and completely overwrites that session file with its version of the session (which does not include the form1 formkey).

What to do about this? I suppose one option would be to always create a new session file when a new session is started, even if the session is empty on the first request of the session (in the above example, an empty session file would be created on accessing the index page). Some apps never use the session, though, and would therefore have a bunch of unnecessary session files (though I suppose you could still use the global settings to turn off sessions).

Another option is to leave it up to the developer to save something to the session in the parent page request when the page contains multiple Ajax requests that will be accessing the session. Maybe we could provide a convenience method for this, such as session.save(), which would force saving the file even if the session is empty (such a method might have other uses, such as saving and then unlocking a session in the middle of a request).

Other ideas?

I think there's a bigger problem with sessions in cookies and the db -- they aren't locked at all, so you can get race conditions with them even once the session has initially been saved.

Anthony

On Monday, April 22, 2013 4:59:16 PM UTC-4, Niphlod wrote:

umpf.... I can't understand why this is not working "ok".

The problem lies indeed in the fact that one ajax request overwrites the session, if the session file is not there yet.....
I can only guess that the logic is failing to acquire a lock before creating the (new) file ....
If a request has completed yet, hence the session file is present, all goes perfectly ok (it's probably the reason why nobody else ever noticed "the glitch", and why prepending a "session.hello = 'world'" to the index function makes all go smoothly, it actually creates the session file before the two ajax requests come in)
can anyone testing this confirm this behaviour?
steps to reproduce:
- delete session file in the session/ dir
- open the index page
- session file ends up with one or another formname keys, never both
- reload the page
- session file holds both formnames
- put session.hello = 'world' in the index() function before the return dict()
- delete the session file in the sessions/ dir
- reload the page
- session file holds both formnames

If this is indeed the behaviour, we narrowed it down to a "glitch" that happens in one case only: 2 concurrent requests comes in and there is no session file yet.
We can start from there to coordinate efforts for the patch..... it's definitely not an issue with javascript.

[Niphlod]

On Tuesday, April 23, 2013 4:00:11 AM UTC+2, Anthony wrote:

The problem is that we made a change so that no session file is created at all if this is a new session and the session remains empty. So, when the index page is requested, no session file is created. Next, the form1 Ajax request comes in, and because there is no session file, there is nothing to read or lock. Meanwhile, the form2 Ajax request comes in, and again, there is no file to read or lock. So, both requests start with an empty session and add their respective formkeys to it. Next, the form1 request creates a session file and writes its version of the session to it. Finally, the form2 request opens and completely overwrites that session file with its version of the session (which does not include the form1 formkey).

ok, that's consistent with what observed
 

What to do about this? I suppose one option would be to always create a new session file when a new session is started, even if the session is empty on the first request of the session (in the above example, an empty session file would be created on accessing the index page). Some apps never use the session, though, and would therefore have a bunch of unnecessary session files (though I suppose you could still use the global settings to turn off sessions).

turning to the old system makes me "sick & sad" . Creating a session only when it's needed is "the right thing to do"© .
 

Another option is to leave it up to the developer to save something to the session in the parent page request when the page contains multiple Ajax requests that will be accessing the session. Maybe we could provide a convenience method for this, such as session.save(), which would force saving the file even if the session is empty (such a method might have other uses, such as saving and then unlocking a session in the middle of a request).

wouldn't a simple note in the book close the deal ? This happens only when there is no session file and the user concurrently requests something that needs to be saved to session. In any case, leaving up to the developer is the right choice.
 

Other ideas?

I think there's a bigger problem with sessions in cookies and the db -- they aren't locked at all, so you can get race conditions with them even once the session has initially been saved.

Anthony

it's a problem "in general": either you want concurrency or you want locking..... can we even theoretically solve that problem ? I don't think so.
 

[Ricardo Pedroso]

On Tue, Apr 23, 2013 at 8:28 AM, Niphlod <nip...@gmail.com> wrote:
>
>
> On Tuesday, April 23, 2013 4:00:11 AM UTC+2, Anthony wrote:
>>
>>
>> What to do about this? I suppose one option would be to always create a
>> new session file when a new session is started, even if the session is empty
>> on the first request of the session (in the above example, an empty session
>> file would be created on accessing the index page). Some apps never use the
>> session, though, and would therefore have a bunch of unnecessary session
>> files (though I suppose you could still use the global settings to turn off
>> sessions).
>
>
> turning to the old system makes me "sick & sad" . Creating a session only
> when it's needed is "the right thing to do"© .

This is a corner case and I guess easily identified in code.

I think it will be possible before write the session file
for the case there was none:

- acquire a thread lock
(or a temp file (eg: sessiog_file_name.tmp ) to sentinel
other processes
if running in multiprocessing, that can be checked with wsgi env)

- do another check if the session file exists
- if yes read an merge the current request data else pass
- write the file
- release the lock
(delete the tmp file if in multiprocessing).


Would this be viable?


Ricardo

[Anthony]

What to do about this? I suppose one option would be to always create a new session file when a new session is started, even if the session is empty on the first request of the session (in the above example, an empty session file would be created on accessing the index page). Some apps never use the session, though, and would therefore have a bunch of unnecessary session files (though I suppose you could still use the global settings to turn off sessions).

turning to the old system makes me "sick & sad" . Creating a session only when it's needed is "the right thing to do"© .

The above suggestion isn't quite going back to the old system. In the old system, the session file was created on the first request (of a new browser session) and re-written on every request (even if it didn't change) -- the latter was more a source of inefficiency than the former. The above suggestion would return to creating the session file on the first request, but it would not re-write the session on every request. Anyway, I'm not saying we should necessarily take this approach, just spelling out the issues and options.

 

Another option is to leave it up to the developer to save something to the session in the parent page request when the page contains multiple Ajax requests that will be accessing the session. Maybe we could provide a convenience method for this, such as session.save(), which would force saving the file even if the session is empty (such a method might have other uses, such as saving and then unlocking a session in the middle of a request).

wouldn't a simple note in the book close the deal ?

Saving dummy data to the session feels like a hack, so I think a new method like session.save() would be nice.

 

I think there's a bigger problem with sessions in cookies and the db -- they aren't locked at all, so you can get race conditions with them even once the session has initially been saved.

it's a problem "in general": either you want concurrency or you want locking..... can we even theoretically solve that problem ? I don't think so. 

I think with sessions we generally want locking (or at least the option to lock) -- in order to avoid race conditions (presumably that's why locking the file-based sessions was implemented to begin with). When storing sessions in the db, the table definition does include a "locked" field, but it is not used -- there is this commented line in session.connect():

# rows[0].update_record(locked=True)

Maybe that can be used. Not sure what to do about cookie-based sessions.

More generally, maybe we could allow more control over session locking by adding a "lock" argument to session.connect(). Those who want to take advantage of the option could globally disable automatic sessions and manually open the session via session.connect() in the app (choosing whether to do so with or without locking, depending on whether it matters for the particular action).

Anthony

[Niphlod]

I think it will be possible before write the session file
for the case there was none:

- acquire a thread lock
          (or a temp file (eg: sessiog_file_name.tmp ) to sentinel
other processes
           if running in multiprocessing, that can be checked with wsgi env)  

- do another check if the session file exists
- if yes read an merge the current request data else pass
- write the file
- release the lock
     (delete the tmp file if in multiprocessing).

way toooo much overhead for a corner-case . "merge" is hard. What if there's a session counter? you'd want to sum(). But what if my app needs an avg() ? or needs to save a string with the same key? who decides what merge should do/prefer ?

The above suggestion isn't quite going back to the old system. In the old system, the session file was created on the first request (of a new browser session) and re-written on every request (even if it didn't change) -- the latter was more a source of inefficiency than the former. The above suggestion would return to creating the session file on the first request, but it would not re-write the session on every request. Anyway, I'm not saying we should necessarily take this approach, just spelling out the issues and options.

ok, I dind't mean to be literal, but still feels too much to add to the default machinery, given that it's needed in a corner case

Saving dummy data to the session feels like a hack, so I think a new method like session.save() would be nice.

+1

 

I think with sessions we generally want locking (or at least the option to lock) -- in order to avoid race conditions (presumably that's why locking the file-based sessions was implemented to begin with). When storing sessions in the db, the table definition does include a "locked" field, but it is not used -- there is this commented line in session.connect():

the problem is that the perfect thing to do would be row locking, but not all db engines support that.
Assuming that you want to use a "flag" column to store the "locked" attribute (so it would be backend indipendant), this means that saving session would need to:
- see if record is locked
- if not, turn locking on
- commit (so you're sure that no-other would update it in the meantime) ... here you'd need to start an implicit transaction cause you can't commit what your app generated in your standing cursor, you want to commit only the "turn locking on"
- update the data
- turn locking off
- commit the locking off, the data, and all the data your cursor had ready

If the record is locked in the first step, you'd need to wait (poll) until isn't locked anymore. Contingency issues can effectively block your response until that record gets unlocked..... Seems that here too there's too much overhead and machinery to take into account.

[Anthony]

the problem is that the perfect thing to do would be row locking, but not all db engines support that.
Assuming that you want to use a "flag" column to store the "locked" attribute (so it would be backend indipendant), this means that saving session would need to:
- see if record is locked
- if not, turn locking on
- commit (so you're sure that no-other would update it in the meantime) ... here you'd need to start an implicit transaction cause you can't commit what your app generated in your standing cursor, you want to commit only the "turn locking on"
- update the data
- turn locking off
- commit the locking off, the data, and all the data your cursor had ready

Maybe we could make use of .select(..., for_update=True). I guess we'd still have to poll for lock release, but is that really complicated?

Anthony 

[Niphlod]

yep, but unfortunately select(for_update) is not available on all backends.
The critical point is then having two separate cursors to be able to commit only what's needed when is needed.
Plus, for every request changing a session variable you pass from a single update to a select and 3 updates...
I guess it's technically possible but too heavy to "suggest" it as a default.

On the "web2py 'teaches' how do do things without breaking" chapter, using session and ajax is definitely NOT a smart move.
We are discussing on how to do it with files and db, and I guess I can adapt the redis backend to work with that but what about cookies? That is definitely unresolvable ...

tl;dr: I'm not saying "don't" but lets get out priorities straight: burdening the default implementation to cover a corner-case that in any case is not recommended seems a far stretch.

[Anthony]

yep, but unfortunately select(for_update) is not available on all backends.

I think the nosql adapters are the only ones that don't support it -- so don't store sessions in a nosql db if you need to serialize session access.

 

On the "web2py 'teaches' how do do things without breaking" chapter, using session and ajax is definitely NOT a smart move.

I don't think wanting to support multiple Ajax forms on a page is outlandish in 2013. Nevertheless, the best way to support this case may not be to serialize session access. Perhaps instead we could have per page or even per session formkeys to protect against CSRF. That wouldn't protect against double submission, but we might handle that via JS on the client (i.e., disabling the submit button).

Anthony

[Niphlod]

On Tuesday, April 23, 2013 3:37:37 PM UTC+2, Anthony wrote:

yep, but unfortunately select(for_update) is not available on all backends.

I think the nosql adapters are the only ones that don't support it -- so don't store sessions in a nosql db if you need to serialize session access.

mssql too (I can't say for "elusive" support on informix, firebase, db2, cubrid and teradata).
 

 

On the "web2py 'teaches' how do do things without breaking" chapter, using session and ajax is definitely NOT a smart move.

I don't think wanting to support multiple Ajax forms on a page is outlandish in 2013. Nevertheless, the best way to support this case may not be to serialize session access. Perhaps instead we could have per page or even per session formkeys to protect against CSRF. That wouldn't protect against double submission, but we might handle that via JS on the client (i.e., disabling the submit button).

Anthony

+1 for having a token per session instead of a token per form (should be easier to implement also a wrapper library for who's working on Angular (or whatever js library) .... assuming that all posts related to angular are in fact a reflection on how many users are interested/invested in working with it)

I'm currently working on a rewrite of web2py.js that hooks up to data- elements, according to a "new" generation of A() "special methods" and form submits. Definitely +1 on the double-submit prevented by js.
 

[Anthony]

I think the nosql adapters are the only ones that don't support it -- so don't store sessions in a nosql db if you need to serialize session access.

mssql too (I can't say for "elusive" support on informix, firebase, db2, cubrid and teradata).

I was just going by which adapters set can_select_for_update = False. As I understand it, MSSQL supports the syntax, but by default it locks more than just the selected row -- so it works, but with more locking than necessary. Maybe not a good solution.

 

+1 for having a token per session instead of a token per form (should be easier to implement also a wrapper library for who's working on Angular (or whatever js library) .... assuming that all posts related to angular are in fact a reflection on how many users are interested/invested in working with it)

Yes, I was thinking of that specifically in the context of SPA's built with frameworks like Angular, etc.

 

Anthony

[黄祥]

i face the same situation, already use the formname and clear session. the form must submit twice to get it work.
e.g.
controllers/default.py
def sale_order():
    return locals()

def sale_order_form():
    form = SQLFORM.factory(
        Field('product', 'reference product',
              requires = IS_IN_DB(db((db.product.quantity > 0) ),
                                  db.product.id, db.product._format) ),
        Field('quantity', 'integer', requires = IS_NOT_EMPTY() ),
        )
    if form.process(formname = 'myform1').accepted:
        response.flash = T('Form accepted')
       
        id = int(request.vars.product)
       
        row = db(db.product.id == id).select().first()
       
        quantity = int(request.vars.quantity)
        price = int(row.selling_price)
        session.sale_order[id] = quantity, price
       
        response.js =  "jQuery('#sale_order_checkout').get(0).reload()"
    elif form.errors:
        response.flash = T('Form has errors')
    return dict(form = form)

def sale_order_checkout():
    link_callback = 'sale_order_callback'
    link_empty = 'sale_order_empty'
    return dict(session_detail = session.sale_order, link_callback = link_callback,
                link_empty = link_empty)

views/default/sale_order.html
{{extend 'layout.html'}}

{{=LOAD('default', 'sale_order_form.load', ajax = True,
        target = 'sale_order_form') }}

{{=LOAD('default', 'sale_order_checkout.load', ajax = True,
        target = 'sale_order_checkout') }}

views/default/sale_order_form.load
{{=form}}

any idea how to face this?

thanks and best regards,
stifan

[Anthony]

It's not clear this is the same problem, as you have only one form in a component, and the other component doesn't appear to save to the session. Read through the thread to repeat the diagnostics and see if you really are observing the same behavior (which implies the second component is saving something to the session). If so, the workaround is to have the parent action save some dummy data to the session in order to create a session file (which will be locked on the subsequent Ajax requests, preventing the race condition). If you have your sessions in the database or in cookies, then your out of luck, as they cannot be locked.

Anthony

[黄祥]

hm, sorry, when i open in google groups it appears in web2py-users but in gmail i received it from web2py-developers. k, back to the problems. in web2py-users first report said :

If the form has errors, the error messages are displayed after a
second submit. So, I need to click two times the submit button to
display them.

the condition i face is : i have 2 components, first is form to fill the product in session (SQLFORM.factory), the second is the checkout which is again have a form that have a function to modify the session via ajax callback.

the problem is : if the form has errors the error messages are displayed after a second submit, same for if the form is accepted, the data in form are save in session after a second submit, so, i need to click submit two times.

for session i have it on files not in database, and for dummy data, i'm not sure i'm getting it, because it's the order, i can't create the dummy data in the session, cause it can affect the total price. still about session, already check with response.toolbar() seems that the first time i land it is there with no value
e.g.

sale_order

:

after second submit

sale_order

:

1 : 2 20000

any idea why the form in first component must submit twice to response either error or accepted?

[Anthony]

Did you leave out some code, because the code you have shown for sale_order_checkout includes no form nor any writing to the session?

The idea of the dummy data is just to write some nonsense data to the session in the parent function:

def sale_order():
    session.dummy = 'dummy data'
    ...

That will force a session file to be created when the parent page is loaded. Once a session file exists, it will be locked by each Ajax request, so there will be no race condition caused by the simultaneous Ajax requests.

Note, if this is the problem, it should only appear when you are starting a new session. Once you have visited a page that saves to the session and a session file has been created, there should be no subsequent problems during the same session. If you observe the same problem happening repeatedly during the same session, then something else is going on.

Anthony

[黄祥]

already tried to write the dummy session data in (sale_order and sale_order_form) but still same (need submit twice), yet i got some bonus, traceback error said :

AttributeError: 'str' object has no attribute 'items'

controllers/default.py
def sale_order():
    session.sale_order = 'test'
    return locals()

def sale_order_form():
    session.sale_order = 'test'

    form = SQLFORM.factory(
        Field('product', 'reference product',
              requires = IS_IN_DB(db((db.product.

quantity > 0) ),
                                  db.product.id, db.product._format) ),
        Field('quantity', 'integer', requires = IS_NOT_EMPTY() ),
        )
    if form.process(formname = 'myform1').accepted:
        response.flash = T('Form accepted')
       
        id = int(request.vars.product)
       
        row = db(db.product.id == id).select().first()
       
        quantity = int(request.vars.quantity)
        price = int(row.selling_price)
        session.sale_order[id] = quantity, price
       
        response.js =  "jQuery('#sale_order_checkout').get(0).reload()"
    elif form.errors:
        response.flash = T('Form has errors')
    return dict(form = form)

views/default/sale_order_checkout.load
{{total_quantity = 0 }}
{{grand_total = 0 }}

<table class = "table table-condensed table-hover">
    <tr>
        <th>{{=T('Product') }}</th>
        <th>{{=T('Price') }}</th>
        <th>{{=T('Quantity') }}</th>
        <th>{{=T('Total Price') }}</th>
        <th>{{=T('Action') }}</td>
    </tr>
{{for id, (quantity, price) in session_detail.items():}}
{{product = db.product(id) }}
{{total_price = quantity * price}}
{{total_quantity += quantity}}
{{grand_total += total_price}}
    <tr>
        <td>
            {{=SPAN(product.name) }}
        </td>
        <td>{{=SPAN('Rp. %s' % format(price, ",d").replace(",", ".") ) }}</td>
        <td>
            <form>
                <input name = "{{='quantity_%s' % id}}" value = "{{=quantity}}"
                onkeyup = "ajax('{{=URL(link_callback, vars = dict(id = id, price = price,
                action = 'adjust_total') ) }}', ['{{='quantity_%s' % id}}'], ':eval' )" />
            </form>
        </td>
        <td>Rp. {{=SPAN('%s' % format(total_price, ",d").replace(",", "."), _id = 'total_price_%s' % id) }}</td>
        <td>
            {{=SPAN(A(I(_class = 'glyphicon glyphicon-remove-sign'), callback = URL(link_callback,
                    vars = dict(id = id, action = 'remove') ), delete = 'tr',
                    _title = 'Remove %s' % product.name) ) }}
        </td>
    </tr>
{{pass}}
    <tr>
        <td></td>
        <td>{{=B(T('Grand Total') ) }}</td>
        <td>{{=SPAN('%s' % format(total_quantity, ",d").replace(",", "."), _id = 'total_quantity') }}</td>
        <td>Rp. {{=SPAN('%s' % format(grand_total, ",d").replace(",", "."), _id = 'grand_total') }}</td>
        <td></td>
    </tr>
</table>

{{=SPAN(A(T('Empty'), _href = URL(link_empty), _title = 'Empty',
          _onclick = "javascript:return confirm('Are you sure you want to empty?')",
          _class = 'btn btn-info') ) }}

best regards,
stifan