Problem with form and disappearing field data

[Jim Steil]

I have a form with a textarea control.  If I paste in the following data and click submit, the data disappears and I get the 'Enter a value' message below the field (requires=IS_NOT_EMPTY() is set on the field.

The text is:

decided we’d have a blah blah blah they don’t need

It is text that I cut from a Word document and pasted in to my form field.  Notice the apostrophe's are not the same apostrophe character generated when I type on here

ex - '  is not equal to ’

In form.py, line 784 is the code:

original_value = post_vars.get(field.name)

Checking the debugger (pycharm) I see that before this line post_vars has a key of 'activity' (my field name) and the value I can see is the text string above.  

But, after line 784 executes, original_value is equal to None.

My question - is there a way to reliably escape those funky apostrophes (and other special characters) so that the field value is not lost?

Obviously I'm not very strong on encoding/decoding/internationalization and could use some pointers.

-Jim

[CarlosDB]

I think this problem is related to these 2 threads:

https://groups.google.com/g/py4web/c/U9mkmvl0g2I
https://groups.google.com/g/py4web/c/fuI6I_7LbSk/m/AZYm7kmqAAAJ

and with this commit:

https://github.com/web2py/py4web/commit/46cdf43bd86ad4598dd08fe34bd1aa37c898a628

Commenting out the added lines in the commit form works fine, but Grid search forms don't work with non-us-ascii characters

Let's see if we all manage to solve it...

Carlos.

[Jim Steil]

Thanks Carlos

I'll look at those today and see if I can make some sense of it.

-Jim

[Jacinto Parga]

I have seen the same issue. 

This happens when form or grid are used with non ascii characters: 

1. Text area data are not saved in db, they are even deleted if there are some non asccii characters (example á, ñ, ü...)
2. If a field string with a IS_NOT_EMPTY() validator is used then it rise an error 'Enter a value'

But

I can add non ascii characters when the dashboard db manager is used. Dashboard database manager works fine.

ombott   0.0.14
py4web  1.20220807.1
pydal       20220807.1

[Jacinto Parga]

This worked for me too, but I don't if there are side effects

https://discord.com/channels/836984759535534120/836984759535534125/1006343338292682912

Change line 143 in ombott/request/helpers.py

recode_unicode = False

[Massimo]

The proposed suggestion seems correct. I think it is easier to monkeypatch ombott in py4web that make a change to it.

I made the change in master and it seems to work for me. Please double check it. If no side effects we can release this.

Massimo

[Val K]

Already fixed in ombott 0.1.0

Here is bottle discussion
https://github.com/bottlepy/bottle/pull/1068

вторник, 20 сентября 2022 г. в 08:23:33 UTC+3, Massimo:

[Massimo DiPierro]

oops. missed that thank you Val.

--
You received this message because you are subscribed to the Google Groups "py4web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py4web+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/552bd3e1-7aa5-40bf-b737-f2db371cdf8an%40googlegroups.com.

[Massimo]

Unfortunately the patch still appears necessary with ombott 0.1.0.

[Jim Steil]

I have confirmed that this fixed my issue.

-Jim

[Jacinto Parga]

Mine too. It works, but you're right both changes are necessary: ombott 0.1.0 and py4web patch

[Val K]

which patch?

вторник, 20 сентября 2022 г. в 19:28:28 UTC+3, jpa...@gmail.com:

[Jim Steil]

Mine issue was fixed with the py4web patch.  I tested without the ombott patch and my issue was resolved.  I added the ombott patch and that (with the py4web patch) works too.

-Jim

[Val K]

if you mean recode_unicode = False in helpers.py  - I'm sure, this patch breaks   'application/x-www-form-urlencoded' forms 

вторник, 20 сентября 2022 г. в 19:52:10 UTC+3, Val K:

[Christian Varas]

I have  recode_unicode = False in helpers.py  for a month or more in a production app and I haven't experienced any undesired  effect after setting it to False.

Cheers.

Chris.

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/ad3c7823-324e-4bd5-83f1-3ac492832cacn%40googlegroups.com.

[Jacinto Parga]

No, I'm not talking about recode_unicode=False. 

I know that was not the solution but a tricky workaround.

 I am talking about py4web monkey patch, with the new version of ombott you have deployed .

[Massimo]

The monkey patch only sets recode_unicode=False. It works for me. For now I have released 20220920.1 including it.

[CarlosDB]

I’m very sorry :-( but I think that with this last modification we have returned to what I think is the origin of the problem.

Adding recode_unicode=False brings me back to problems with search_queries in Grid. With non-us-ascii characters there are encoded problems.

Removing the following, Grid works fine but Form doesn’t work fine.
line 126 py4web/core.py
FormsDict.recode_unicode = False

Removing this too, Form works fine but Grid doesn’t.

lineas 122, 123, 124, 125 de py4web/core.py
# monkey patching bottle/ombott
request.query.__class__.get, request.query.__class__.getraw = (
    request.query.__class__.getunicode,
    request.query.__class__.get,
)

If Grid works Form does not work well or vice versa.

Here is an example that is not working well with the latest modifications.

from py4web import action, request, abort, redirect, URL, Field
from .common import db, session
from py4web.utils.grid import Grid

db.define_table(
    't_test_grid',
    Field('f_test1', 'string', length=30),
)
db.commit()

if not db(db.t_test_grid).select():
    db.t_test_grid.insert(f_test1='niña')
    db.t_test_grid.insert(f_test1='niño')
    db.t_test_grid.insert(f_test1='camión')
    db.t_test_grid.insert(f_test1='cigüeña')

db.commit()

@action('grid_test')
@action('grid_test/<path:path>')
@action.uses('grid_test.html', db)
def grid_test(path=None):

    t_test_grid = db.t_test_grid
    grid = Grid(
        path=path,
        query=t_test_grid,
        search_queries=[
            ('Test', lambda value: t_test_grid.f_test1.contains(value)),
        ],
        rows_per_page=10,
    )

    return dict(grid=grid)

I don’t know if it is well understood.

I think the problem with Form started with this thread : https://groups.google.com/g/py4web/c/fuI6I_7LbSk/m/AZYm7kmqAAAJ

Carlos.

​

[Massimo DiPierro]

Form should be the gold standard. Grid should just use form without transformation. will get to the bottom of it tonight. would help a lot to have a collection of examples to try in one place and turn that into a test.

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/236efb0e-07c0-4935-bcc5-72a5546f5cf5n%40googlegroups.com.

[Massimo]

I tested forms with the latest code released yesterday. I could not reproduce any problem. I did not try grid but I checked the code and it use Form without additional parsing therefore I cannot see any problem.

Please make sure you pip install -r requirements.txt and run the latest py4web. If you have a problem please, post an example to reproduce.

Massimo

[Val K]

Grid uses request.query  (request.GET is the same, just an alias)  which is  'application/x-www-form-urlencoded'.

With monkey patch, all forms in 'application/x-www-form-urlencoded' will not work properly with unicode. 

 

четверг, 22 сентября 2022 г. в 07:48:03 UTC+3, Massimo:

[Val K]

@Massimo

https://github.com/valq7711/ombott/tree/forms-factory

made it configurable, so you can request._forms_factory = MyFormsClass

четверг, 22 сентября 2022 г. в 11:32:20 UTC+3, Val K:

[Massimo DiPierro]

thank you. will review this asap. i would still like an example of something that does not currently work well in grid.

py4web forms are multipart/form-data https://github.com/web2py/py4web/blob/master/py4web/utils/form.py#L295 whether or not used in grid.

grid uses request.query but only for things like page index which should not contain Unicode. 

is the problem with search in grid which may use GET forms? i did not try GET forms.

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/0469fb85-2833-47b3-a5cd-49cd0e3847bcn%40googlegroups.com.

[Val K]

typo
request.__class__._forms_factory = MyFormsClass

четверг, 22 сентября 2022 г. в 12:21:16 UTC+3, Val K:

[Val K]

as I understand there is search-field in grid which can be unicode

четверг, 22 сентября 2022 г. в 12:31:00 UTC+3, Val K:

[Val K]

> is the problem with search in grid which may use GET forms? i did not try GET forms.

yeah,  GET forms should be recoded

четверг, 22 сентября 2022 г. в 12:32:16 UTC+3, Val K:

[Val K]

also consider that ombott sets `recode_unicode` attr  on instance of `_forms_factory`

so you can use that to adjust your custom class logic

четверг, 22 сентября 2022 г. в 12:38:02 UTC+3, Val K:

[CarlosDB]

Here is a quick example to reproduce the problem.

Currently Form works fine and the search in Grid does not work well.

controllers.py

from py4web import action, request, abort, redirect, URL, Field

from py4web.utils.form import Form
from py4web.utils.grid import Grid
from yatl.helpers import A

from .common import db, session

db.define_table(
    't_test_grid',
    Field('f_test1', 'string', length=30),
)

db.define_table(
    't_test_form',
    Field('f_test1', 'text', default=''),

)

if not db(db.t_test_grid).select():
    db.t_test_grid.insert(f_test1='niña')
    db.t_test_grid.insert(f_test1='niño')
    db.t_test_grid.insert(f_test1='camión')
    db.t_test_grid.insert(f_test1='cigüeña')

db.commit()


@action('grid_test')
@action('grid_test/<path:path>')

@action.uses('test.html', db)

def grid_test(path=None):

    t_test_grid = db.t_test_grid
    grid = Grid(
        path=path,
        query=t_test_grid,
        search_queries=[
            ('Test', lambda value: t_test_grid.f_test1.contains(value)),
        ],
        rows_per_page=10,
    )

    return dict(stuff=grid, link=A('form', _href=URL('form_test')))


@action('form_test', method=['GET', 'POST'])
@action.uses('test.html', db)
def form_test():

    t_test_form = db.t_test_form

    if not db(t_test_form).select():
        t_test_form.insert(f_test1='áéíóú ñÑ Üü')

    record = db(
        t_test_form
    ).select().last()

    form = Form(t_test_form, record)
    if form.accepted:
        redirect(URL('form_test'))

    return dict(stuff=form, link=A('grid', _href=URL('grid_test')))

test.html

[[extend 'layout.html']]
[[=link]]
[[=stuff]]

Carlos

​

[Massimo]

I think the source of problems is

https://github.com/valq7711/ombott/blob/main/ombott/request/helpers.py#L12

The urlunquote function is used to parse the query string parameters and assumes they are latin1 even if they are unicode.

This happens before the parsed value are stored in FormsDict. This affects GET forms only (and indirectly grid).

I think that line should be removed or, at least, there should be a mechanism to monkey patch it. Currently there is not because the ombott.request module is shadowed by the request object.

Massimo

[Massimo]

I found a way to monkey patch this.

https://github.com/web2py/py4web/commit/413bd7eafc5b45aeefb129d1885ca25a7f95d9b4

It is fragile as ombott may change.

Why does ombott need to parse query string variables as latin1 and not unicode? Maybe https://github.com/valq7711/ombott/blob/main/ombott/request/helpers.py#L12 can be removed or made configurable.

[Val K]

latin1 is the legacy of bottle

i did some research and it seems that there is no common convention about encoding between frameworks, some use latin1 as more robust, some use utf-8 as more convenient. 

So lets migrate to utf8

суббота, 24 сентября 2022 г. в 09:53:33 UTC+3, Massimo:

[Massimo DiPierro]

👍

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/28a86c40-90c4-4f58-b5c1-0962fb748784n%40googlegroups.com.

[Val K]

test please

https://github.com/valq7711/ombott/tree/utf8-for-all-forms

суббота, 24 сентября 2022 г. в 23:24:17 UTC+3, Massimo:

[Jacinto Parga]

Hello.

Yes it works in first test. Tested addition, edition and search in http://127.0.0.1:8000/showcase/example_html_grid

[Jacinto Parga]

Second test. I have upgraded py4web with pip and forms fail in edition and adition. 

Search in grid works according to the last changes.

Tested in:

 http://127.0.0.1:8000/showcase/example_html_grid

http://127.0.0.1:8000/showcase/create_form

ombott 0.1.0

py4web 1.20220925.1
pydal 20220814.1

Looking for the problem

[Jacinto Parga]

The failure comes from form.py last commit:

https://github.com/web2py/py4web/commit/20e8b0a7cf2002e73b2585bd3244ba06379d1b86

It avoids register creation or edition in forms.

Don't know if it is the only one effect.

[Massimo DiPierro]

will undo it for now.

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/937793b9-110c-48cf-bcb0-3675f6048243n%40googlegroups.com.

[Val K]

Just let me know  if it is ready for pushing to pypi 

воскресенье, 25 сентября 2022 г. в 18:36:43 UTC+3, Massimo:

[Val K]

I also want to point out that, formally, this breaking change, as I've removed FormsDict methods that are no longer needed. So is there anyone who uses these methods? e.g. request.forms.getunicode or request.forms.decode (the same applies to request.query, request.POST, request.files)

воскресенье, 25 сентября 2022 г. в 22:24:22 UTC+3, Val K:

[Kevin Keller]

I used request.POST just last week. 

What am I supposed to use?

Request.json?

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/65b91715-cd59-4a5e-8245-11d577920155n%40googlegroups.com.

[Val K]

It is only about methods: decode and getunicode. - do you have request.POST.getunicode or request.POST.decode somewhere in your code?

воскресенье, 25 сентября 2022 г. в 22:44:01 UTC+3, kell...@gmail.com:

[Val K]

Instead `getunicode` use just `get`

`decode` no longer needed as all that stuff is alredy decoded. 

There maybe only issue: `decode` returns copy - so I will add copy method to return instance of FormsDict with dot-access supported. 

So instead `decode` use `copy` if you need a copy

воскресенье, 25 сентября 2022 г. в 22:47:59 UTC+3, Val K:

[Val K]

@Massimo  updated (added FormsDict.copy())

https://github.com/valq7711/ombott/tree/utf8-for-all-forms  

воскресенье, 25 сентября 2022 г. в 23:08:59 UTC+3, Val K:

[Massimo DiPierro]

all looks good. i would prefer if __getattr__ where to have default None instead of "".

To view this discussion on the web visit https://groups.google.com/d/msgid/py4web/8f46f680-3e50-4a28-90df-97d6f59ddcfen%40googlegroups.com.

[Massimo DiPierro]

one more thing. i would rename the request folder because it is shadowed by the request object. make monkey patching for testing hard.

[Val K]

понедельник, 26 сентября 2022 г. в 00:33:54 UTC+3, Massimo:

[Jacinto Parga]

.../python3.10/site-packages/py4web/core.py", line 55, in <module>

from ombott.request.helpers import FormsDict
ModuleNotFoundError: No module named 'ombott.request'

ombott-1.0.0

Py4web: 1.20220926.1

[Jacinto Parga]

In core.py  https://github.com/web2py/py4web/blob/master/py4web/core.py#L55

# Third party modules
import ombott as bottle
from ombott.request_pkg.helpers import FormsDict

[CarlosDB]

Now both Form and Grid are working perfectly for me!!!

Thanks!