Validation messages of extra fields in auth_user don't comply with available translations

[João Matos]

When creating extra fields for auth_user, the validation messages for those fields are always in English. They don't comply with the translations available.

This does not happen on the default fields.

I added this to db.py

auth.settings.extra_fields['auth_user'] = [Field('pagination', 'integer', default=9, label=T('Pagination'), notnull=True, required=True, requires=IS_INT_IN_RANGE(5, 51))]

When inserting an invalid value, the returned message is always

Enter an integer between 5 and 50

even when the web2py language is not English and the languages files have the correct translation available.

web2py 2.18.4
Python 3.7.1 x86
Firefox 66.0.2 x64
Windows 7 Pro x64 SP1+all updates

[Kevin Keller]

Wow that must be annoying. 

Did you find any workaround? 

[João Matos]

No, I think it is a web2py bug. I reported it in Git.

[João Matos]

I have confirmed that it is indeed a bug in web2py.
If I add

T.force('pt')



before

auth.settings.extra_fields['auth_user'] = [ Field('pagination', 'integer', default=9, label=T('Pagination'), notnull=True, required=True, requires=IS_INT_IN_RANGE(5, 51), ), ]

the validation messages all come out in Portuguese (even if I change the language to English).

So the problem is that the code for the extra_fields is not language independent.

[Anthony]

Note, this is due to the recent refactoring of all built-in validators (moved from web2py to pydal). Previously, all default error messages were translated, but that is no longer the case. Although the automatic translation was not documented, I would still consider this a break in backward compatibility (particularly given that all Auth error messages are still being translated automatically).

Anthony

[João Matos]

All my other error messages (from built-in validators) are auto-translated. Including for the default auth_user fields (eg. username). Only the extra_fields are not.

Is there a fix?

[Anthony]

Hmm, I see that the framework does add the T translator to the Validator class, so validation messages should be getting translated. There is nothing special about Auth extra_fields -- the validators you add there are no different from the validators added to other fields.

Can you give an example of code using the same validator where the message is translated outside of extra_fields but not within extra_fields?

Also, in your example, you mentioned when setting T.force('pt'), all messages came out in Portuguese, even if you "change the language to English". What do you mean by "change the language to English" there? Setting T.force('pt') should force all translations to Portuguese unless you override the T.force('pt').

Finally, are you running/sending requests to more than one application, not all of which have the same translation files?

Anthony

[João Matos]

All my grids and forms get auto-translated validation messages correctly. Here is an example

        table = db.sn_counter  # type: Table

        grid = SQLFORM.grid(
            table,
            csv=False,
            details=False,
            ondelete=on_delete,  # Grid only.
            onvalidation=on_validation,  # Form only.
            orderby=db.sn_counter.name,
            paginate=session.auth.user.pagination,
            # represent_none='',  # Grid and view form only.
            sortable=False,
        )  # type: gluon.DIV

which uses this table

db.define_table('sn_counter',
                Field('name', 'string', label=T('Name'), length=NAME_LEN, notnull=True,
                      required=True, unique=True),
                Field('last_assigned', 'integer', default=0, label=T('Last assigned'),
                      notnull=True, required=True),
                Field('canceled_on', 'datetime', label=T('Canceled on'), writable=False),
                Field('canceled_by', 'reference auth_user', label=T('Canceled by'),
                      writable=False),
                Field('cancel_approved_by', 'reference auth_user',
                      label=T('Cancel approved by'), writable=False),
                auth.signature,
                # common_filter = lambda query: db.sn_counter.is_active == True,
                format='%(name)s',
                # plural=T('sn counters'), singular=T('sn counter'),
                )

# db.sn_counter.name.requires = (IS_NOT_EMPTY(), IS_NOT_IN_DB(db, 'sn_counter.name'))
db.sn_counter.last_assigned.requires = IS_INT_IN_RANGE(0, 100000)

I give the option to the user to change the language between PT and EN. Yes, it overrides T.force('pt') with T.force('en') and update sthe page.

My original code didn't have a T.force('pt') before the extra_fields and so all my validation messages appeared correctly in PT or EN, except for the validation messages of the extra_fields (they always appeared in EN).

If I add the T.force('pt') before adding the extra_fields, all my validation messages appear correctly in PT or EN, but all the extra_fields validation messages appear only in PT.

No, I'm using just one application.

[Anthony]

Can you post a minimal app the reproduces the problem?

[João Matos]

I created a minimal app and it showed the problem.

I di some further testing and confirmed that there is a relation with the positioning of the T.force().

If the T.force() is after the extra_fields creation (which reside in db.py), the fields validation messages are always in English, even after changing the language (the changing language code resides in db1.py, but before the table defs).

If I change the language changing code to before the extra_fields. The extra_fields validation messages show up with the correct language.

Do you still want me to send you the minimal app?

[Anthony]

Yes, post the app.

[João Matos]

The app will start in Portuguese. You can change that by uncommenting the T.force() in db1.py, but it would be good to do the testing with Portuguese first to see the problem.

After registering, create a new record and in the field ùltimo atribuído (last_assigned) input -1 which will return a validation error message in Portuguese (correct behavior).

If you go to Perfil (Profile) and to the same to one of the Paginação (Pagination) fields it will return a validation error message in English (the pagination fields are the extra_fields).

If you comment the T.force() from db1.py and uncomment the T.force() that is present in db.py before the extra_fields creation and redo the 2 tests, you will see that both who in Portuguese.

I'm unable to add the app file as an attachment. It always returns an error.

So you can find it in WeTransfer in this link

https://wetransfer.com/downloads/5d1a0f5303b2a61ee27f64a9ce73ed7c20190406180555/90183abbd7bf1cbaab5ab1011cbdafed20190406180555/81e5a8

[Anthony]

The problem is unique to the IS_*_IN_RANGE validators, all of which create a translated error message at the time they are instantiated rather than at validation time. For example, here is the init code for IS_INT_IN_RANGE: https://github.com/web2py/pydal/blob/e973f27b69a8499e4e30e5663f3fba74bf87364c/pydal/validators.py#L817.

So, you need to set the translation language before defining one of these validators. Note, this has nothing to do with Auth extra_fields, just with the point in the code at which the validators are defined.

I don't see why the error message logic cannot be moved from the __init__ method to the validate method in these range validators, which would push the translation to validation time, as with all the other validators. So, feel free to submit an issue about this in the pydal repo and refer to this post (or better yet, make a pull request).

Anthony

[João Matos]

Thanks Anthony. I will open an issue (I still haven't enough know how to make a PR). I hope some time in the future.

I don't want to abuse your help, but I have another issue that is viewable with the minimal app I sent you.

If you create 2 records with different dates, and try to change the grid's sort order you will notice that the URL changes, the page reloads but the grid doesn't change.

I checked the db stats SQL statements and they stay the same (they don't change the ORDER).

I think there is a bug with the sorting of date columns when they are the predefined orderby. Can you try it and give me your input.

Thanks.

[Massimo Di Pierro]

fixed in master. Thanks for reporting and diagnosis.

[Anthony]

On Saturday, April 6, 2019 at 3:41:50 PM UTC-4, João Matos wrote:

Thanks Anthony. I will open an issue (I still haven't enough know how to make a PR). I hope some time in the future.

I don't want to abuse your help, but I have another issue that is viewable with the minimal app I sent you.

If you create 2 records with different dates, and try to change the grid's sort order you will notice that the URL changes, the page reloads but the grid doesn't change.

I checked the db stats SQL statements and they stay the same (they don't change the ORDER).

I think there is a bug with the sorting of date columns when they are the predefined orderby. Can you try it and give me your input.

Definitely a bug -- feel free to report on Github. May have to do with the fact that when clicking to sort, the grid code inverts the order for date/time fields, but does not appear to do so when a date/time field is specified in the orderby argument.

Anthony

[João Matos]

Thank you for the feedback Anthony.

That was my thinking but I was hoping to be wrong.

Any idea where in the web2py code is the grid sorting? I would like to take a look.

[João Matos]

I think I found the problem (I don't have the solution).

In gluon\sqlhtml.py there is this function

        def fix_orderby(orderby):
            if not auto_pagination:
                return orderby
            # enforce always an ORDER clause to avoid
            # pagination errors. field_id is needed anyhow,
            # is unique and usually indexed. See issue #679
            if not orderby:
                orderby = field_id
            elif isinstance(orderby, list):
                orderby = reduce(lambda a,b: a|b, orderby)
            elif isinstance(orderby, Field) and orderby is not field_id:
                print('inside2')
                # here we're with an ASC order on a field stored as orderby
                orderby = orderby | field_id
            elif (isinstance(orderby, Expression) and
                  orderby.first and orderby.first is not field_id):
                print('inside3')
                # here we're with a DESC order on a field stored as orderby.first
                orderby = orderby | field_id
            return orderby

where I added the 2 prints.

A field that works correct shows inside2 on the first run and inside3 on the second.

The date field always shows inside2.

Any ideas?

Do you use Telegram? There is an online web2py international group. It would be great if you and others web2py core devs joined.

domingo, 7 de Abril de 2019 às 19:22:34 UTC+1, Anthony escreveu:

[João Matos]

I think I found the source of the problem.

After following the function in the previous message, I arrived at this code inside the linsert function (line 2779) .

                exception = sort_field.type in ('date', 'datetime', 'time')
                if exception:
                    desc_icon, asc_icon = sorter_icons
                    orderby = (order[:1] == '~' and sort_field) or ~sort_field
                else:
                    orderby = (order[:1] == '~' and ~sort_field) or sort_field

If I change it to

                exception = sort_field.type in ('date', 'datetime', 'time')
                # if exception:
                #    desc_icon, asc_icon = sorter_icons
                #    orderby = (order[:1] == '~' and sort_field) or ~sort_field
                #else:
                orderby = (order[:1] == '~' and ~sort_field) or sort_field

Now the date sorting works fine.

There is also another code segment in line 2641 with the same purpose that may need the same change, but that segment isn't run with my application, only the segment at line 2779.

[Anthony]

Yes, as noted, that code inverts the sort order for date/time fields when the field is clicked, but there is no inversion when using the orderby argument. I'm not sure why the order is inverted for such fields. Maybe bring it up in the developers group.

[João Matos]

Thanks Anthony. Already updated the Github issue and created a post in the developers.