Old DAL bug (?) ValueError: invalid literal for int() with base 10: 'AssetTypes\\Character.png'

[Arnon Marcus]

We are using an old version of web2py in production (1.89.5), and are having issues with the DAL:

    if limit: return db().select(db[table].ALL,orderby=db[table][orderby],limitby=(0,limit))
  File "C:\wamp\www\web2py_1_89_5\gluon\sql.py", line 3307, in select
    return self.parse(db, rows, self.colnames, SetClass=Set)
  File "C:\wamp\www\web2py_1_89_5\gluon\sql.py", line 3353, in parse
    colset[fieldname] = rid = Reference(value)
ValueError: invalid literal for int() with base 10: 'AssetTypes\\Character.png'

The relevant section we pinpointed is this:

    @staticmethod

    def parse(db, rows, colnames, blob_decode=True, SetClass=None):

         .......

                    if not '.' in referee:

                        colset[fieldname] = rid = Reference(value)

                        (rid._table, rid._record) = (db[referee], None)

                    else: ### reference not by id

                        colset[fieldname] = value

                elif field_type == 'boolean':

We suspect that what is happening is a shift in the field-names list, that is causing the DAL to treat a string field as an id field.

We have some "suspected culprits" for thinking this - we suspect that there might be a discrepency between the order/length of the data-fields being returned, and that order/length in the field-names being passes,  which leads us back to suspect that something might be wrong with the "colnames" being passes in to "parse" from "select", but that is being set in "_select":

    def _select(self, *fields, **attributes):

       ......

        # ## if not fields specified take them all from the requested tables

        if not fields:

            fields = [self._db[table].ALL for table in self._tables]

        sql_f = ', '.join([str(f) for f in fields])

        tablenames = parse_tablenames(self.sql_w + ' ' + sql_f)

        if len(tablenames) < 1:

            raise SyntaxError, 'Set: no tables selected'

        w2p_tablenames = [ t for t in tablenames if isinstance(self._db[t],Table) ]

        self.colnames = [c.strip() for c in sql_f.split(', ')]

        if self.sql_w:

            sql_w = ' WHERE ' + self.sql_w

       ....

    def select(self, *fields, **attributes):

        """

        Always returns a Rows object, even if it may be empty

        """    

     ...

        return self.parse(db, rows, self.colnames, SetClass=Set)

    @staticmethod

    def parse(db, rows, colnames, blob_decode=True, SetClass=None):

        virtualtables = []

        new_rows = []

        for (i,row) in enumerate(rows):

            new_row = Row()

            for j,colname in enumerate(colnames):

                value = row[j]

I checked to see what has been changed regarding this in the current version:

    def _select(self, query, fields, attributes):

        .....

        if len(tablenames) < 1:

            raise SyntaxError('Set: no tables selected')

        sql_f = ', '.join(map(self.expand, fields))

        self._colnames = [c.strip() for c in sql_f.split(', ')]

        if query:

            sql_w = ' WHERE ' + self.expand(query)

....

    def _select_aux(self,sql,fields,attributes):

       ....

        return processor(rows,fields,self._colnames,cacheable=cacheable)

    def select(self, query, fields, attributes):

        """

        Always returns a Rows object, possibly empty.

       .....

            return self._select_aux(sql,fields,attributes)

...

    def parse(self, rows, fields, colnames, blob_decode=True,

               .....

                    value = self.parse_value(value,ft,blob_decode)

                    if field.filter_out:

                        value = field.filter_out(value)

                    colset[fieldname] = value

It seems a much "safer" implementation.

Now, we have scheduled an upgrade of web2py in our road-map, but that will take some time...

In the mean-time, I was wondering weather we could use some/all of the new dal.py + sql.py to replace our current web2py's ones.

I was wondering if that would be safe, or if we could get into unforeseen trouble doing that...

I had seen Massimo saying that the dal.py is now web-framework agnostic

The question is weather the "old" one is also that... In a way, does this mean that dal.py versions are inter-changeable across web2py versions, at least in this case...

What do you say? 

 

[Arnon Marcus]

Can somebody pleas respond?

We are quite desperate here about this...

[Massimo Di Pierro]

That is a very old version of web2py. It is hard to support something so old. you should be able to upgrade and nothing should break.

The only case I have seen something like this is when a string field contained data (strings) and was converted in a reference field using sqlite. Or a "list:string" converted in a "list:reference".

[Arnon Marcus]

On Thursday, April 18, 2013 4:53:30 PM UTC+3, Massimo Di Pierro wrote:

That is a very old version of web2py. It is hard to support something so old. you should be able to upgrade and nothing should break.

The only case I have seen something like this is when a string field contained data (strings) and was converted in a reference field using sqlite. Or a "list:string" converted in a "list:reference".

I was not asking for support - I don't want this bug to be "debugged" - I am asking something else:

"Can I replace the dal.py+sql.py of this version with the dal.py of the new versions, without issues, or are there dependencies I should be aware of?"

I am asking from the context of seeing your presentation in which you said that the new dal.py is framework-agnostic, so I wanted to ask - is it also "web2py-versions" agnostic? At least in this case?

[Massimo Di Pierro]

Nothing depends on sql.py. You can do what you say.

[Arnon Marcus]

Ok, good to here.

Any changes to the dal.py that might reflect upon the way our current db.py is interpreted?

We are using PostgreSQL 9.1 x64 with the latest psycopg.

Has anything changed, say, in "group_by", or "order_by" or any of the data-types, or any of the "require"s ?

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/G0tx-Gg_uD4/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.