FreeTDS returns MS SQL money type as Python float, not Decimal

483 views
Skip to first unread message

mwolfe02

unread,
Jul 30, 2010, 9:35:11 AM7/30/10
to web2py-users
For background, refer to related questions on SO:
http://stackoverflow.com/questions/3364699/
http://stackoverflow.com/questions/3371795/

I'm developing my app on Windows, but will be deploying it on a Linux
server. The following lines are from the top of my db.py:

if os.name=='posix': #Ubuntu
db = DAL('mssql://user:p...@10.0.8.30/TaxDB?DRIVER={FreeTDS}',
migrate=False)
elif os.name=='nt': #Windows
db = DAL('mssql://us...@127.0.0.1/TaxDB_Dev?
Trusted_Connection=Yes', migrate=False)

The problem I am running into is that on Windows, my MS SQL money
fields are returned as Python Decimals. However, on Linux, those same
fields are returned as floats. I think the issue is with FreeTDS.
The above lines are the _only_ lines that are different between my
production and development versions. Obviously, the environments
themselves are vastly different.

Is there an alternative to FreeTDS to connect to SQL Server from
Linux? Do I need to provide some additional info in my connection
string? Am I just stuck because I'm accessing SQL Server from Linux?

Thanks in advance for any insight.

Michele Comitini

unread,
Jul 30, 2010, 9:42:37 AM7/30/10
to web...@googlegroups.com
you can try with mxodbc http://www.egenix.com/products/python/mxODBC/

2010/7/30 mwolfe02 <michael.jo...@gmail.com>

mwolfe02

unread,
Jul 30, 2010, 10:07:59 AM7/30/10
to web2py-users
Ideally, I'd like to avoid having to purchase a commercial product.
Plus, I don't think it would work. mxODBC appears to be a replacement
for pyodbc. It looks like mxODBC uses FreeTDS just like pyodbc. The
problem appears to be with FreeTDS, so mxODBC doesn't look like it
would be a fix for my particular problem. In fact, mxODBC says as
much itself:

(From the mxODBC change log, Changes from 2.0.7 to 3.0.0:
http://www.egenix.com/products/python/mxODBC/changelog.html)

"Added new experimental subpackage mx.ODBC.FreeTDS which allows
connecting to MS SQL Server from Unix. This is still far from even
being called alpha due to the current lack of implementation support
of many important ODBC APIs in the FreeTDS ODBC driver."

The problem still seems to be FreeTDS.

On Jul 30, 9:42 am, Michele Comitini <michele.comit...@gmail.com>
wrote:
> you can try with mxodbchttp://www.egenix.com/products/python/mxODBC/
>
> 2010/7/30 mwolfe02 <michael.joseph.wo...@gmail.com>
>
>
>
> > For background, refer to related questions on SO:
> >http://stackoverflow.com/questions/3364699/
> >http://stackoverflow.com/questions/3371795/
>
> > I'm developing my app on Windows, but will be deploying it on a Linux
> > server.  The following lines are from the top of my db.py:
>
> > if os.name=='posix':  #Ubuntu
> >    db = DAL('mssql://user:p...@10.0.8.30/TaxDB?DRIVER={FreeTDS}<http://user:p...@10.0.8.30/TaxDB?DRIVER=%7BFreeTDS%7D>
> > ',
> > migrate=False)
> > elif os.name=='nt':   #Windows
> >    db = DAL('mssql://u...@127.0.0.1/TaxDB_Dev?

mdipierro

unread,
Jul 30, 2010, 10:32:29 AM7/30/10
to web2py-users
Are they both connecting to the same database?
Can you provide an example of how you insert and how you extract data?

This is to understand the issue better. In fact
gluon/sql.py contains the following code:


...
elif field.type[:7] == 'decimal' and value != None:
decimals = [int(x) for x in
field.type[8:-1].split(',')][-1]
if field._db._dbname == 'sqlite':
value = ('%.'+str(decimals)+'f') % value
if not isinstance(value,decimal.Decimal):
value = decimal.Decimal(str(value))
...

so technically even if the database where to return float (wrong),
web2py should still return decimal. Is this the case for you?



On Jul 30, 8:35 am, mwolfe02 <michael.joseph.wo...@gmail.com> wrote:
> For background, refer to related questions on SO:http://stackoverflow.com/questions/3364699/http://stackoverflow.com/questions/3371795/
>
> I'm developing my app on Windows, but will be deploying it on a Linux
> server.  The following lines are from the top of my db.py:
>
> if os.name=='posix':  #Ubuntu
>     db = DAL('mssql://user:p...@10.0.8.30/TaxDB?DRIVER={FreeTDS}',
> migrate=False)
> elif os.name=='nt':   #Windows
>     db = DAL('mssql://u...@127.0.0.1/TaxDB_Dev?

mwolfe02

unread,
Jul 30, 2010, 10:43:26 AM7/30/10
to web2py-users
I'm using .executesql. I added a line to sql.py for debugging
purposes. It returns the second column of the first row of results (a
SUM of a SQL Server money column):

diff --git a/gluon/sql.py b/gluon/sql.py
--- a/gluon/sql.py
+++ b/gluon/sql.py
@@ -1389,6 +1389,7 @@
data = self._cursor.fetchall()
# convert the list for each row into a dictionary so it's
# easier to work with. row['field_name'] rather than
row[0]
+ raise RuntimeError(type(data[0][1]))
return [dict(zip(fields,row)) for row in data]
# see if any results returned from database
try:


When I run on Windows, I get:

RuntimeError: <class 'decimal.Decimal'>


When I run on Linux, I get:

RuntimeError: <type 'float'>




On Jul 30, 10:32 am, mdipierro <mdipie...@cs.depaul.edu> wrote:
> Are they both connecting to the same database?
> Can you provide an example of how you insert and how you extract data?
>
> This is to understand the issue better. In fact
> gluon/sql.py contains the following code:
>
> ...
> elif field.type[:7] == 'decimal' and value != None:
>                     decimals = [int(x) for x in
> field.type[8:-1].split(',')][-1]
>                     if field._db._dbname == 'sqlite':
>                         value = ('%.'+str(decimals)+'f') % value
>                     if not isinstance(value,decimal.Decimal):
>                         value = decimal.Decimal(str(value))
> ...
>
> so technically even if the database where to return float (wrong),
> web2py should still return decimal. Is this the case for you?
>
> On Jul 30, 8:35 am, mwolfe02 <michael.joseph.wo...@gmail.com> wrote:
>
>
>
> > For background, refer to related questions on SO:http://stackoverflow.com/questions/3364699/http://stackoverflow.com/q...

mwolfe02

unread,
Jul 30, 2010, 10:49:22 AM7/30/10
to web2py-users
Massimo, I realize I didn't address your question about connecting to
the same database.

In my original post it did show that I was connecting to different
databases (although the dev database is just a restore of the
production database backup). However, I changed those lines to
connect to the exact same database with the exact same credentials and
got the same results:

if os.name=='posix': #Ubuntu
db = DAL('mssql://user:p...@10.0.8.30/TaxDB?DRIVER={FreeTDS}',
migrate=False)
elif os.name=='nt': #Windows
db = DAL('mssql://user:p...@10.0.8.30/TaxDB', migrate=False)

mdipierro

unread,
Jul 30, 2010, 10:50:41 AM7/30/10
to web2py-users
can you insert with db.table.insert(decimalfield=decimal(...))?

Michele Comitini

unread,
Jul 30, 2010, 11:08:54 AM7/30/10
to web...@googlegroups.com


2010/7/30 mwolfe02 <michael.jo...@gmail.com>

Ideally, I'd like to avoid having to purchase a commercial product.

Alas you are using a commercial product.
indeed this is what you should look at: http://www.egenix.com/products/python/mxODBCConnect/
 
That said it could be just a problem with the python adapter of freetds pymssql.

Michael Wolfe

unread,
Jul 30, 2010, 11:19:46 AM7/30/10
to web...@googlegroups.com
Touché! Point taken on the commercial software.... :)

As a fall-back plan, there is an acceptable workaround. I can
explicitly convert the floats I receive into Decimals. Based on
Massimo's post, it looks like this will only be an issue for money
fields returned from a .executesql statement. For all other queries
where I can leverage the DAL, I think web2py will handle things just
fine.

I'm going to post a message to the FreeTDS message board and see what
I can learn.

mwolfe02

unread,
Jul 30, 2010, 11:52:36 AM7/30/10
to web2py-users
Massimo,

Per your first post:

If I run the query using DAL, I get a Decimal type back for both
environments. Of course, there is no way for web2py to make that
coercion for fields returned from .executesql. Since that minimizes
the pain greatly for me, I think I will just go ahead with explicitly
converting to Decimals when working with .executesql.

If I learn anything with reference to FreeTDS I'll be sure to come
back and post that here.

Thanks to you and Michele for the help.

On Jul 30, 11:19 am, Michael Wolfe <michael.joseph.wo...@gmail.com>
wrote:
> Touché!  Point taken on the commercial software.... :)
>
> As a fall-back plan, there is an acceptable workaround.  I can
> explicitly convert the floats I receive into Decimals.  Based on
> Massimo's post, it looks like this will only be an issue for money
> fields returned from a .executesql statement.  For all other queries
> where I can leverage the DAL, I think web2py will handle things just
> fine.
>
> I'm going to post a message to the FreeTDS message board and see what
> I can learn.
>
> On Fri, Jul 30, 2010 at 11:08 AM, Michele Comitini
>
>
>
> <michele.comit...@gmail.com> wrote:
>
> > 2010/7/30 mwolfe02 <michael.joseph.wo...@gmail.com>

mwolfe02

unread,
Aug 5, 2010, 4:20:14 PM8/5/10
to web2py-users
Turns out it was a bug in FreeTDS... See here:
http://lists.ibiblio.org/pipermail/freetds/2010q3/026214.html

This is fixed in the CVS head of FreeTDS as of Aug 4, 2010. Until
this change makes it into a release, the nightly build is available
here:
ftp://ibiblio.unc.edu/pub/Linux/ALPHA/freetds/current/freetds-current.tgz

I built this and confirmed that it works correctly with web2py. I
built it with the following commands:

cd ~
wget ftp://ftp.ibiblio.org/pub/Linux/ALPHA/freetds/current/freetds-current.tgz
tar xzvf freetds-stable.tgz
cd freetds-0.83.dev.20100804/ (Your version of TDS will be
different)
./configure --prefix=/usr --sysconfdir=/etc --with-tdsver=8.0 --enable-
msdblib
sudo make && sudo make install clean

However, the option to set the TDS Version to 8.0 did not seem to take
for some reason (maybe I should have used 'sudo' to execute the
configure command as root). I fought with that for a long time before
realizing I could pass that setting in the web2py connection string:

db = DAL('mssql://user:p...@dbserver.domain.com/DBName?
DRIVER=FreeTDS;TDS_Version=8.0')

Hopefully this information will be useful to someone.


On Jul 30, 11:52 am, mwolfe02 <michael.joseph.wo...@gmail.com> wrote:
> Massimo,
>
> Per your first post:
>
> If I run the query using DAL, I get a Decimal type back for both
> environments.  Of course, there is no way for web2py to make that
> coercion for fields returned from .executesql.  Since that minimizes
> the pain greatly for me, I think I will just go ahead with explicitly
> converting to Decimals when working with .executesql.
>
> If I learn anything with reference toFreeTDSI'll be sure to come
> back and post that here.
>
> Thanks to you and Michele for the help.
>
> On Jul 30, 11:19 am, Michael Wolfe <michael.joseph.wo...@gmail.com>
> wrote:
>
>
>
> > Touché!  Point taken on the commercial software.... :)
>
> > As a fall-back plan, there is an acceptable workaround.  I can
> > explicitly convert the floats I receive into Decimals.  Based on
> > Massimo's post, it looks like this will only be an issue for money
> > fields returned from a .executesql statement.  For all other queries
> > where I can leverage the DAL, I think web2py will handle things just
> > fine.
>
> > I'm going to post a message to theFreeTDSmessage board and see what
> > I can learn.
>
> > On Fri, Jul 30, 2010 at 11:08 AM, Michele Comitini
>
> > <michele.comit...@gmail.com> wrote:
>
> > > 2010/7/30 mwolfe02 <michael.joseph.wo...@gmail.com>
>
> > >> Ideally, I'd like to avoid having to purchase a commercial product.
>
> > > Alas you are using a commercial product.
> > > indeed this is what you should look at:
> > >http://www.egenix.com/products/python/mxODBCConnect/
>
> > > That said it could be just a problem with the python adapter offreetds
> > > pymssql.
>
> > >> Plus, I don't think it would work.  mxODBC appears to be a replacement
> > >> for pyodbc.  It looks like mxODBC usesFreeTDSjust like pyodbc.  The
> > >> problem appears to be withFreeTDS, so mxODBC doesn't look like it
> > >> would be a fix for my particular problem.  In fact, mxODBC says as
> > >> much itself:
>
> > >> (From the mxODBC change log, Changes from 2.0.7 to 3.0.0:
> > >>http://www.egenix.com/products/python/mxODBC/changelog.html)
>
> > >> "Added new experimental subpackage mx.ODBC.FreeTDSwhich allows
> > >> connecting to MS SQL Server from Unix. This is still far from even
> > >> being called alpha due to the current lack of implementation support
> > >> of many important ODBC APIs in theFreeTDSODBC driver."
>
> > >> The problem still seems to beFreeTDS.
>
> > >> On Jul 30, 9:42 am, Michele Comitini <michele.comit...@gmail.com>
> > >> wrote:
> > >> > you can try with mxodbchttp://www.egenix.com/products/python/mxODBC/
>
> > >> > 2010/7/30 mwolfe02 <michael.joseph.wo...@gmail.com>
>
> > >> > > For background, refer to related questions on SO:
> > >> > >http://stackoverflow.com/questions/3364699/
> > >> > >http://stackoverflow.com/questions/3371795/
>
> > >> > > I'm developing my app on Windows, but will be deploying it on a Linux
> > >> > > server.  The following lines are from the top of my db.py:
>
> > >> > > if os.name=='posix':  #Ubuntu
> > >> > >    db =
> > >> > > DAL('mssql://user:p...@10.0.8.30/TaxDB?DRIVER={FreeTDS}<http://user:p...@10.0.8.30/TaxDB?DRIVER=%7BFreeTDS%7D>
> > >> > > ',
> > >> > > migrate=False)
> > >> > > elif os.name=='nt':   #Windows
> > >> > >    db = DAL('mssql://u...@127.0.0.1/TaxDB_Dev?
> > >> > > Trusted_Connection=Yes', migrate=False)
>
> > >> > > The problem I am running into is that on Windows, my MS SQL money
> > >> > > fields are returned as Python Decimals.  However, on Linux, those same
> > >> > > fields are returned as floats.  I think the issue is withFreeTDS.
> > >> > > The above lines are the _only_ lines that are different between my
> > >> > > production and development versions.  Obviously, the environments
> > >> > > themselves are vastly different.
>
> > >> > > Is there an alternative toFreeTDSto connect to SQL Server from
Reply all
Reply to author
Forward
0 new messages