pyodbc very slow when compiled from source (very fast when using distribution binary)

[lom...@gmail.com]

Hi,

I am trying to compile pyodbc from source (to be able to use a more recent version than the one on the official distribution repo).
Compiling under Mint, the resulting pyodbc is extremely slow. Compiling under Ubuntu there does not seem to be an issue. Using the official binary from Mint is fast enough, although it is outdated (version 2.1.7).
I compared the compiler and version for Ubuntu and Mint, they are the same (gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3), and the output of the compilation are identical.

Would anyone know of what could cause this or if something could help?


The tests and numbers are as follow:

• I am using pyodbc versus Vertica, an analytics platform with an odbc backend. This is load optimized database, which has an equivalent of the mysql 'LOAD DATA INFILE', which is what I am interested in now.
  
• To be sure the issue is with pyodbc I ran the following tests, using a 15GB file:
• dd if=test.csv of=/dev/null => 3 minutes, meaning that reading from disk is fast enough
• using the binary client instead of odbc => 14MB/s of bandwidth, meaning that Vertica can handle it
• using isql instead of pyodbc => 14MB/s of bandwidth, meaning that odbc can handle it
• using the distribution pyodbc => 14MB/s of bandwidth, meaning that pyodbc could handle it
• using my own pyodbc : => 300kB/s meaning that there is an issue
  

The output of my pip install is as follow:

(adwords)guillaume@D116 ~/dwh/dwh/etl/adwords_importer $ pip install pyodbc
Downloading/unpacking pyodbc
  Using download cache from /home/guillaume/.pip_download_cache/http%3A%2F%2Fpyodbc.googlecode.com%2Ffiles%2Fpyodbc-3.0.6.zip
  Running setup.py egg_info for package pyodbc
   
    warning: no files found matching 'tests/*'
Installing collected packages: pyodbc
  Running setup.py install for pyodbc
    building 'pyodbc' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/sqlwchar.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/sqlwchar.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbcdbg.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbcdbg.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/row.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/row.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/connection.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/connection.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/cnxninfo.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/cnxninfo.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/errors.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/errors.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/params.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/params.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/getdata.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/getdata.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbcmodule.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbcmodule.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/buffer.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/buffer.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/cursor.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/cursor.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7 -c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbccompat.cpp -o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbccompat.o -Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/sqlwchar.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbcdbg.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/row.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/connection.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/cnxninfo.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/errors.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/params.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/getdata.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbcmodule.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/buffer.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/cursor.o build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbccompat.o -lodbc -o build/lib.linux-x86_64-2.7/pyodbc.so
   
    warning: no files found matching 'tests/*'
Successfully installed pyodbc
Cleaning up...


Thanks,
Guillaume