Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
pyodbc
Home
+ new post
About this group
Join this group
pyodbc very slow when compiled from source (very fast when using distribution binary)
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   1 message - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
lomi...@gmail.com  
View profile  
 More options Aug 27 2012, 3:17 am
From: lomi...@gmail.com
Date: Mon, 27 Aug 2012 00:17:26 -0700 (PDT)
Local: Mon, Aug 27 2012 3:17 am
Subject: pyodbc very slow when compiled from source (very fast when using distribution binary)
Print | Individual message | Show original | Report this message | Find messages by this author

Hi,

I am trying to compile pyodbc from source (to be able to use a more recent
version than the one on the official distribution repo).
Compiling under Mint, the resulting pyodbc is extremely slow. Compiling
under Ubuntu there does not seem to be an issue. Using the official binary
from Mint is fast enough, although it is outdated (version 2.1.7).
I compared the compiler and version for Ubuntu and Mint, they are the same
(gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3), and the output of the
compilation are identical.

Would anyone know of what could cause this or if something could help?

The tests and numbers are as follow:

   - I am using pyodbc versus Vertica, an analytics platform with an odbc
   backend. This is load optimized database, which has an equivalent of the
   mysql 'LOAD DATA INFILE', which is what I am interested in now.
   - To be sure the issue is with pyodbc I ran the following tests, using a
   15GB file:
      - dd if=test.csv of=/dev/null => 3 minutes, meaning that reading from
      disk is fast enough
      - using the binary client instead of odbc => 14MB/s of bandwidth,
      meaning that Vertica can handle it
      - using isql instead of pyodbc => 14MB/s of bandwidth, meaning that
      odbc can handle it
      - using the distribution pyodbc => 14MB/s of bandwidth, meaning that
      pyodbc could handle it
      - using my own pyodbc : => 300kB/s meaning that there is an issue

The output of my pip install is as follow:

(adwords)guillaume@D116 ~/dwh/dwh/etl/adwords_importer $ pip install pyodbc
Downloading/unpacking pyodbc
  Using download cache from
/home/guillaume/.pip_download_cache/http%3A%2F%2Fpyodbc.googlecode.com%2Ffi les%2Fpyodbc-3.0.6.zip
  Running setup.py egg_info for package pyodbc

    warning: no files found matching 'tests/*'
Installing collected packages: pyodbc
  Running setup.py install for pyodbc
    building 'pyodbc' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/sqlwchar.cpp -o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/sqlwchar.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbcdbg.cpp -o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/pyodbcdbg.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/row.cpp -o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/row.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/connection.cpp -o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/connection.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/cnxninfo.cpp -o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/cnxninfo.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/errors.cpp -o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/errors.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/params.cpp -o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/params.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/getdata.cpp -o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/getdata.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbcmodule.cpp
-o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/pyodbcmodule.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/buffer.cpp -o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/buffer.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/cursor.cpp -o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/cursor.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -DPYODBC_VERSION=3.0.6 -I/usr/include/python2.7
-c /home/guillaume/.virtualenvs/adwords/build/pyodbc/src/pyodbccompat.cpp
-o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/pyodbccompat.o
-Wno-write-strings
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid
for Ada/C/ObjC but not for C++ [enabled by default]
    g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions
-Wl,-Bsymbolic-functions -Wl,-z,relro
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/sqlwchar.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/pyodbcdbg.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/row.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/connection.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/cnxninfo.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/errors.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/params.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/getdata.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/pyodbcmodule.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/buffer.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/cursor.o
build/temp.linux-x86_64-2.7/home/guillaume/.virtualenvs/adwords/build/pyodb c/src/pyodbccompat.o
-lodbc -o build/lib.linux-x86_64-2.7/pyodbc.so

    warning: no files found matching 'tests/*'
Successfully installed pyodbc
Cleaning up...

Thanks,
Guillaume


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2013 Google