python bindings to hypertable

[Mateusz Berezecki]

Hi list readers,

In case somebody was interested in this or even possibly helping out
(any python pros here?),
I'm creating python bindings to the Hypertable using Boost.Python.
It's far from perfect and stable/ready yet but the sneak-peek preview is
at : http://github.com/mateuszb/hypertable/tree/master/contrib/cc/PythonBinding/test.py
It supports read operations only at this stage.

Note well that after compilation (you need Python development
packages) you need to
rename libht.so to ht.so (sorry, no automation at this stage)

Later on, the example is self-explanatory

import ht

# create client
client = ht.Client("ht", "/opt/hypertable/0.9.0.7/conf/hypertable.cfg")

# open the table
table = client.open_table("METADATA")

# create scan specification
scan_spec = ht.ScanSpec()
scan_spec.start_row = "\x00"
scan_spec.end_row = "\xff\xff"
scan_spec.start_row_inclusive = True
scan_spec.end_row_inclusive = True
scan_spec.max_versions = 1

# initialize the scanner with it
scanner = table.create_scanner(scan_spec)

# create cell holding consecutive values from the scanner
cell = ht.Cell()

# scan it!
while scanner.next(cell):
# note that instead of cell.value and cell.value_len fields
# you get single cell.value() function returning the value from the cell
print "%s:%s %s" % (cell.row_key, cell.column_family, cell.value())

Mateusz

[Paolo Ambrosio]

Hi Mateusz,

I had no time to try your code, but python bindings are definitely
something that I will need soon. It would be interesting to know if
something is being done for the Thrift implementation previuosly
discussed on the user list [1] [2].

Paolo

.. [1] http://groups.google.com/group/hypertable-user/browse_thread/thread/b3425327092568a6
.. [2] http://groups.google.com/group/hypertable-user/browse_thread/thread/431f541b146f31e5/fec70741c9b27f9d

[osimons]

On Jul 29, 1:22 am, "Mateusz Berezecki" <mateu...@gmail.com> wrote:
> Hi list readers,
>
> In case somebody was interested in this or even possibly helping out
> (any python pros here?),
> I'm creating python bindings to the Hypertable using Boost.Python.
> It's far from perfect and stable/ready yet but the sneak-peek preview is

> at :http://github.com/mateuszb/hypertable/tree/master/contrib/cc/PythonBi...

> It supports read operations only at this stage.
>
> Note well that after compilation (you need Python development
> packages) you need to
> rename libht.so to ht.so (sorry, no automation at this stage)
>
> Later on, the example is self-explanatory
>

I'm very interested, but don't seem to get a libht.so when building. I
do see a libht.a inside the build directory, but that is all.

Any instructions I've missed?


:::simon

[Mateusz Berezecki]

You'd have to run cmake with -DBUILD_SHARED_LIBS=ON
and then rename libht.so to ht.so . Sorry that this is not yet automated.

Also, I'm going to push a write support in the next 12 hours,
so you will be able to insert data to hypertable from python.

Mateusz

[osimons]

On Aug 1, 2:38 am, "Mateusz Berezecki" <mateu...@gmail.com> wrote:

> On Fri, Aug 1, 2008 at 2:31 AM, osimons <oddsim...@gmail.com> wrote:
>
> > I'm very interested, but don't seem to get a libht.so when building. I
> > do see a libht.a inside the build directory, but that is all.
> > Any instructions I've missed?
>
> You'd have to run cmake with -DBUILD_SHARED_LIBS=ON
> and then rename libht.so to ht.so . Sorry that this is not yet automated.

Thanks, that helped progress. However, now the linker seems to have a
problem finding the Boost Python library:

[100%] Building CXX object contrib/cc/PythonBinding/CMakeFiles/ht.dir/
HypertableBindings.o
Linking CXX shared library libht.so
/usr/bin/ld: cannot find -lboost_python-mt
collect2: ld returned 1 exit status
make[5]: *** [contrib/cc/PythonBinding/libht.so] Error 1

Configure claims to detect Boost just fine, and Hypertable without the
Python bindings compiles and links without problems:

-- Looking for required boost libraries...
-- Boost include dir: /usr/local/include/boost-1_35
-- Boost thread lib: /usr/local/lib/libboost_thread-gcc41-mt.so
-- Boost program options lib: /usr/local/lib/libboost_program_options-
gcc41-mt.so
-- Boost iostreams lib: /usr/local/lib/libboost_iostreams-gcc41-mt.so
-- Boost lib dir: /usr/local/lib
-- Boost version: 1_35

The relevant libraries are located like this:

/usr/local/lib/libboost_python-gcc41-mt-1_35.so
/usr/local/lib/libboost_python-gcc41-mt-1_35.so.1.35.0
/usr/local/lib/libboost_python-gcc41-mt.so

> Also, I'm going to push a write support in the next 12 hours,
> so you will be able to insert data to hypertable from python.
>

Oh. Nice! Hope to get it working soon to try everything out.


:::simon

[Mateusz Berezecki]

On Fri, Aug 1, 2008 at 4:35 PM, osimons <odds...@gmail.com> wrote:
>
>
>
> On Aug 1, 2:38 am, "Mateusz Berezecki" <mateu...@gmail.com> wrote:
>> On Fri, Aug 1, 2008 at 2:31 AM, osimons <oddsim...@gmail.com> wrote:
>>
>> > I'm very interested, but don't seem to get a libht.so when building. I
>> > do see a libht.a inside the build directory, but that is all.
>> > Any instructions I've missed?
>>
>> You'd have to run cmake with -DBUILD_SHARED_LIBS=ON
>> and then rename libht.so to ht.so . Sorry that this is not yet automated.
>
> Thanks, that helped progress. However, now the linker seems to have a
> problem finding the Boost Python library:
>
> [100%] Building CXX object contrib/cc/PythonBinding/CMakeFiles/ht.dir/
> HypertableBindings.o
> Linking CXX shared library libht.so
> /usr/bin/ld: cannot find -lboost_python-mt
> collect2: ld returned 1 exit status
> make[5]: *** [contrib/cc/PythonBinding/libht.so] Error 1
>

I probably should have put this the same way other boost libraries
are detected in the CMake file, i.e. by explicitly configuring
paths by variables passed to the cmake.

> Configure claims to detect Boost just fine, and Hypertable without the
> Python bindings compiles and links without problems:
>
> -- Looking for required boost libraries...
> -- Boost include dir: /usr/local/include/boost-1_35
> -- Boost thread lib: /usr/local/lib/libboost_thread-gcc41-mt.so
> -- Boost program options lib: /usr/local/lib/libboost_program_options-
> gcc41-mt.so
> -- Boost iostreams lib: /usr/local/lib/libboost_iostreams-gcc41-mt.so
> -- Boost lib dir: /usr/local/lib
> -- Boost version: 1_35
>
> The relevant libraries are located like this:
>
> /usr/local/lib/libboost_python-gcc41-mt-1_35.so
> /usr/local/lib/libboost_python-gcc41-mt-1_35.so.1.35.0
> /usr/local/lib/libboost_python-gcc41-mt.so
>

Could you please rename the library inside the
contrib/cc/PythonBindings/CMakeLists.txt
to the appropriate name of your boost python library?

I didn't have time to focus on it yet.

>> Also, I'm going to push a write support in the next 12 hours,
>> so you will be able to insert data to hypertable from python.
>>

it's live now under

git://github.com/mateuszb/hypertable.git

Mateusz

[osimons]

On Aug 1, 5:08 pm, "Mateusz Berezecki" <mateu...@gmail.com> wrote:
> I probably should have put this the same way other boost libraries
> are detected in the CMake file, i.e. by explicitly configuring
> paths by variables passed to the cmake.
>

> I didn't have time to focus on it yet.

Below is a small patch that I did that makes it locate it correctly:

diff --git a/cmake/FindBoost.cmake b/cmake/FindBoost.cmake
index d78c88d..d1da9fc 100644
--- a/cmake/FindBoost.cmake
+++ b/cmake/FindBoost.cmake
@@ -9,6 +9,7 @@
# BOOST_THREAD_LIB - The name of the boost thread library
# BOOST_PROGRAM_OPTIONS_LIB - The name of the boost program options
library
# BOOST_IOSTREAMS_LIB - The name of the boost program options
library
+# BOOST_PYTHON_LIB - The name of the boost python library
#
----------------------------------------------------------------------------
#
# Usage:
@@ -114,10 +115,12 @@ if (Boost_INCLUDE_DIR)
FIND_BOOST_LIBRARY(BOOST_THREAD_LIB thread ${Boost_PARENT})
FIND_BOOST_LIBRARY(BOOST_PROGRAM_OPTIONS_LIB program_options $
{Boost_PARENT})
FIND_BOOST_LIBRARY(BOOST_IOSTREAMS_LIB iostreams ${Boost_PARENT})
+ FIND_BOOST_LIBRARY(BOOST_PYTHON_LIB python ${Boost_PARENT})

message(STATUS "Boost thread lib: ${BOOST_THREAD_LIB}")
message(STATUS "Boost program options lib: $
{BOOST_PROGRAM_OPTIONS_LIB}")
message(STATUS "Boost iostreams lib: ${BOOST_IOSTREAMS_LIB}")
+ message(STATUS "Boost python lib: ${BOOST_PYTHON_LIB}")
get_filename_component(Boost_LIBRARY_DIR ${BOOST_THREAD_LIB} PATH)
message(STATUS "Boost lib dir: ${Boost_LIBRARY_DIR}")

diff --git a/contrib/cc/PythonBinding/CMakeLists.txt b/contrib/cc/
PythonBinding/CMakeLists.txt
index c648261..2bfbb83 100644
--- a/contrib/cc/PythonBinding/CMakeLists.txt
+++ b/contrib/cc/PythonBinding/CMakeLists.txt
@@ -3,5 +3,5 @@ add_library(ht HypertableBindings.cc)
include_directories(${PYTHON_INCLUDE_PATH} ${Boost_INCLUDE_DIRS}
"../../../src/cc")
link_directories(${Boost_LIBRARIES})

-target_link_libraries(ht boost_python-mt Hypertable)
+target_link_libraries(ht ${BOOST_PYTHON_LIB} Hypertable)

Next, I have yet another compile need that I'm not sure how to do... I
want to compile the bindings against a custom python (2.5.x) at a
custom /opt location, and not the default CentOS python (2.4.3). What
is the best way to work around this? Modify some compile library path
setting? I tried some of the swtiches I found in cmake files (-
DPYTHON_LIBS_FOUND and -DPYTHON_INCLUDE_PATH), but they don't seem to
make much difference. The main locator seems to be main CMakeLists.txt
line 86: find_package(PythonLibs)

Is there some general library path search path variable that can be
set for the location mechanisms?


:::simon

[Mateusz Berezecki]

Thanks a lot for the patch !

> Next, I have yet another compile need that I'm not sure how to do... I
> want to compile the bindings against a custom python (2.5.x) at a
> custom /opt location, and not the default CentOS python (2.4.3). What
> is the best way to work around this? Modify some compile library path
> setting? I tried some of the swtiches I found in cmake files (-
> DPYTHON_LIBS_FOUND and -DPYTHON_INCLUDE_PATH), but they don't seem to
> make much difference. The main locator seems to be main CMakeLists.txt
> line 86: find_package(PythonLibs)

I think there's also find_package for python interpreter or something like this,
but I am not a CMake expert...

>
> Is there some general library path search path variable that can be
> set for the location mechanisms?

There certainly is one but I don't know the name and how to set
it with CMake. As I said, I don't know CMake good enough yet
as I just started using it.

Mateusz

[osimons]

> > Next, I have yet another compile need that I'm not sure how to do... I
> > want to compile the bindings against a custom python (2.5.x) at a
> > custom /opt location, and not the default CentOS python (2.4.3). What
> > is the best way to work around this? Modify some compile library path
> > setting? I tried some of the swtiches I found in cmake files (-
> > DPYTHON_LIBS_FOUND and -DPYTHON_INCLUDE_PATH), but they don't seem to
> > make much difference. The main locator seems to be main CMakeLists.txt
> > line 86: find_package(PythonLibs)
>
> I think there's also find_package for python interpreter or something like this,
> but I am not a CMake expert...
>
>
>
> > Is there some general library path search path variable that can be
> > set for the location mechanisms?
>
> There certainly is one but I don't know the name and how to set
> it with CMake. As I said, I don't know CMake good enough yet
> as I just started using it.

It seems CMake have some issues with Python-related custom
configurations:

http://www.cmake.org/Bug/view.php?id=2257

I read through the cmake Modules/FindPythonLibs.cmake code, and see no
obvious way of getting it to work. Only clean solution I see is to
write a custom locator macro that also could provide guidance for
actually installing the bindings.

For instance:
cmake ... -DWITH_PYTHON=/opt/bin/python2.5

Then based on logic in patches on the mentioned cmake ticket, we can
query that interpreter for proper paths, and that would also provide
the necessary information to actually install the bindings. Without
the option, the bindings are just built with default python config and
not installed (current behaviour).


:::simon

[dmkd]

Trying to build HypertableBindings.cc
cmake -DBUILD_SHARED_LIBS=ON /path/to/PythonBinding/
make

<.....>
/usr/local/src/hypertable-0.9.0.10-alpha/contrib/cc/PythonBinding/
HypertableBindings.h:166: instantiated from here
/usr/include/boost/noncopyable.hpp:27: error:
‘boost::noncopyable_::noncopyable::noncopyable(const
boost::noncopyable_::noncopyable&)’ is private
/usr/local/src/hypertable-0.9.0.10-alpha/contrib/cc/
PythonBinding/../../../src/cc/Hypertable/Lib/ScanSpec.h:118: error:
<....>