Problem with Kealib Python bindings

[fortozs]

I'm a macports user and I need to get my kealib working again. The port on macports has been failing for a long time now. I managed to install kealib from source after a little tinkering, and 

gdalinfo --formats | grep KEA

shows that KEA is an available format. However, when I try to open a .kea file in Python, I get the following message...

ERROR 4: `injune_p142_casi_sub_ll.kea' not recognized as a supported file format.

First, I tried uninstalling and reinstalling py27-gdal, but that didn't work. Then I tried running the setup.py file in the python folder. It seemed to install fine, but still no luck. I downloaded the package using Mercurial and installed using CMake. I am on MacOS 10.3.2 and using gdal version 2.2.3. My GDAL_DRIVER_PATH is /opt/local/lib/gdalplugins, and I have a compiled .dylib in there. Am I perhaps overlooking some system setting in Python? If anyone has any ideas, please let me know. Thanks.

[Pete Bunting [pfb]]

Hi,

 

I am afraid I don’t use MacPorts so difficult to comment on this issue.

 

If gdalinfo finds the KEA driver can you try doing a gdal_translate of the file to another format – just to make sure that your GDAL installation is really working correctly with KEA.

 

If that works then I can only assume that your gdal installation with KEA isn’t the version which your python installation is referring too.

 

The KEA driver and python aren’t related other, accessing KEA files goes through GDAL in the same way as other formats support by GDAL, than through GDAL so it is going to be a GDAL installation issue rather than python directly.  

 

Since GDAL 2.0 the KEA driver is now in the GDAL source so if the HDF5 C++ library is present KEA is built. The fact you have been able to build KEA yourself probably implies that the HDF5 C++ library is available on your system is this is the same version, from MacPorts, that GDAL was built against KEA should have been build and come down with your GDAL installation. Might be worth removing your GDAL_DRIVER_PATH variable and seeing whether KEA is still present?

 

As I said, I am not familiar with MacPorts. Can you create different environments? If you can then maybe try a clean environment and install in that without your modifications?

 

Alternatively, you could use conda with the conda-forge channel. That is the system I use and we “try” to maintain.

 

Cheers, Pete

 

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: pete.b...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************

"Please consider the environment before printing this email or any documents attached”

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

--------------------------------------------------------------------
Prifysgol Aberystwyth www.aber.ac.uk
Prifysgol y Flwyddyn ar gyfer Ansawdd Dysgu - The Times & The Sunday Times 2018.

Aberystwyth University www.aber.ac.uk
University of the Year for Teaching Quality - The Times & The Sunday Times 2018.

[fortozs]

I'll try disabling the driver path and see if it comes up with anything. It seems to be going to the correct (only?) installation, but I guess it won't hurt to try. Macports is not like Conda in that it's not typical to have multiple environments. I've tried Conda and didn't like it as much as macports, except it is very cool that it's cross-platform. My problem is that Conda libraries will sometimes conflict with my macports installation, so I do my best to avoid using it. In the past macports had a port for kealib and rsgislib, but the maintainer has been quiet for quite some time now. I'll look over your points and see if I can't come up with something. Thanks.

To post to this group, send email to rsgisli...@googlegroups.com.

Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[fortozs]

Removing the driver path causes gdal to not find KEA.

[Pete Bunting [pfb]]

Hi,

 

I’ve had a quick look at the macports build instructions (https://github.com/macports/macports-ports) and what happens if you install kealib via macports? I see there are rsgislib build instructions, does it work to download rsgislib via macports as that uses kealib directly for functionality which is not available via GDAL.

 

I noticed that KEA isn’t in the driver configures for GDAL on macports so maybe it would be worth contacting the guys who maintain the macports builds for have that added to the GDAL build instructions. There are options in the hdf5 build for the C++ library to be built so shouldn’t be a problem.

To post to this group, send email to rsgislib...@googlegroups.com.

Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[fortozs]

If I install via macports, the build fails. Something about "naughty" and "nice", I really don't know, lol. 

:info:build       kealib::KEAAttributeTable::createKeaStringCompTypeMem() in KEAAttributeTable.cpp.o

:info:build       ...

:info:build ld: symbol(s) not found for architecture x86_64

:info:build clang: error: linker command failed with exit code 1 (use -v to see invocation)

:info:build make[2]: *** [src/libkea.1.4.4.dylib] Error 1

:info:build make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_gis_kealib/kealib/work/kealib-1.4.4/trunk'

:info:build make[1]: *** [src/CMakeFiles/kea.dir/all] Error 2

:info:build make[1]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_gis_kealib/kealib/work/kealib-1.4.4/trunk'

:info:build make: *** [all] Error 2

:info:build make: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_gis_kealib/kealib/work/kealib-1.4.4/trunk'

:info:build Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_gis_kealib/kealib/work/kealib-1.4.4/trunk" && /usr/bin/make -w all VERBOSE=ON 

:info:build Exit code: 2

:error:build Failed to build kealib: command execution failed

:debug:build Error code: CHILDSTATUS 70104 2

:debug:build Backtrace: command execution failed

:debug:build     while executing

:debug:build "system {*}$notty {*}$nice $fullcmdstring"

:debug:build     invoked from within

:debug:build "command_exec build"

:debug:build     (procedure "portbuild::build_main" line 8)

:debug:build     invoked from within

:debug:build "$procedure $targetname"

:error:build See /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_gis_kealib/kealib/main.log for details.

I know that it broke when gdal went to 2.0, and hasn't worked since. I will open another ticket and see if we can't get this sorted out. I've built gdal, kealib, and rsgislib from source before, but it would be a pain to do this every time one of them is updated. What I've resorted to the last year or so is using Conda and renaming my Conda directory when not in use. This will get me by if necessary, but it'd be nice if I could get it compiled manually, and have it play well with my package system. I appreciate your help. I will be working to figure out what the problem is.

[fortozs]

As far as installing rsgislib, macports doesn't recognize the kealib installation, I believe because manual installation won't update my macports database. It just sees kealib as a dependency, tries to build kealib, and fails.

[fortozs]

So it is definitely not related to Python. I can replicate the problem with

$ gdalinfo --formats | grep KEA

  KEA -raster- (w+): KEA Image Format (.kea)

$ gdalinfo injune_p142_casi_sub_ll.kea

ERROR 4: `injune_p142_casi_sub_ll.kea' not recognized as a supported file format.

So for some reason, it will compile and thinks KEA is supported, but it can't actually use kealib for some reason. Using ccmake, I made the following settings. Does anything just stick out as terribly wrong? I try to avoid compiling from source, so it could be some rookie mistake. I've seen cases where packages and dependencies all need to be installed by the same compiler. Could it be related to the fact gdal was complied with clang?

CMAKE_INSTALL_PREFIX=/opt/local

BUILD_SHARED_LIBS=ON

CMAKE_VERBOSE_MAKEFILE=ON

HDF5_INCLUDE_DIR=/opt/local/include

HDF5_LIB_PATH=/opt/local/lib

GDAL_INCLUDE_DIR=/opt/local/include

GDAL_LIB_PATH=/opt/local/lib

CMAKE_AR=/opt/local/bin/gcc-ar-mp-5

CMAKE_C_COMPILER=/opt/local/bin/gcc

CMAKE_C_COMPILER_AR=/opt/local/bin/gcc-ar-mp-5

CMAKE_C_COMPILER_RANLIB=/opt/local/bin/gcc-ranlib-mp-5

CMAKE_C_FLAGS='-arch x86_64'

CMAKE_RANLIB=/opt/local/bin/gcc-ranlib-mp-5

CMAKE_CXX_COMPILER=/opt/local/bin/g++

CMAKE_CXX_COMPILER_AR=/opt/local/bin/gcc-ar-mp-5

CMAKE_CXX_COMPILER_RANLIB=/opt/local/bin/gcc-ranlib-mp-5

CMAKE_CXX_FLAGS='-arch x86_64 -I/opt/local/include'

CMAKE_SHARED_LINKER_FLAGS='-L/opt/local/lib -lhdf5_cpp -lhdf5_hl_cpp -lhdf5 -lhdf5_hl'

HDF5_STATIC_LIBS=OFF

KEAHDF5_STATIC_LIBS=ON

LIBKEA_WITH_GDAL=ON

[Pete Bunting [pfb]]

Hi,

 

I am not sure, I can’t see anything immediately wrong with your options for cmake. However, I think the best solution here is to look at getting the MacPorts builds working. Do you know, or can find out, who created them in the first place?

 

From my quick look at them I don’t think it would be too hard to getting them working again, particularly if you knew about MacPorts. Adding the KEA driver so it is built by GDAL is probably the best solution. Unfortunately, while I am a Mac user I don’t use MacPorts and don’t have the time to look at over the coming months.

 

I’d recommend using conda (conda-forge) but rather than renaming your miniconda directory you could set up a similar system to myself. I have multiple conda environments for development, building packages and ‘normal work’ and when I start a terminal it asks which I want, if I just hit Enter then it defaults to my normal environment. You could have it given you the option between MacPorts and conda… I have the following in my .bash_profile:

 

read -p "Which environment do you want:

1) Conda OSGEO (Default - just hit enter)

2) Conda Default

3) Conda Google Earth Engine

4) Conda rsgislib dev

5) Conda spdlib dev

6) None...

" RESP

if [ "$RESP" = "1" ]; then

  echo "Loading Conda OSGEO..."

  export PATH="/Users/pete/miniconda3/bin:$PATH"

  source activate au-eoed-v3

elif [ "$RESP" = "2" ]; then

  echo "Loading Conda Default..."

  export PATH="/Users/pete/miniconda3/bin:$PATH"

elif [ "$RESP" = "3" ]; then

  echo "Loading Conda Google Cloud / Earth Engine..."

  export PATH="/Users/pete/miniconda3/bin:$PATH"

  source activate eegoog

  # The next line updates PATH for the Google Cloud SDK.

  if [ -f '/Users/pete/google-cloud-sdk/path.bash.inc' ]; then source '/Users/pete/google-cloud-sdk/path.bash.inc'; fi

  # The next line enables shell command completion for gcloud.

  if [ -f '/Users/pete/google-cloud-sdk/completion.bash.inc' ]; then source '/Users/pete/google-cloud-sdk/completion.bash.inc'; fi

elif [ "$RESP" = "4" ]; then

  echo "Loading Conda RSGISLib Dev Environment..."

  export PATH="/Users/pete/miniconda3/bin:/Users/pete/Temp/rsgislibdev/bin:/Users/pete/Temp/arcsidev/bin:$PATH"

  export PYTHONPATH="/Users/pete/Temp/rsgislibdev/lib/python3.5/site-packages:/Users/pete/Temp/arcsidev/lib/python3.5/site-packages:$PYTHONPATH"

  export DYLD_LIBRARY_PATH="/Users/pete/Temp/rsgislibdev/lib"

  source activate rsgislibdev

elif [ "$RESP" = "5" ]; then

  echo "Loading Conda SPDLib Dev Environment..."

  export PATH="/Users/pete/miniconda3/bin:/Users/pete/Temp/spdlibdev/bin:$PATH"

  export PYTHONPATH="/Users/pete/Temp/spdlibdev/lib/python3.5/site-packages:$PYTHONPATH"

  export DYLD_LIBRARY_PATH="/Users/pete/Temp/spdlibdev/lib"

  source activate spdlibdev

elif [ "$RESP" = "6" ]; then

  echo "Do Nothing..."

else

  echo "Loading Conda OSGEO..."

  export PATH="/Users/pete/miniconda3/bin:$PATH"

  source activate au-eoed-v3

fi

 

Best wishes,

 

Pete

 

 

 

 

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: pete.b...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************

"Please consider the environment before printing this email or any documents attached”

 

 

From: <rsgislib...@googlegroups.com> on behalf of fortozs <for...@gmail.com>
Date: Saturday, 20 January 2018 at 08:31
To: RSGISLib Support <rsgislib...@googlegroups.com>
Subject: Re: [rsgislib-support] Problem with Kealib Python bindings

 

So it is definitely not related to Python. I can replicate the problem with

--

You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.

To post to this group, send email to rsgislib...@googlegroups.com.

Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[fortozs]

Thanks for the info. I actually have talked to the maintainer of gdal and kealib and am waiting to hear back from him. I managed to patch the kealib portfile to the point where it will install as I did here, but I still have the same issue (sees the plugin but it doesn't work). One question, does it matter what version of gcc that I use for the install? One of my issues was that it wasn't building with Macports default compiler, Clang. It now builds with gcc5, I wonder if maybe a different version might yield better results.

[Pete Bunting [pfb]]

Hi,

 

Compiler versions can cause problems but usually you’d get an error about not finding something in a library or incompatible library or something as the software tried to use the library in question.

 

Clang is the default on a mac so if you don’t specify a compiler is should pick that one up automatically, I think.

 

Cheers, Pete

 

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: pete.b...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************

"Please consider the environment before printing this email or any documents attached”

 

 

From: <rsgislib...@googlegroups.com> on behalf of fortozs <for...@gmail.com>
Date: Saturday, 20 January 2018 at 12:18
To: RSGISLib Support <rsgislib...@googlegroups.com>
Subject: Re: [rsgislib-support] Problem with Kealib Python bindings

 

Thanks for the info. I actually have talked to the maintainer of gdal and kealib and am waiting to hear back from him. I managed to patch the kealib portfile to the point where it will install as I did here, but I still have the same issue (sees the plugin but it doesn't work). One question, does it matter what version of gcc that I use for the install? One of my issues was that it wasn't building with Macports default compiler, Clang. It now builds with gcc5, I wonder if maybe a different version might yield better results.

--

You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[fortozs]

Well, I can't say for sure if it was a compiler issue, because I also realized I was missing the fftw dependency, but reinstalling all dependencies with the same gcc compiler seems to have done the trick! My original macports ticket on this issue was opened 20 months ago, so to say I am very happy would be a major understatement. I have a very rough patch here if anyone is interested in a macports install. This patch was specific to gcc 5, but I decided to switch to gcc 7.2 while I was at it. I can provide the proven gcc7 patch if anyone is interested.

https://trac.macports.org/ticket/51584

Now I can finally reopen all those atmospheric corrections I put so much work into. Thanks so much for your help.