HDF5 on Windows?

[Tim Holy]

Has anyone gotten the HDF5 package working on Windows? If so, do you mind
sharing how you did it? (I haven't really tried, maybe it's easy, but I'm such
a Windows newbie that I suspect my attempts will be rather fumbling at first.)

If no one has done this, clearly it's something that I should figure out how to
get working and add to the package/instructions/whatever.

Best,
--Tim

[Tim Holy]

Thanks for your reply. While I'm a complete newbie on Windows, my suspicion is
that this is a library issue. There are at least two possibilities:

1. Perhaps it's simply not finding the library. Any luck if you modify line 21
of HDF5/src/plain.jl and specify the full path to the HDF5 dll? (If you have a
dll, that is, see below.)

2. There's an incompatibility in the compiled versions of Julia and libhdf5.
If I understand correctly, on Windows julia is compiled with MinGW. The
download page for HDF5 lists versions built with several compilers, of which
the cygwin variants seem likely to be the closest match. I'm skeptical that
the VS (Visual Studio) version will work with julia.
I downloaded the cygwin variant, and there is no ".dll" version of the
library, just ".a" and ".la" files. It seems rather likely that these won't
work for us.

I'm currently guessing that we need to compile libhdf5 from source. With
MinGW? But won't that force everyone to have a development environment set up?
Or can we host our own binary somewhere?

Best,
--Tim


On Thursday, June 06, 2013 02:00:48 AM you wrote:
> Actually had some issues with this yesterday, and then gave up. I am using
> the pre-compiled binaries (pre0.2.0) and could not get it to work. I wanted
> to use the MAT package, but suspect that the HDF5 package is not working.
> Here is a copy of the output.
>
> _
> _ _ _(_)_ | A fresh approach to technical computing
> (_) | (_) (_) | Documentation: http://docs.julialang.org
> _ _ _| |_ __ _ | Type "help()" to list help topics
>
> | | | | | | |/ _` | |
> | | |
> | | |_| | | | (_| | | Version 0.2.0
>
> _/ |\__'_|_|_|\__'_| | c42e3a6-WINNT-i686 (2013-05-28 19:39:07)
>
> |__/ |
>
> julia> Pkg.installed()
> ["Stats"=>v"0.2.1","Calculus"=>v"0.1.1","Distributions"=>v"0.1.3","Options"=
> >v"0
> .2.0","Optim"=>v"0.1.5","MAT"=>v"0.2.0","Zlib"=>v"0.0.0","HDF5"=>v"0.2.2"]
>
> julia> using MAT
> ERROR: could not load module libhdf5: no error
> at C:\Users\Thuis\AppData\Roaming\julia\packages\HDF5\src\plain.jl:21
> at C:\Users\Thuis\AppData\Roaming\julia\packages\HDF5\src\HDF5.jl:1
> at C:\Users\Thuis\AppData\Roaming\julia\packages\MAT\src\MAT_HDF5.jl:30
> at C:\Users\Thuis\AppData\Roaming\julia\packages\MAT\src\MAT.jl:25
>
> julia> using HDF5
>
> julia> using MAT
> Warning: replacing module MAT_HDF5
> ERROR: HDF5ReferenceObj not defined
> at C:\Users\Thuis\AppData\Roaming\julia\packages\MAT\src\MAT_HDF5.jl:37
> at C:\Users\Thuis\AppData\Roaming\julia\packages\MAT\src\MAT.jl:25
>
> julia>
>
>
> PS; I have installed the HDF5 precompiled binaries,(HDF5-1.8.11 Pre-built
> Binary Distributions)
> PS2; Somewhat of a beginner, so if I have missed or incorrectly interpreted
> some of the instructions, please enlighten me.

[J Luis]

I don't have time right now to try this but I suspect that you are having a dependency problem. HDF5 probably depends on netcdf + zlib and all the dependencies must be in the path. The "Dependency Walker" is the tool to check the dependencies.

Quinta-feira, 6 de Junho de 2013 12:32:20 UTC+1, Jonne Guyt escreveu:

Completely out of my depth, but I tried to follow up on your suggestions. I have downloaded the 64 bit VS version of the precompiled binary. There is a hdf5.dll file included in the following folder:

C:\Program Files\HDF_Group\HDF5\1.8.11\bin

I have replaced line 21 of the plain.jl file with:

const libhdf5 = dlopen("C:\\Program Files\\HDF_Group\\HDF5\\1.8.11\\bin\\hdf5.dll")

This results in:

julia> using MAT

ERROR: could not load module C:\Program Files\HDF_Group\HDF5\1.8.11\bin\hdf5.dll

: no error

at C:\Users\Thuis\AppData\Roaming\julia\packages\HDF5\src\plain.jl:21

at C:\Users\Thuis\AppData\Roaming\julia\packages\HDF5\src\HDF5.jl:1

at C:\Users\Thuis\AppData\Roaming\julia\packages\MAT\src\MAT_HDF5.jl:30

at C:\Users\Thuis\AppData\Roaming\julia\packages\MAT\src\MAT.jl:25

julia> using MAT

Warning: replacing module MAT_HDF5

ERROR: HDF5ReferenceObj not defined

at C:\Users\Thuis\AppData\Roaming\julia\packages\MAT\src\MAT_HDF5.jl:37

at C:\Users\Thuis\AppData\Roaming\julia\packages\MAT\src\MAT.jl:25

I am guessing I am doing something incorrect here. I am more than willing to follow up on any suggestions, but given my lack of programming skills/knowledge I think I am not the best candidate to venture further.

[J Luis]

OK, so I gave it a try and got the same error(s) and that using an HDF5 dll build by me and that I use often.
I checked with the "Dependency Walker" and and those symbols, starting at MAT_HDF5.jl:37, are not present in the dll. Can it be that those "Fs" stand for "Fortran" and are not present in C only versions of the HDF5 dlls?

[J Luis]

Update. Don't know what happened before but I closed the laptop lid, went home, had dinner, woke up the machine ... and it now works. Just tested the small example of the HDF5 page and it worked (using a VS build hdf5 dll)

[Tim Holy]

On Thursday, June 06, 2013 12:55:42 PM J Luis wrote:
> Update. Don't know what happened before but I closed the laptop lid, went
> home, had dinner, woke up the machine ... and it now works. Just tested the
> small example of the HDF5 page and it worked (using a VS build hdf5 dll)

Very cool! I will try this myself sometime, but I'm about to go away for a
week, so it will be a while. Congrats!

--Tim

[J Luis]

Very cool! I will try this myself sometime, but I'm about to go away for a
week, so it will be a while. Congrats!

To you off course.
But there is a related issue that must be addressed, which is the names of the shared libs/ DLLs. They often are not the same when build with in unix like way where the become "libsomething.so" in contrast to pure Windows build (for example with projects and cmake generated projects) where they may become "something.dll"

Joaquim

[Tim Holy]

HDF5 is now working on Windows, and it should find the correct dll
automatically.

In other HDF5 news, Simon Kornblith has made a whole string of very nice
improvements. Most recently, he added support for mmapping "suitable" HDF5
datasets, with dramatic speedups if you read out lots of very small subsets of
such arrays. Given this and all the work that has gone on in readcsv etc,
Julia's data set I/O is getting quite the overhaul these days.

Best,
--Tim

[Viral Shah]

Tim,

This is fantastic. You are right that our I/O capabilities are getting better. I should also mention that with libExpat, we also have preliminary XML capabilities.

-viral

[J Luis]

HDF5 is now working on Windows, and it should find the correct dll
automatically.

Take as an example the zlib lib. It's called zlib1.dll on Windows and libz.so on unix, so one cannot have a unique ccal instructions that works for both OSs. My point is that it's not guarantied that *nix and windows names are equal and differ only by extension.

Joaquim
 

[Tim Holy]

On Wednesday, July 03, 2013 11:22:45 AM J Luis wrote:
> > HDF5 is now working on Windows, and it should find the correct dll
> > automatically.
>
> Take as an example the zlib lib. It's called zlib1.dll on Windows and
> libz.so on unix, so one cannot have a unique ccal instructions that works
> for both OSs. My point is that it's not guarantied that *nix and windows
> names are equal and differ only by extension.

Sure. See lines 22-40 of the latest HDF5/plain.jl, where it:
1. Uses different names for unix and windows ("libhdf5" for unix, "hdf5.dll"
for windows)
2. Does the right thing even if the user has both 32-bit and 64-bit
installations of HDF5 (dlopen_e allows you to check whether you can load a
particular library).

One thing this doesn't do is to handle installation for the user; there's
still a "navigate to a particular web page, click download, run the installer"
step in the HDF5 documentation.

Best,
--Tim

[J Luis]

Tim thanks for the pointer but what I'm trying to say is not specific to the HDF5 package where I had to modify that, because I use my own builds and they happen to have different names (a practice that I use in order to escape the dll hell).
I have noticed that several libraries come out with different names when they are compiled with MinGW, where the use of the 'lib' prefix is common, or by MSVC projects (where they tend to lack the 'lib' prefix).
Since there are no standard names for a same product on Windows I feel that it would be good to have a general mechanism to address this issue. I just don't know what it is.

Joaquim

[Tim Holy]

On Wednesday, July 03, 2013 02:44:48 PM J Luis wrote:
> Since there are no standard names for a same product on Windows I feel that
> it would be good to have a general mechanism to address this issue. I just
> don't know what it is.

If you know the collection of all possible names, it's easy to try them all
and stop when you get a hit.

If you're concerned that won't always work, then maybe here's a possible
strategy:
1. When the package is installed, have the package's build.jl search for the
library, and if found store the absolute path in a file.
2. Each time a user loads the package, the file storing the path is read, and
the path is used to load the dll.

That way, if you need to specify a custom location, you can do so (by editing
the file) without modifying code. This might not be a bad thing to have in
general, for cases where the user has multiple viable versions of the library
installed but only one of them works well with Julia code. One can test for
these things, but there's a certain security in having the result of the
automatic tests stored in a human-editable file.

--Tim

[J Luis]

One alternative that I thought in experimenting the next I need it is to just make a symbolic link with the name expected by the package. Perhaps it would be good if all packages that link to shared libs advertise the names that they are expecting. It would than be easy to please both the package and the local name/address. The drawback of this is that it leaves XP out the story.

[Jameson Nash]

symbolic links on Windows are a bit of a joke -- why did microsoft
think it was ok to require admin permissions for this operation?

I recently add two functions to Base for this purpose (#3364) with
usage as shown below:
1. push!(Sys.DL_LOAD_PATH, "/path/to/search/lib") -- for use by the
user (or package manager) to manually alter the search path
2. const libtk = Sys.find_library(["libtk8.6", "libtk86", "libtk",
"libtk8.5", "libtk85"]) -- for use by the developer to select the
first library name found out of a list of alternatives and return it

I find that MinGW prefers using the same names for libraries as MSVC.
Cygwin prefers the linux name, but we don't support that environment
at all.