C code in a Julia package

[lapeyre....@gmail.com]

Can someone point me to examples of C code in Julia packages ? I am using BinDeps to download two C++ libraries. After a lot of blind trial and error I have a build.jl that does what I need. But, I need to write a c wrapper for the C++ libraries.  One idea is to have BinDeps handle the build, if I can fool it into thinking it is an external library. But, I don't know how to do that. BinDeps seems to want to download something. I find it rather opaque. Otherwise I need to find a way to write most of what BinDeps does by hand for my C code: find paths to top level of the package and various sub dirs; do change dirs, making, copying, linking at run time, etc. I spent quite a bit of time searching julia-user, and -dev, grepping and finding in package trees, reading docs and source code,  etc. no luck. Thanks.

--John

[Elliot Saba]

Hey John,

If you can explain a bit as to what you want to do with your libraries, that would help.  What I understand from you so far is that:

A) You have to C++ libraries that you are able to download and compile

B) You want to write a wrapper that sits between Julia and the C++, written in C but used by Julia

Is that correct?

-E

[John Lapeyre]

Hi Elliot,

Thanks for the offer.

 > B) You want to write a wrapper that sits between Julia and the C++,
   written in C but used by Julia

Yes. Writing the wrapper is not the problem. Really what I'd like to
know is how to organize, compile and link C code. How do I find the
toplevel of the package directory ? How does BinDeps find the
information to write into deps.jl ? Ideally, I would have BinDeps do
all of it, as it does for the downloaded code, because, then I don't
have to know any details. But the C code is stored locally, in the
package. It doesn't need to be downloaded.  I could easily solve the
problem by putting the .c file in another git repo so that it needs to
be downloaded. But, thats not the correct solution. Since BinDeps is
not really documented, I have been searching packages at random,
looking for examples for c code and uses of BinDeps.

--John

On 12/24/2014 06:37 AM, Elliot Saba wrote:

> Hey John,
>
> If you can explain a bit as to what you want to do with your libraries, that would help.  What I understand from you so far is that:
>
> A) You have to C++ libraries that you are able to download and compile
>

>

[Elliot Saba]

BinDeps is pretty flexible.  I'm guessing you have seen a BinDeps invocation that is based off of the Autotools step?  (Here's an example for Nettle.jl)  That does a lot of things for you, and doesn't really show off the flexibility that well.

You can do things in a much more fine-grained manner, here's an example for NLopt, where zip files are downloaded, extracted, and .dll's are copied.  I would suggest putting your C code and a Makefile in a PACKAGE_ROOT/deps/src folder, then you can write up a set of simple build steps to cd into that directory, (Notice BinDeps.srcdir(dep) will give you paths like you were asking above) you can run `make`, and then install the binary anywhere you want.

I agree that BinDeps isn't very well documented.  Any effort you can spare in that direction would be very appreciated, as you learn more about it and can help explain to others how to use it.

-E

[John Lapeyre]

Thanks. Seeing a list of packages that use BinDeps helps. In
retrospect, I could have put BinDeps in the subject line so this
thread could be found more easily.

The two downloaded packages use Autotools. BinDeps works great once I got the incantation right.
So, I have been trying to follow the BinDeps test code in faking that the wrapper code (one file) uses Autotools by supplying a
configure script, so that ./configure, make , make install works. I was hoping to essentially copy the two or three lines per
dep that are in my current build.jl.

Anyway, this is probably enough help to get me quite a bit further.

--John

[John Lapeyre]

On 12/24/2014 07:22 PM, Elliot Saba wrote:

> I would suggest putting your C code and a Makefile in a
>  PACKAGE_ROOT/deps/src folder, then you can write up a set of simple

btw. that's what I already did. But Pkg.build("PrimeSieve") still fails with
this error:

  Could not obtain sources for dependency cprimecount
  while loading /home/jlapeyre/.julia/v0.3/PrimeSieve/deps/build.jl

I have been trying to supply some kind of noop for provides(Sources,...,
since the source is already in place. No luck yet. I'll read the other
build.jl examples.

--John

[Elliot Saba]

Autotools without a ./configure script can be done by using the MakeTargets buildstep generator.

Here's an example for Cairo.jl, where the zlib dependency is built by downloading the sources for Zlib, then running make on the win32/Makefile.gcc makefile.  If you have a generic Makefile file, you should be able to omit any arguments to the MakeTargets generator.

-E

[John Lapeyre]

I think the BuildProcess step will not be a problem. I am almost
certain I can just say its plain vanilla Autotools. If not, there
are many examples in packages of complicated BuildProcess arguments.

In all of the examples I have seen (maybe I missed something), the
source is downloaded. My source is not downloaded, its already there.
BinDeps never gets to the BuildProcess part in my case because it does
not know how to find the source, and I don't know how to tell it that
the source is already there. It fails before it ever tries to build
anything.

That's also why I didnt put BinDeps in the subject line. This is
not really an external dependency. But it seems natural to include it
with the external library dependencies, since it is just a small piece
of wrapper code around them.

--John

[Elliot Saba]

Do you have your code posted anywhere so I can try it locally?

-E

[John Lapeyre]

On 12/24/2014 08:32 PM, Elliot Saba wrote:
> Do you have your code posted anywhere so I can try it locally?

Well, I just pushed the branch here:

https://github.com/jlapeyre/PrimeSieve.jl/tree/addprimecount

But it won't work as is. I have to bundle the Makefile back
into the configure script, etc. If I build the wrapper library
by hand and write the deps.jl by hand, then the module loads and
I can call the two external libs via Julia via the wrapper... Uh sorry,
that's confusing.

--John

[John Lapeyre]

Ok, I pushed an update to the branch:

https://github.com/jlapeyre/PrimeSieve.jl/tree/master

In this dir, "configure && make && make install" do what I want:

https://github.com/jlapeyre/PrimeSieve.jl/tree/addprimecount/deps/src/cprimecount

After building by hand, this handwritten deps.jl works:

https://github.com/jlapeyre/PrimeSieve.jl/blob/addprimecount/deps/deps.jl.save

This is the build.jl that fails. If I remove all of the cprimecount stuff,
it does download, build, and install the two external libraries:

https://github.com/jlapeyre/PrimeSieve.jl/blob/addprimecount/deps/build.jl

--John

On 12/24/2014 08:32 PM, Elliot Saba wrote:

[John Lapeyre]

Reading the BinDeps code, it seems that, unlike for Binaries, there is
no other way to provide the source.

https://github.com/JuliaLang/BinDeps.jl/blob/master/src/dependencies.jl#L261-L263

I also see that in a handwritten Provides(SimpleBuild line...), I explicitly ask
to get the source:

https://github.com/JuliaOpt/Ipopt.jl/blob/master/deps/build.jl#L19-Lundefined

So, I'll try to copy one of these and omit GetSources. I lose the automation, but
if it works, its ok.

--John

On 12/24/2014 08:32 PM, Elliot Saba wrote:

[John Lapeyre]

Thanks for the help Elliot. My entire build works now.

For the record, this is the code that differs from that for
a basic build of an external library with autotools:

srcdir = BinDeps.srcdir(cprimecount)
srcdir = joinpath(BinDeps.depsdir(cprimecount),"src","cprimecount")

provides(SimpleBuild,
    (@build_steps begin
        @build_steps begin
            ChangeDirectory(srcdir)
            `./configure`
            `make`
            `make install`            
        end
    end),cprimecount, os = :Unix)

I't just like the code for Ipopt and other packages except I omit
"GetSources". I didn't realize that the Build process was also trying
to download. I thought this was happening at an earlier stage.

--John

On 12/24/2014 08:32 PM, Elliot Saba wrote:

[Elliot Saba]

Great.  So glad that we could get it working.

-E