JavaCPP and Maven

[Ryan Moquin]

Hi, I ran across JavaCPP the other day and thought it looked like an interesting approach.  I wanted to give it a try to bridge an open source c++ library.  My question is how to use the JavaCPP plugin.  I have been looking through the bytedeco repos to see if I can find a usage of the plugin to help me to configure it properly.  I did find a pom that has it declared, but no configuration.  I obvious missed it somewhere, so I was hoping maybe someone could point me in the right direction.  I'm sure now that I asked for help, I'll magically run across what I need the next time I look, which usually ends up being the case.

Thanks for any help!

Ryan

[Samuel Audet]

Hello,

I think what you are looking for is right here:
https://github.com/bytedeco/javacpp-presets/wiki/Create-New-Presets

Let me know if you have any questions regarding that!

Samuel

[Ryan Moquin]

Thanks Samuel!  I totally missed the wiki on github.  I was looking at it on googlecode, but since github had been moved around, I usually forget it's there!

I do have everything hooked up in Maven the best I can tell, unfortunately it seems like I'm not getting the Java classes processed by JavaCPP.  I can hopefully figure out what I'm missing on the Wiki.

Thanks!

Ryan

[Ryan Moquin]

Hi Samuel,

Coincidentally, a big part of my problem (as embarrassing as it is to admit this mistake) was that since I was experimenting with javacpp (and wasn't sure how to use the auto header parsing, even after reading the wiki, which I'll explain in a second) I was using my own com.* namespace.  Since I'm so used to typing groupIds in Maven as org.* (because of apache deps) I had typed my package names in the poms as org.* and in the javacpp annotated class it was com.*.  This confusion was furthered by some potential bugs in the javacpp Maven plugin, which I think are affecting the 0.9 and head code.

Now, I know you'll want me to create bug reports for those, but I am positive I can quickly fix them for you and open a pull request as well create an issue (plus I want to be sure of the cause first).  I believe I can also alleviate the need (or most of the need) for having a separate sh script.  I should be able to do this very quickly (I've been using Maven for years and love it, I've written plugins and done lots of advanced stuff).

Besides contributing some potential fixes and improvements to javacpp that I know I can make, my original problem is that I'm trying to generate wrappers for this project:

https://github.com/richards-tech/RTIMULib/

Particularly I've been working on just getting the core RTIMULib to work (https://github.com/richards-tech/RTIMULib/tree/master/RTIMULib), the other parts of the repo aren't libraries, but helper/visualization applications that can be run separately.  After a few days of trying to understand how the maven plugin is working (which was a bit confusing because sometimes the plugin thinks it can't find the javacpp annotated source file when it should be able to for some reason), I was able to get javacpp to generate a Java file from the RTIMULib headers and then a cpp file from that.  The cpp file has quite a few errors in it.  I removed a public native void zero() function that was generated from something in the headers (not sure what yet) because the compiler wouldn't compile it, there was also a c++ RTFloat type that is aliased to "float" which wasn't mapped correctly.  After manually trying to fix those so I could see what happens, I started getting some c++ parsing errors and some other c++ related errors that don't look overly complicated to fix, but my c++ is extremely rusting after having not used it in quite a lot of years.

Do you think you might be able to see if you can give me a hand with that library?  Since I'm stuck, I'm going to figure out these javacpp issues I'm running into and hopefully you can check out that other project and see if it is something easy to fix or not (I'm honestly not trying to make you do my work).  I figure right now it would be more productive for me to make these contributions I know I can make quickly and see if you can give me a hand with that other library.  Also, that other library is very cool and I think having it added to your presets will good as well.

Sorry this email is so long, I'm going to work on making these javacpp improvements and submitting them.  I can push the project I have for wrapping RTIMULib, but it's nothing special.  The only main differences are what I've added to my maven pom to do what the .sh seems to be doing (other than the initial compile, but I can probably add something in for).

Let me know what you think!

Hopefully I can make these fixes and changes and have them ready later today (I hope).

Ryan

[Ryan Moquin]

Also, forgot to mention, after I makes these maven improvements, I can write up some nice documentation that will hopefully help to clear up some confusing things with wrapping libraries.  I think this project is really cool and definitely think it's worth helping to further it.

I'll get working on that stuff now so I can get it submitted to you.

Ryan

[Samuel Audet]

Better documentation, bug fixes, sure that's all very welcome :) Even if the functionality isn't final, you can send pull requests, and we can discuss the changes there, and make some adjustments before merging them. Please target the `develop` branches though because that's where I'm making changes between releases.

Of course you're not expected to know how to fix all the errors that show up when trying to create new presets. That's why we have this mailing list :) It's very hard to have a generic bridge between Java and C++, but we can create mappings without too much effort for most existing libraries, and by creating "presets" like this, once they get created, they can be used by everyone easily. That's the whole point. So yes, please send your questions and let's get RTIMULib in there for the benefit of all! Thanks

Samuel

[Ryan Moquin]

Hi Samuel,

Thanks!  I forked javacpp and have pushed a bunch of changes to it in my rmoquin/javacpp repo.  I upgraded it to the Maven 3 plugin api which is much nicer and I doubt there is many using 2.x anymore.  There were a lot of important improvements and bugs fixed in 3.x.  Anyhow, I also correct the git urls in the pom and some other random stuff.  I made a bunch of random improvements, none of which were the main change I wanted to make.  It runs fine if you want me to put in a pull request with the changes there so far (I can branch and put a pull off of that to make it easy when I make more changes).

Anyhow, was able to figure out how to use the InfoMap to fix the two java compile issues, the first use the RTFLOAT which is an alias for float, wasn't being substituted correctly in all places and I had to have it rename the zero() function to jzero(), since I figured out zero() is a method in the Pointer parent class. :)

What I'm stuck on now, is I just can't seem to figure out how to get the plugin to create the cpp files, compile it and jar it.  If I run it and have it output the source to src/main/java, then it only creates the Java class.  If I run it again, then I get the cpp files.  I can't seem to figure out where those things happen in the plugin (the code is a little tricky to follow currently, so figure I'll just ask :))  The one thing is that in Maven, you shouldn't have a plugin generate code outside of the target directly.  If I can figure out how the plugin does those things in the code (I just need enough to know where it is, I can figure out the rest), I'll make the changes to make it do it the maven way, which along with some other things I was going to propose to make the process much cleaner and easier (hopefully you'll agree :)).  

Let me know.  I want to try to get this easiest enough to understand so that people can very quickly and easily create these wrapped libraries without much ramp up time.  That'll help to encourage people to do it.  Let me know, I'm anxious to get this stuff running :)

BTW - If I do have it generate the cpp and try to compile it on a linux machine (my windows machine isn't setup right), I get a ton of errors, here are the first bunch I'm not sure how to solve yet:

[INFO] g++ -I/root/rtimulib-jni/target/cpp-src/ -I/home/root/jdk1.8.0_25/include -I/home/root/jdk1.8.0_25/include/linux

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp -Wl,-rpath,$ORIGIN/ -march=i686 -m32 -Wall -O3 -fPIC -shared -s -o

/root/rtimulib-jni/src/main/java/libjniRTIMUWrapper.so -L/root/rtimulib-jni/target/cpp-libs/ -Wl,-rpath,/root/rtimulib-j

ni/target/cpp-libs/ -llibRTIMULib

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp: In function ‘void JavaCPP_com_rkmoquin_rtimu_00024LSM9DS0_1CACHE_1

BLOCK_deallocate(void*)’:

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp:260:97: error: ‘::LSM9DS0_CACHE_BLOCK’ has not been declared

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp:260:119: error: expected primary-expression before ‘)’ token

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp:260:120: error: expected ‘;’ before ‘p’

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp: In function ‘void JavaCPP_com_rkmoquin_rtimu_00024GD20M303DLHC_1CA

CHE_1BLOCK_deallocate(void*)’:

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp:262:102: error: ‘::GD20M303DLHC_CACHE_BLOCK’ has not been declared

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp:262:129: error: expected primary-expression before ‘)’ token

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp:262:130: error: expected ‘;’ before ‘p’

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp: In function ‘void JavaCPP_com_rkmoquin_rtimu_00024GD20HM303D_1CACH

E_1BLOCK_deallocate(void*)’:

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp:264:100: error: ‘::GD20HM303D_CACHE_BLOCK’ has not been declared

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp:264:125: error: expected primary-expression before ‘)’ token

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp:264:126: error: expected ‘;’ before ‘p’

/root/rtimulib-jni/src/main/java/jniRTIMUWrapper.cpp: In function ‘void JavaCPP_com_rkmoquin_rtimu_00024LSM9DS0_1CACHE_1

BLOCK_deallocateArray(void*)’:

Any hints?  Thanks!

Ryan

[Ryan Moquin]

Actually, I realized that by default, I think only the mpu9150 support is compiled in unless you change a define.  I took out the other device headers and left that in.  It appears the code compiles now, the linking fails not able to find the compiled library, so I gotta figure that out.  I hope it works when I do, curious to see if this works.

Also still want to find out how the plugin invokes each thing to try to get it all working in the same maven build...

Ryan

[Ryan Moquin]

Ok, the linking problem was solved by poking through properties in generic.properties in javacpp.. apparently the library being linked needs to be prefixed with lib.  So compiling the jni would look like -lMyLib for a library file named libMyLib.so.

Wonder if it works now.

Ryan

[Samuel Audet]

Hi,

Sorry for the late reply. I have been rather busy lately.

On 11/30/2014 12:44 AM, Ryan Moquin wrote:
> Hi Samuel,
>

> Thanks! I forked javacpp and have pushed a bunch of changes to it in my
> rmoquin/javacpp repo. I upgraded it to the Maven 3 plugin api which is
> much nicer and I doubt there is many using 2.x anymore. There were a
> lot of important improvements and bugs fixed in 3.x. Anyhow, I also
> correct the git urls in the pom and some other random stuff. I made a
> bunch of random improvements, none of which were the main change I
> wanted to make. It runs fine if you want me to put in a pull request
> with the changes there so far (I can branch and put a pull off of that
> to make it easy when I make more changes).

Sounds good :) Like I said, we can discuss the changes on GitHub, and
make necessary modifications there, before merging any pull requests.

So, let's see, the first thing you'd need to fix is to reapply your
changes to the 'develop' branch. Let me know when you're done, thanks!

> Anyhow, was able to figure out how to use the InfoMap to fix the two
> java compile issues, the first use the RTFLOAT which is an alias for
> float, wasn't being substituted correctly in all places and I had to
> have it rename the zero() function to jzero(), since I figured out
> zero() is a method in the Pointer parent class. :)

Great!

> What I'm stuck on now, is I just can't seem to figure out how to get the
> plugin to create the cpp files, compile it and jar it. If I run it and
> have it output the source to src/main/java, then it only creates the
> Java class. If I run it again, then I get the cpp files. I can't seem
> to figure out where those things happen in the plugin (the code is a

We basically need to have everything shown in the sample here to have it
work properly:
https://github.com/bytedeco/javacpp-presets/wiki/Create-New-Presets#the-pomxml-file

> little tricky to follow currently, so figure I'll just ask :)) The one
> thing is that in Maven, you shouldn't have a plugin generate code
> outside of the target directly. If I can figure out how the plugin does
> those things in the code (I just need enough to know where it is, I can
> figure out the rest), I'll make the changes to make it do it the maven
> way, which along with some other things I was going to propose to make
> the process much cleaner and easier (hopefully you'll agree :)).
>
> Let me know. I want to try to get this easiest enough to understand so
> that people can very quickly and easily create these wrapped libraries
> without much ramp up time. That'll help to encourage people to do it.

I know that we should put everything in `target`, but if we do that, the
`helper` classes in `src` depend on them, and because they would get
deleted on `clean`, the project won't build unless the sources get
generated on each build, so putting generated sources in `target` make
things more complicated... If you figure out a better way to do all that
though, by all means, please let me know, thanks!

Everything concerning the plugin is defined in the pom.xml file, so take
a look there, and let me know if you have any questions concerning that.

> Let me know, I'm anxious to get this stuff running :)

Good, keep your spirits up like that! It will take some time to get used
to everything, so please be patient. It will eventually pay off :)

> Any hints? Thanks!

It seems like `LSM9DS0_CACHE_MODE` isn't getting defined. Adding
`@Platform(define="LSM9DS0_CACHE_MODE", ...)` annotation should fix
that, but if it's not functionality that is intended to be used by a
user, we would instead do something like `infoMap.put(new
Info("LSM9DS0_CACHE_MODE").define(false))` to let the `Parser` know it's
not something we're interested in.

> Ok, the linking problem was solved by poking through properties in generic.properties in javacpp.. apparently the library being linked needs to be prefixed with lib. So compiling the jni would look like -lMyLib for a library file named libMyLib.so.

Yes, that's the standard naming convention on Linux. It's defined here
on this line in the case of Linux x86, for example:
https://github.com/bytedeco/javacpp/blob/master/src/main/resources/org/bytedeco/javacpp/properties/linux-x86.properties#L20

So, is it working now? :)

Samuel

[Samuel Audet]

On 12/09/2014 10:04 AM, Ryan Moquin wrote:
> No problem, everyone gets busy. :) I did get it all to work, which is
> really cool. I'm impressed since this sort of thing isn't easy. None
> of the other libraries that do this seem to be active anymore. I think
> having the presets libraries is great since it prevents people from
> having to refigure out how to wrap those libraries, especially the more
> complicated ones.

Yes, pretty much everyone else has given up on making other languages
interoperate with C++. I hope we can change things...

> Usually the master branch is the one currently under the most developent
> (which if someone . If you are using the develop branch as the new
> development branch, then I can easily tag what I've and issue a pull
> request to that branch.

I used to do that, but on GitHub, the README.md file from the master
branch pretty much becomes a home page, so it's inconvenient having
incomplete info displayed there. Unless you know a way to work around
that, having a develop branch is pretty common:
http://nvie.com/posts/a-successful-git-branching-model/
aka "gitflow".

> You can certainly make all the suggestions and corrections you want to
> any of the code I submit the pull request for to make sure you are happy
> with it.

Looks good in general, thanks! A few things to fix though:

1. It doesn't merge cleanly with the develop branch.

2. There are many aesthetic changes. Single-line annotations and Javadoc
comments are perfectly acceptable:
https://google-styleguide.googlecode.com/svn/trunk/javaguide.html
For new code, I don't mind, but let's keep formatting as it is for
existing code.

3. Are there any benefits in using the latest versions of all Maven
plugins? If the old ones work fine, I'm not sure I want to change them
at this point. If isn't broken, please don't fix it :)

Let me know when you're done with that and I'll recheck more thoroughly,
thanks!

> I have basically the equivalent in my pom of what's in the presets.
> I'll get it committed shortly so you can see.

Great! Please make sure to prepare a cppbuild.sh script for all
supported platforms, if possible, thanks.

> The way Maven works is that it it defaults to a lifecycle that is
> ordered properly. Meaning that the reason they discourage generating
> code into the source directory, because plugins have everything they
> need to generate the sources in target and have it available to compile
> against before the code in source gets compiled. I'll make the
> modificarions to the plugin and put them in a separate branch so you see
> what I'm talking about. It's not obvious if you haven't come across
> plugins that do it. No worries though since I'll show you how plugins
> are meant to do this and see what you think.

No need to prepare a branch, I get your point, but there are two
problems with that approach:

1. We need the C/C++ header files just to regenerate the Java source
files. Failing that, some Java classes won't compile at all. So users
have to install everything from scratch, even when the native libraries
are fine.

2. The source code gets removed on clean, and that doesn't play well
with Git.

If you know of a better way to handle those two cases, I'm all ears!

> Even though you wrote and published the code doesn't mean they have any
> obligation to respond to questions within any time frame if at all. If
> the person does, then lucky me. Otherwise, until they respond, I assume
> that if I really want to use that project that bad then I need to be
> prepared to figure it out myself. That's just the way open source
> works, no need to apologize.
>
> I got it working, that's all that matters.

Perfect!

> Anyhow, I couldn't have written that project myself anytime soon, but I
> can at least contribute code that should help improve it (provided you
> agree with my proposals of course :) ). I had my hard drive die on me,
> so I'm getting everything set back up and then I'll get those pull
> requests opened. :)

Sounds great, take your time! But I'd like to make a release soon, so
let's try to get the important stuff in first, thanks!

Samuel

[Samuel Audet]

Hi, Ryan,

On 12/09/2014 10:41 PM, Samuel Audet wrote:
> 2. There are many aesthetic changes. Single-line annotations and Javadoc
> comments are perfectly acceptable:
> https://google-styleguide.googlecode.com/svn/trunk/javaguide.html
> For new code, I don't mind, but let's keep formatting as it is for
> existing code.

Just wanted to clarify this here. I don't mind either that the style
changes a bit when the code or the comments are modified or corrected in
some way. Thanks for all those corrections. Let's get that and the rest
to merge cleanly with the develop branch. Thanks!

Samuel

[Samuel Audet]

Hi, Ryan,

BTW, any reasons to exclude the mailing list from the reply?

On 12/12/2014 11:56 AM, Ryan Moquin wrote:
> Interesting article. I agree with some of the things he's saying, but
> it feels like the develop branch is being used as the master, and the
> master branch is used to reflect the latest release tag, which will be
> the same as the topmost release on the "Releases" page on Github. I'm
> not really sure the advantage of develop over master. If you really
> want develop as the default development branch, then you can go into the
> repository settings and change which branch is the default. Having the
> default branch always tagged as the latest release will confuse most
> people into thinking the project isn't active, IMHO, unless they know
> the development branch isn't master.
>
> I think what I'm not convinced of is that basically you have to do two
> merges from feature/release branches to the develop and master
> branches. Then if the branch was a feature branch, you simply delete
> it. If it was a master branch, you delete it, then you recreate the
> release from Master. Maven will also handle the release creation, by
> automatically updating the versions, tagging the release, updating the
> "trunk" (whichever one that is) with the next snapshot version, and
> deploying the release artifacts to the chosen maven repository. Even
> better, use jenkins to invoke the maven release.
>
> Anyhow, that's just my opinion on it. Maybe I don't full understand
> what he was saying.

Sure, I don't really care about the develop branch myself. I'd be happy
with master as default for development too. That's not an issue. My
issue is with the README.md files...

Jenkins and stuff, sure that'd be great! If you figure out a way to make
this work well across platforms, let me know! Ideally, I'd like to make
use of free services like Travis CI. Someone started to look at that not
so long ago, but I haven't been receiving updates recently:
https://github.com/bytedeco/javacpp-presets/issues/22
https://github.com/bytedeco/javacpp-presets/issues/26

> I'm not sure what you mean by the readme.md <http://readme.md/> being
> incomplete in master? Most repositories I've seen fully use the
> readme.md <http://readme.md> to give an intro to the project, what it's
> about, and anything else useful someone might want to know about the
> project.

The version numbers change, so the links with Maven central are not
consistent, and sometimes the API changes a bit, and the few lines of
sample code in there are not in sync with the release code. What can we
do about that? Are people looking for something "stable" and for
consistent information used to look into the non-default branch?

> I didn't work off the development branch, so the commits I started my
> changes from are, I forked the master branch since I didn't realize you
> use a develop branch. I thought you just had done any development since
> the last release. I can make my fork compa

Yes please do! Thanks

> The benefits of using the latest plugin api is basically the same as
> using the latest api for any project (real annotations instead of "fake
> annotations using javadocs"). There are lots of improvements that can
> be leveraged, but since the old plugin apis seem to work ok, I'll leave
> it as it was. Those changes weren't made to fix anything, I considered
> it to be an enhancement.

Well, they would need to be tested a little bit. We can do that after
the next release, and try to hone out any bugs that occur then.

> The code I was wrapping was only built for embedded linux platforms, so
> that's the only platform that it supports.

Sure, that's fine then. BTW, does it build `linux-arm`?

> The users do not have to install any of them in order to compile against
> them using Java classes. They just simply have to exist somewhere
> accessible. That's one of the problems Maven has a preferred way to
> solve this sort of thing. It's better if the users don't have to have
> them installed, since that's opening up a whole world of trouble when
> that person inevitable runs into issues building and installing the
> libraries. A perfect example is the Node.js stuff. It's all cute that
> when I install npm modules on linux, they magically succeed. When I try
> to install them on Windows, I get all kinds of confusing errors because
> I don't know what has to be setup manually on windows to be able to
> build and install the npms. I don't want to start myself on that
> tangent, just wanted to give an example.

Of course ideally users don't need anything to use the artifacts. But if
there is a bug in some class, and they want to fix it, and that change
doesn't affect anything w.r.t the native libraries, even if we were to
rebuild everything, isn't it better to offer a way to rebuild the Java
class files, without needing all the native stuff?

> It's not that it doesn't play well with Git, it's a maintenance headache
> for any SCM, Regardless, the source code will get cleaned, and then it
> will get generated again like it did the first time.

Yes, it /usually/ gets generated the same... Anyway, if we put the
generated Java code in `target`, any proposals to make this less of a
maintenance headache for SCM?

> You'd have to tell me what you consider to be the important stuff. In
> my own fork, I'm just making whatever modifications I feel like since
> they are isolated and then if there is anything you would like to have
> in the original codebase, The only things I think I actually fixed were
> a couple things in the pom, so that's the only thing that might be
> helpful, but those changes really won't be helpful either since you
> aren't leveraging Maven for release management.

I was hoping /you/ could tell me what is important. :) If there is
nothing in particular that you feel should be in right away, then it can
wait, yes... Let me know!

> Before the code would be pulled back into your repo, I would go through it to match your code as much as possible.. When I'm in a fork I made, I tend to make changes as I go along, I do make a lot of trivial changes, but nothing says you have to take everything (or anything) I've done.
>
> When I make changes certain style changes, it's not because the way it was is wrong. I just make changes based upon my preferences. The Google doc you reference above is simply how Google expects internal Google code to be formatted. There is nothing wrong with how they format comments or single line annotations. I usually remove annotations that are only there because of Eclipse because it used it's own compiler and it complains about everything by default to the point that source code gets cluttered up with annotations that serve no purpose other than to get Eclipse to shut up. It feels like a waste of space to have unnecessary annotations in the code (if an annotation is to ignore a warning, then the code should be changed to not generate the warning rather than use an annotation.)
>
> Java comments use the C and C++ formats, which means either using a multiline capable comment designator:
> /*
>
> */
>
> or single line comment:
>
> //
>
> There is nothing wrong with adding extra characters to multiline comment blocks to your hearts content. Since IDEs and decent text editors change the styling of comments in the source code, I find all the extra decorations unnecessary and distracting. That's only my preference and doesn't mean I expect anyone to agree with me. It's just my preference.You can also use javadoc style comments also, it just feels unnecessary as well since you only need to use that format for generating javadocs. If nothing useful is put in the javadoc comments, there isn't any point in even putting javadoc comments (not saying you did that, just elaborating on my view).

That's alright. I wasn't criticizing anything, I am just wary of making
changes just to make changes, that's all :)

Yes, I could remove changes that I consider unnecessary before merging
them with my repo, sure, but then we can't use GitHub's pull request
feature. It forces us to merge everything, so we have to ask the person
sending the pull request to modify the commits appropriately before
making the merge, that's all. Unless you know of some way to modify
someone else's fork before merge?

Samuel

[Samuel Audet]

Hi, Ryan,

It's done, I've released version 0.10! Now let's get your changes in and
test the hell out of them before the next release :)

Following your advice, I'll be using the master branch as target branch
for all changes, and I think I'll add a README_latest.md or something
for changes to the README.md file that would otherwise be confusing
w.r.t the current release.

So, please rebase the fork you have with the master branch, thanks!

Samuel