Cross-compile

[Norman Maurer]

Hi there,

I started to use the nar plugin a few days ago and I really love it :) One thing that I was looking for was if it is possible to cross-compile for different archs.

Anyone know ?

[Curtis Rueden]

Hi Norman,

> I started to use the nar plugin a few days ago and I really love it :)

> One thing that I was looking for was if it is possible to

> cross-compile for different archs.

You can't do it out of the box, unfortunately.

It was briefly mentioned awhile back in a wishlist brainstorm thread:

https://groups.google.com/forum/#!msg/maven-nar/sdIP_s0qhs4/tJrFDiFPjRkJ

This feature is quite difficult, because cross-compilation is in general quite difficult. Relatively recent changes to GCC made cross-compiling with it much less feasible. I believe it much more doable using TinyCC (and not sure about LLVM), but still non-trivial.

Further, in the case of NAR, you would need the relevant tool chains installed on your system, and then we would need to teach NAR to detect and use them in the appropriate cases.

Patches very welcome. ;-)

Regards,

Curtis

P.S. In the meantime, my project solved this problem with a Jenkins farm that compiles our target AOLs on slave nodes, then aggregates the results back onto the master node for deployment. Rather complex, but it works. For grungy details:

http://trac.imagej.net/ticket/832#comment:11

http://jenkins.imagej.net/job/ImageJ-launcher

--
You received this message because you are subscribed to the Google Groups "NAR Maven plugin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maven-nar+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Brice Copy]

Hello Curtis,

I'm trying to cross-compile a 32 bits SO on a 64-bits Linux instance.

I already have pre-compiled NAR libraries for Linux 32 bits, I would simply like to download them and link to them with the GCC "-m32" flag.

Can you tell me how I can specify the NAR dependency in my POM to download the 32 bits dependencies, instead of the 64 bits ones ?

( I expect this is what you did in your project, from your description).

Thanks,

Brice

[Johannes Schindelin]

Hi Brice,

On Tue, 13 Aug 2013, Brice Copy wrote:

> I'm trying to cross-compile a 32 bits SO on a 64-bits Linux instance.
>
> I already have pre-compiled NAR libraries for Linux 32 bits, I would simply
> like to download them and link to them with the GCC "-m32" flag.
>
> Can you tell me how I can specify the NAR dependency in my POM to download
> the 32 bits dependencies, instead of the 64 bits ones ?
> ( I expect this is what you did in your project, from your description).

Have a look at https://github.com/imagej/imagej/blob/master/app/pom.xml
(search for 'ij-launcher' which is our native component).

We also "cross-compile" for i386 on x86_64 with our Jenkins server; this
is a bit tricky and depends on the platform (we do it for Windows, MacOSX
and Linux, and every platform requires a different way to compile the
32-bit version).

If you're interested to know that configuration, please let me know, I
will look it up when I am on a real computer again (not on a phone).

Ciao,
Johannes

[Johannes Schindelin]

Hi Brice,

please use reply-to-all in order not to exclude anybody! We're doing Open
Source here, everybody is entitled to participate.

On Tue, 13 Aug 2013, Brice Copy wrote:

> yes indeed, I'd like to see the config for cross compilation - I'm now
> playing about with the NAR's AOL properties, but I get various NPEs and not
> sure what could be wrong.

Okay, I will dig it up (still on my phone, spent almost an hour crafting
an important mail).

Ciao,
Johannes

[jonas.p...@gmail.com]

Hi Norman and all...

I have been trying to get cross-compile to work as well.
I can assure you that the problem is NOT that cross-compile is necessarily tricky. At least not from where Im standing. A pattern i've seen that should be easi enough to support is prefix-g++(.exe.). For instance I use arm-atollic-eabi-g++.exe and so on, the prefix is consisten for eferything else...

Now I would think that it should be possible to support this in nar-plugin but no. The code in cpptasks is far to over-designed do this easily.

I am also a little disappointed about the lack of integrations friendliness. Take a look at the aol.properties. I see a promising simple way of supporting anything with this as a base. Why must the code try to be overly and unmaintainably smart? (read ms-smart) THATS why the code is complex!

I am struggling with a decission right now. Do I attempt to support nar-plugin or do I look for alternatives? Even writing my own plugin seams like an idea.

But I am waiting for actual sign of a living nar-maven-plugin project still....

[Curtis Rueden]

Hi Jonas,

> I am struggling with a decission right now. Do I attempt to support

> nar-plugin or do I look for alternatives? Even writing my own plugin

> seams like an idea.

> 

> But I am waiting for actual sign of a living nar-maven-plugin project

> still....

The project is as alive as its contributors. What the project does not have right now is a full-time dedicated developer using it for a specific use case, making daily commits. But it does have a few of us committed to maintaining the project, reviewing submitted patches, etc. All of us are busy, but all of us do use NAR for something.

So really, whichever way you decide, it will be a self-fulfilling prophecy. If you contribute, the project is alive. And if no one contributes, it will lie dormant.

Therefore I strongly suggest basing your decision on whatever you think will truly work best for your project, save you time (in both the short and long terms). If you really believe NAR is "overly and unmaintainably smart" then perhaps creating your own plugin makes more sense. But if you think you could contribute a patch to address these shortcomings, then I would encourage you to do that. You would be benefiting not only yourself but also a larger community.

Regards,

Curtis

[Jonas Printzén]

Hi Curtis, thanx for responding....

... of cource you are right. I was hoping that no all of you where as me, to little time and lots of ideas :-)

Thing is I kind of put myself in a position. I started to use nar in a few private projects. And I loved it!! I didn't need anything but "vanilla compile" with minor flag-tuning so no problemo.

So, as I am a consultant, I choose to setup and display how smooth my scm-manager+jenkins+artifactory setup runs. Commiting, watch the blinking blobs wander and .... TADA! :-)

Then came the 'thing'. My customer needs to compile for windows using a mingw and arm32 (c3,c4,c9) using cross compiles. I tried to find a way to get profile to set PATH, no good and I tried fiddling with aol.properties. No good either, since cpp-tasks don't accept any name. (remember, I ruled PATH is static).

So now I have been looking at the code. (No gym tonight...:-)

I resorted to copying the gcc.Linker/Compiler/Librarian into gcc.arm32 and tuning some (peeked at sun-solaris-cross)

Im glad I did. I find myself with a working hack! (Yep , runns!) and a growing understanding of the cpptasks-/nar-plugin code.

I may actually end up cleaning up my hack and start providing...

So , just so you know, except for the cross-compile problem (bare_metal so nothing else works)

I Love the maven building my C++ code with nar-maven-plugin!!!

CU :-)

2013/9/12 Curtis Rueden <ctru...@wisc.edu>

[Johannes Schindelin]

Hi Jonas,

On Thu, 12 Sep 2013, jonas.p...@gmail.com wrote:

> I can assure you that the problem is NOT that cross-compile is
> necessarily tricky. At least not from where Im standing. A pattern i've
> seen that should be easi enough to support is prefix-g++(.exe.).

As you can see here:

http://jenkins.imagej.net/job/ImageJ-launcher/label=Windows/1047/consoleFull

we actually *do* have cross-compilation working, although it is "only"
MinGW-w64 from MinGW. We actually use the same compiler (gcc.exe) for both
32-bit and 64-bit builds, and the way MinGW-w64 is set up, the compiler is
a *cross* compiler. The only thing you need to do is put the appropriate
directory in front of PATH and you do not need the prefix.

Ciao,
Johannes

[Jonas Printzén]

Hi,

I would have to disagree. To be serious about cross-compiles the best case to

study is MIPS or ARP bare-metal compilers in windows.I use atollic and sourcery

based on gnu-gcc/g++. The commands are in both cases renamed with a prefix.

The architecture can then of cource not be detected by investigating the host you run at

but must be explicitly set. And the commands must always be prefixed with

"arm-none-eabi-" for sourcery variant.  This is also what many crosstools out there

does.

After digging in to the code for the commandLog setting I find myself less

intimidated by the (still over-designed :-) ) code. I think I can see a way to

simply add a prefix-tag to the Cpp and C configurations and passing them down to

the compilation.

Still waiting to see how last pull-req goes however ....

/Cheers -----------

2013/9/12 Johannes Schindelin <Johannes....@gmx.de>

[nathan...@gmail.com]

I am working on getting some (simple) cross compiling support in place - specifically the ability to compile iOS on OSX and Android on Linux. I have been working on a few patches, some of which have been merged - others which should be coming in the next little while.

Specifically, you may be interested in https://github.com/maven-nar/nar-maven-plugin/pull/53 (the ability to specify the path of the tool you want to use), and https://github.com/maven-nar/nar-maven-plugin/pull/45 (ability to specify a different OS rather than using os.name). Another one that hasn't been merged yet is https://github.com/maven-nar/nar-maven-plugin/pull/54 (skipping of nar-test when doing a cross-compile).

Any suggestions/pointers that you may have are welcome!

[sdma...@gmail.com]

I have followed the NAR project for a while now but never quite taken the leap to use it. I am fairly experienced with Maven for Java work, and appreciate hugely its promotion of small 'artifacts' as the unit of build/composition.

In my C work, we are cross-compiling for 32-bit arm processors (which run an embedded variant of Linux), using a arm-linux-gcc toolchain which runs on x86. So most of my work is cross-compilation, but it is also useful sometimes to build 'natively' ie for x86 (or x64 depending on my dev box) for some unit tests, etc.

Currently I do this mostly in make, using a directory structure stolen my my Maven experiences:

Makefile
src/main/include
src/main/c
src/test/c
target/arm/Makefile
target/native/Makefile

I then cd to say target/arm/ and run make. VPATH finds the source files (.c, .h) and the binary goes in pwd, i.e. target/arm. I can then cd ../native and build for native. Most targets are in the 'basedir' Makfile, with the ones in e.g. target/arm just driving which CC to use.

I would love to think that NAR would bring to me most/all the benefits of modularity of Java/Maven to my C build environment. Is this the case??

On a related note: I have never quite known the best mapping between Maven and Git. Should a single Maven artifact be stored in its own Git repo? Can Git sub-modules then accommodate Maven sub-modules?

Thanks

Stuart

[work...@gmail.com]

How did you get nar to work with mingw/mingw-w64. I need to compile under mingw-w64 but I haven't seen any examples anywhere. I looked at the aol.properties but not sure what/how to modify it.

[Adam Ringel]

How did you get nar to work with mingw?  I haven't been able to find a working example anywhere.  I am using mingw-w64 and I am assuming that you need to use a modified aol.properties but I can't find any documentation.

On Thursday, September 12, 2013 4:20:25 PM UTC-4, Johannes Schindelin wrote:

[Johannes Schindelin]

Hi Adam,

since I juggle a dozen of mail threads at each given moment, top-posting
makes it hard for me to get the context (while properly-quoted and
inline-answered mails would make it easy), therefore top-posted mails get
a lower priority in my inbox.

On Fri, 28 Feb 2014, Adam Ringel wrote:

> How did you get nar to work with mingw? I haven't been able to find a
> working example anywhere. I am using mingw-w64 and I am assuming that you
> need to use a modified aol.properties but I can't find any documentation.

There is no special change in aol.properties that we use. Rather, we
configure the flags in the pom.xml:

https://github.com/imagej/imagej-launcher/blob/master/pom.xml#L59

In particular, we pass the -m64 or -m32 flag, depending on the
architecture, to the compiler. Also, we make sure to configure the name
"gcc" in both <c> and <linker> sections.

I humbly suggest to clone the imagej-launcher project and build it
yourself with 'mvn install' with and without the '-Dos.arch=x86' flag
(after setting PATH and JAVA_HOME correctly). You should be able to strip
it down to a minimal example once you get it working.

Ciao,
Johannes

[Johannes Schindelin]

Hi,

On Thu, 27 Feb 2014, sdma...@gmail.com wrote:

> Currently I do this mostly in make, using a directory structure stolen
> my my Maven experiences:
>
> Makefile
> src/main/include
> src/main/c
> src/test/c
> target/arm/Makefile
> target/native/Makefile

I would personally not put things into target/ directly, but rather copy
artifacts there (AFAICT 'mvn clean' really deletes all of target/,
including any versioned files that might live there, marking them as
deleted).

There is support code for GNU autoconf projects in NAR, but I have no
first-hand experience with that (but IIRC there is even an integration
test about that somewhere in src/it/).

> I would love to think that NAR would bring to me most/all the benefits
> of modularity of Java/Maven to my C build environment. Is this the
> case??

Well... I would say that certain niceties are simply impossible in C world
that we take for granted in the Java world. For example, you will always
have awkward problems with GNU libc because it is *not*
forward-compatible: artifacts compiled against newer libc will *not at
all* work against older libc even if there is no technical reason that the
newer libc should be required.

With those caveats in mind, yes, NAR strives to bring as much of that
modularity to the C builds as possible.

> On a related note: I have never quite known the best mapping between
> Maven and Git. Should a single Maven artifact be stored in its own Git
> repo?

Personally, I worked with some projects where aggregate projects live in a
single source code repository together with their child projects.

However, that only makes sense if those child projects are really tightly
coupled, so recently I find myself factoring out more and more projects
into their individual repositories.

So: both work, you can have multiple Maven projects in the same
repository, or in separate repositories. It is a matter of taste and of
ease of process.

> Can Git sub-modules then accommodate Maven sub-modules?

You could. But why? I'd rather couple Maven projects by version numbers
and have tight release version couplings. That makes for reproducible
builds and for more development discipline, rewarding you with better
software in the long run.

Plus: looking at how submodules work, you cannot help but picture them as
the ugly, unwanted step child Git was forced to adopt but never wanted to.

Ciao,
Johannes

[Stuart Maclean]

Perhaps I didn't make it clear, my set up does NOT currently use maven, so the use of target/arm, target/native as directories in which to build platform-specific binaries does not collide with Maven builds, there is no maven.

I get the feeling, from this thread and this group (a very useful one!) in general, that the complexities of cross-compilation in the C domain are just too problem-specific to be shoe-horned into a maven world where a pom does everything, since on Java class files are being dealt with.

Stu

[Johannes Schindelin]

Hi Stuart,

please note that top-posted replies culling the Cc: list are used as
indicators by my mail workflow to de-prioritize reading it; feel free to
send a properly quoted mail with inlined comments for easy reading, with
the full Cc: list to give your mail a higher priority in my inbox.

Ciao,
Johannes