So I needed to compile some stuff that uses native libs (penumbra and
an undisclosed, totally crazy thing I'm working on) and since imports
of the java libs for certain jni libs can cause the jvm to try to load
the native libraries, one needs to set java.library.path correctly
during the compile step.
I forked leiningen to try to add support for setting java.library.path
in the compile task and I seem to have succeeded. The ant java task
needed to run forked for it to work. I don't know what other
implications that has, maybe it's a bit slower, but at least it works.
Since native libraries are os and architecture dependent I made up a
little naming scheme for directories where leiningen looks for native
libs (unless manually specified in defproject via :native-path).
Under the project root, put native libraries in native/windows/32 if
they are for 32 bit windows, native/windows/64 for 64 bit. OS dirs for
linux and mac os x are called linux and macosx respectively, otherwise
the dir structure should be the same. This way you can keep libs
around for each of the three big platforms and for each architecture
while avoiding name conflicts.
Only relevant changes I made are in compile.clj. I also changed
core.clj but only to provide a default value for the native path key.
Since my modification is to the eval-in-project function it should
also work with tasks that depend on that. But I have not tested this
yet.
So there it is if anyone finds this sort of thing useful.
On Mon, Dec 14, 2009 at 3:44 PM, mac <markus.gustavs...@gmail.com> wrote:
> So I needed to compile some stuff that uses native libs (penumbra and
> an undisclosed, totally crazy thing I'm working on) and since imports
> of the java libs for certain jni libs can cause the jvm to try to load
> the native libraries, one needs to set java.library.path correctly
> during the compile step.
> I forked leiningen to try to add support for setting java.library.path
> in the compile task and I seem to have succeeded. The ant java task
> needed to run forked for it to work. I don't know what other
> implications that has, maybe it's a bit slower, but at least it works.
> Since native libraries are os and architecture dependent I made up a
> little naming scheme for directories where leiningen looks for native
> libs (unless manually specified in defproject via :native-path).
> Under the project root, put native libraries in native/windows/32 if
> they are for 32 bit windows, native/windows/64 for 64 bit. OS dirs for
> linux and mac os x are called linux and macosx respectively, otherwise
> the dir structure should be the same. This way you can keep libs
> around for each of the three big platforms and for each architecture
> while avoiding name conflicts.
I wonder if this directory structure is really the best? Does it perhaps
make sense to support providing an fn which returns the proper
java.library.path? This way the directory path is flexible and we take
advantage of the fact that project.clj is really just some Clojure code.
> Only relevant changes I made are in compile.clj. I also changed
> core.clj but only to provide a default value for the native path key.
> Since my modification is to the eval-in-project function it should
> also work with tasks that depend on that. But I have not tested this
> yet.
> So there it is if anyone finds this sort of thing useful.
Ok this works perfectly for me. I can run lein-swank and connect a remote
repl and run Penumbra examples. One thing that should probably be changed is
that :native-path currently needs to be absolute instead of following the
established Leiningen pattern that relative urls may be used. Were you
planning on fixing this?
On Mon, Dec 14, 2009 at 3:44 PM, mac <markus.gustavs...@gmail.com> wrote:
> So I needed to compile some stuff that uses native libs (penumbra and
> an undisclosed, totally crazy thing I'm working on) and since imports
> of the java libs for certain jni libs can cause the jvm to try to load
> the native libraries, one needs to set java.library.path correctly
> during the compile step.
> I forked leiningen to try to add support for setting java.library.path
> in the compile task and I seem to have succeeded. The ant java task
> needed to run forked for it to work. I don't know what other
> implications that has, maybe it's a bit slower, but at least it works.
> Since native libraries are os and architecture dependent I made up a
> little naming scheme for directories where leiningen looks for native
> libs (unless manually specified in defproject via :native-path).
> Under the project root, put native libraries in native/windows/32 if
> they are for 32 bit windows, native/windows/64 for 64 bit. OS dirs for
> linux and mac os x are called linux and macosx respectively, otherwise
> the dir structure should be the same. This way you can keep libs
> around for each of the three big platforms and for each architecture
> while avoiding name conflicts.
> Only relevant changes I made are in compile.clj. I also changed
> core.clj but only to provide a default value for the native path key.
> Since my modification is to the eval-in-project function it should
> also work with tasks that depend on that. But I have not tested this
> yet.
> So there it is if anyone finds this sort of thing useful.
On Mon, Dec 14, 2009 at 5:35 PM, David Nolen <dnolen.li...@gmail.com> wrote:
> Ok this works perfectly for me. I can run lein-swank and connect a remote
> repl and run Penumbra examples. One thing that should probably be changed is
> that :native-path currently needs to be absolute instead of following the
> established Leiningen pattern that relative urls may be used. Were you
> planning on fixing this?
> On Mon, Dec 14, 2009 at 3:44 PM, mac <markus.gustavs...@gmail.com> wrote:
>> So I needed to compile some stuff that uses native libs (penumbra and
>> an undisclosed, totally crazy thing I'm working on) and since imports
>> of the java libs for certain jni libs can cause the jvm to try to load
>> the native libraries, one needs to set java.library.path correctly
>> during the compile step.
>> I forked leiningen to try to add support for setting java.library.path
>> in the compile task and I seem to have succeeded. The ant java task
>> needed to run forked for it to work. I don't know what other
>> implications that has, maybe it's a bit slower, but at least it works.
>> Since native libraries are os and architecture dependent I made up a
>> little naming scheme for directories where leiningen looks for native
>> libs (unless manually specified in defproject via :native-path).
>> Under the project root, put native libraries in native/windows/32 if
>> they are for 32 bit windows, native/windows/64 for 64 bit. OS dirs for
>> linux and mac os x are called linux and macosx respectively, otherwise
>> the dir structure should be the same. This way you can keep libs
>> around for each of the three big platforms and for each architecture
>> while avoiding name conflicts.
>> Only relevant changes I made are in compile.clj. I also changed
>> core.clj but only to provide a default value for the native path key.
>> Since my modification is to the eval-in-project function it should
>> also work with tasks that depend on that. But I have not tested this
>> yet.
>> So there it is if anyone finds this sort of thing useful.
On Dec 14, 11:35 pm, David Nolen <dnolen.li...@gmail.com> wrote:
> Ok this works perfectly for me. I can run lein-swank and connect a remote
> repl and run Penumbra examples.
Cool. I haven't even tested it with swank myself yet, just ran the
compile task to test if my solution worked, which was funny cause that
started all the examples at the same time.
Watch out for running lein clean in the penumbra dir ;) it will wipe
out your lib directory and you can't get it back with deps.
My whole goal with this was to get a good build/swank script in place
for penumbra so I could try to hack penumbra to use a different
backend (lwjgl). So I'm not planning on doing more to leiningen than
that requires. Next goal is to sort out the deps for penumbra, maybe
put them on clojars if I can't find them on maven so that lein deps
will work.
On Dec 14, 11:12 pm, David Nolen <dnolen.li...@gmail.com> wrote:
> I wonder if this directory structure is really the best? Does it perhaps
> make sense to support providing an fn which returns the proper
> java.library.path? This way the directory path is flexible and we take
> advantage of the fact that project.clj is really just some Clojure code.
Yeah that's a good idea. I like having a default so that you don't
have to do anything in project.clj if you don't want to. But a
function is a good alternative to supplying an absolute path. Maybe
a :native-path-fn key in addition to what is there now? Or I could
just add a type check for the current key, see if it's a string or
function, or would that be confusing?
mac <markus.gustavs...@gmail.com> writes:
> I forked leiningen to try to add support for setting java.library.path
> in the compile task and I seem to have succeeded. The ant java task
> needed to run forked for it to work. I don't know what other
> implications that has, maybe it's a bit slower, but at least it works.
Looks good. A couple comments:
* Calls to the "file" function don't need the separator passed in, so
checking (System/getProperty "file.separator") can be removed.
* It looks like find-native-lib-path checks the OS, but if you add in a
:native-path to project.clj, it overrides that. Does it ever make
sense to add :native-path to project.clj if it hardcodes it to a
single OS?
* I'm picky about whitespace; could you replace your tabs with spaces
and trim trailing whitespace?
* If your git skills are up to it, could you rebase your master so it
doesn't change the groupId etc in project.clj? That would make it
easier to merge. If you're not sure how that's done then I can do it
on my own; it would just make merging easier.
I'd be happy to merge if you could address these questions.
On Dec 16, 6:03 am, Phil Hagelberg <p...@hagelb.org> wrote:
> * Calls to the "file" function don't need the separator passed in, so
> checking (System/getProperty "file.separator") can be removed.
ok
> * It looks like find-native-lib-path checks the OS, but if you add in a
> :native-path to project.clj, it overrides that. Does it ever make
> sense to add :native-path to project.clj if it hardcodes it to a
> single OS?
No I don't think it makes sense to manually add :native-path in
project.clj if you are going to push to clojars or some other public
repo but it might be nice for local experimentation and development. I
hesitate to mix the behaviors so that OS is always taken into account,
it might be very confusing to be able to specify a path and then have
parts of it change depending on your platform.
> * I'm picky about whitespace; could you replace your tabs with spaces
> and trim trailing whitespace?
Sure, I thought I was using spaces :) using a fresh emacs install, not
everything is configured I guess.
> * If your git skills are up to it, could you rebase your master so it
> doesn't change the groupId etc in project.clj? That would make it
> easier to merge. If you're not sure how that's done then I can do it
> on my own; it would just make merging easier.
I'll take a look at it. I'm new to both git and github but that's good
stuff to know how to do so I'll try.
I will just discard any changes I've made to files other than
compile.clj and then make sure I got the latest version of everything
and merging it back should be a breeze.
> I'd be happy to merge if you could address these questions.
> mac <markus.gustavs...@gmail.com> writes:
> > I forked leiningen to try to add support for setting java.library.path
> > in the compile task and I seem to have succeeded. The ant java task
> > needed to run forked for it to work. I don't know what other
> > implications that has, maybe it's a bit slower, but at least it works.
> Looks good. A couple comments:
> * Calls to the "file" function don't need the separator passed in, so
> checking (System/getProperty "file.separator") can be removed.
> * It looks like find-native-lib-path checks the OS, but if you add in a
> :native-path to project.clj, it overrides that. Does it ever make
> sense to add :native-path to project.clj if it hardcodes it to a
> single OS?
> * I'm picky about whitespace; could you replace your tabs with spaces
> and trim trailing whitespace?
> * If your git skills are up to it, could you rebase your master so it
> doesn't change the groupId etc in project.clj? That would make it
> easier to merge. If you're not sure how that's done then I can do it
> on my own; it would just make merging easier.
> I'd be happy to merge if you could address these questions.
> thanks,
> Phil
Fixed.
I was a little clumsy when merging with upstream (should have just
replaced my files with yours) so I had to hand merge bin/lein and
project.clj but I think I got rid of all my garbage, making them
character identical with the upstream version.
Only diffs now should be my additions to compile.clj
mac <markus.gustavs...@gmail.com> writes: > Sure, I thought I was using spaces :) using a fresh emacs install, not > everything is configured I guess.
(setq-default indent-tabs-mode nil) should do the trick.
> I was a little clumsy when merging with upstream (should have just > replaced my files with yours) so I had to hand merge bin/lein and > project.clj but I think I got rid of all my garbage, making them > character identical with the upstream version. > Only diffs now should be my additions to compile.clj
I did some interactive rebasing on top of your branch and pushed it to my setfork branch since I didn't want the changes to project.clj etc cluttering up the history. Could you take a look and see if it breaks anything? I don't have a Mac or a native-library-using project to try it out on.
Actually if you could contribute a test case that would be even better, but I know these kinds of things are hard to test sometimes.
Once I get the go-ahead I will merge to master. Since my changes were rebased from yours, you will need to do "git pull --force technomancy master" when updating. This is why I recommend doing all feature work in a topic branch; when your master branch matches the upstream it's much easier.
> Actually if you could contribute a test case that would be even better, > but I know these kinds of things are hard to test sometimes.
Yeah I don't really have anything small and concrete. I have been testing by doing lein compile on my fork of penumbra. Since the dir structure of penumbra is not the one that leiningen expects it's not an ideal project to test on. But it works if you set :native-path manually and remember not to do lein clean since that will nuke the lib folder.
I have also tested with a project that does not have native libs but needs the fork on mac because it's a swing/awt project and it works just fine both with compile and swank. I can't really put it up on the net because it's part of a project that will probably not be open sourced (at least not in a while).
> Once I get the go-ahead I will merge to master. Since my changes were > rebased from yours, you will need to do "git pull --force technomancy > master" when updating. This is why I recommend doing all feature work in > a topic branch; when your master branch matches the upstream it's much > easier.
Yeah I probably made a mess of my git history again now that I merged in your master changes. But at least I'm learning as I go ;)
mac <markus.gustavs...@gmail.com> writes: >> Once I get the go-ahead I will merge to master. Since my changes were >> rebased from yours, you will need to do "git pull --force technomancy >> master" when updating. This is why I recommend doing all feature work in >> a topic branch; when your master branch matches the upstream it's much >> easier.
> Yeah I probably made a mess of my git history again now that I merged > in your master changes. But at least I'm learning as I go ;)
