how to speedup lein uberjar?

[Sunil S Nandihalli]

Hi Everybody,

  My lein uberjar takes about 2 minutes to run. It also recompiles all the clj files everytime.  Is there a way to speedup lein-uberjar .. Or may be make it incremental?

Thanks,

Sunil.

[Colin Yates]

Hi Sunil,

lein uberjar by default cleans everything. You can tell it not to by setting ‘auto-clean’ to false in your project.clj but uberjar is supposed to be a ‘do a complete build’. Typically the uberjar/war contains productionised clojurescript (e.g. compiled with :advanced) so this works well.

I would recommend living with the 2 minute run and using that time to meditate/stretch/micro-koan.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kul]

Hi,

I think the main reason of it is due to the fact that aot compilation is transitive and compiles all dependent namespaces.
You will have to wait for this issue http://dev.clojure.org/jira/browse/CLJ-322 to get some closure :)


On Monday,

[Matching Socks]

If you do not want much to be compiled ahead-of-time and represented in the jar as class files, you may use the :impl-ns option on :gen-class.

e.g.,

1) project.clj says

:aot [bla.bla-aot]

2) bla.bla_aot.clj says

(ns bla.bla-aot (:gen-class :impl-ns bla.bla))

Ahead-of-time compilation does not follow the impl-ns reference.

3) bla.bla.clj has functions that Java may call as though they were in the bla.bla_aot class: for example, a -main function, which Java may call as bla.bla_aot.main.

[dennis zhuang]

Recommend to use libdir plugin:  https://github.com/djpowell/lein-libdir

Run lein libdir to copy all the dependencies to lib directory, and  just run lein jar to package the project. 

Then you can rsync or copy the project jar with all dependent jars instead of compiling all dependent namespaces that is really slow.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

庄晓丹
Email:        killm...@gmail.com xzh...@avos.com
Site:           http://fnil.net
Twitter:      @killme2008

[Asim Jalis]

Hi Sunil,

I am wondering why you want to speed up lein uberjar. 

How frequently do you need to run this?

What is your use case and workflow around this?

Thanks.

Asim

--

[Sunil S Nandihalli]

Hi Asim,

 I needed to run lein uberjar to submit the jar to run a distributed job on a apache-spark cluster.  and my uberjar is currently taking anywhere from 2 to 4 minutes to finish. I have have had to run uberjar fairly frequently during my iterations ( 4 to 5 times in an hour) . I am new to spark. So I need to quickly iterate with changes in the code. 

Thanks for asking.

Regards,

Sunil.

[Michael Blume]

I think Matching Socks has definitely got the right answer -- anything that prevents your code from being AOT-compiled is going to give you a huge speedup.

[Asim Jalis]

Hi Sunil,

Here is something to think about. Can you use the REPL for your workflow instead? 

If the changes are all in the Spark driver code you can run the driver inside the REPL and then test out the changes in the REPL. Once you have something that works you can save that and deploy an uberjar.

Deploying an uberjar 4-5 times an hour feels like it goes against the spirit of Clojure.

Asim

[Sunil S Nandihalli]

Thanks Asim for this suggestion. I wanted to do it but did not have the bandwidth to explore how to do it.. I am new to spark..  I usually do repl-driven dev for local apps... Have you done repl-driven using spark and clojure? do you have pointers as to how to do it.. I can explore if you tell me that there really are no problems in doing that.

Regards,

Sunil.

[Sunil S Nandihalli]

I can embed an nrepl-server in a driver program and submit that jar and then connect the nrepl-server to my emacs cider and work with it. I have just never done it. May be it is time to understand how instead of just doing 

M-x cider-jack-in

Thanks Asim for pointing in the right direction.

Regards,

Sunil.

[Asim Jalis]

Hi Sunil,

Yes. This should work. I have used something much simpler in the past such as server-socket. This starts listening on a specific port and then you can telnet to that port from your client and start telnet with rlwrap to get history and editing.

Asim 

[Sunil S Nandihalli]

It was too simple to setup .. I just had not spent time .. and also with respect to running the spark-app in yarn-client mode and connecting to it was a good suggestion. Thank you! Sunil.