unit tests fail due to lack of PermGen space

[td...@foursquare.com]

I am encountering "java.lang.OutOfMemoryError: PermGen space" when trying to run the Scoobi unit tests for master (at commit 52967646e0ce3c81464fb21dd32f0426939672f6). I've given SBT 1 GB of PermGen space via "-XX:PermSize=1g -XX:MaxPermSize=1g ". 

SBT is version 0.12.3.

How should I be running unit tests?

[Eric Springer]

With moar xD

My sbt line:

java -Xms1512M -Xmx3536M -Xss2M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=2024M -jar `dirname $0`/sbt-launch.jar "$@"

--
You received this message because you are subscribed to the Google Groups "scoobi-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scoobi-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Eric Springer]

Also, since the JVM doesn't seem to like freeing memory it doesn't strictly need to -- I have a fat and skinny sbt launcher. So I can use the one i need (so for building or testing scoobi, I use the fat one)

[Thomas Dyas]

Understood.

My other question then is, what needs to be done to properly test Scoobi in terms of availability of a Hadoop cluster? (I'm getting failures due to "No cluster execution" and "JobExecException: MapReduce job 'job_local_0028' failed! ..."

You received this message because you are subscribed to a topic in the Google Groups "scoobi-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scoobi-users/AQZP6LIj8sI/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to scoobi-users...@googlegroups.com.

[Eric Torreborre]

Hi Thomas.

"No cluster execution" on a unit test just means that you haven't run your test on the cluster. By default each test is executed in memory and locally. If you want a cluster execution as well you need to pass the following arguments:

-- scoobi cluster

and to *only* run on the cluster:

-- scoobi !inmemory.!local.!cluster

Then, if you're seeing "JobExecException: MapReduce job 'job_local_0028' failed!", you should enable all the logging and try to find where the exception occurs precisely:

-- scoobi verbose.all

It possibly happens in Scoobi's code so please report anything you find strange.

Thanks,

Eric.