Unable to get scoobi Configuration Arguments to work (New User)

35 views

Skip to first unread message

Deepak Jain

unread,

Sep 29, 2014, 1:04:06 AM9/29/14

to scoob...@googlegroups.com

I am starting to learn more about Scoobi. I found couple of issues

1. Unable to run scoobi my-app example on actual cluster.

I have a 2 node hadoop cluster setup. From hadoop cli machine, when i run hadoop fs -ls / , i see the list of directories in HDFS. I have HADOOP_CONF directory pointing to hadoop conf. It runs locally and never on the cluster.

Logs

====

$ export HADOOP_CONF=/etc/hadoop/conf

[dvasthimal@invisio-365818 my-app]$ ls -l /etc/hadoop/conf

total 152

-rw-r--r-- 1 hdfs hadoop 1744 Sep 26 02:55 capacity-scheduler.xml

-rw-r--r-- 1 hdfs root 1021 Sep 26 02:58 commons-logging.properties

-rw-r--r-- 1 hdfs hadoop 1335 Aug 27 20:14 configuration.xsl

-rw-r--r-- 1 root root 318 Aug 27 20:14 container-executor.cfg

-rw-r--r-- 1 hdfs hadoop 1989 Sep 26 02:58 core-site.xml

-rw-r--r-- 1 root root 774 Aug 27 20:14 core-site.xml.rpmnew

dvasthimal@invisio-365818 my-app]$ rm -rf output.scoobie*; sbt "run-main WordCount -Dmapred.max.map.failures.percent=20 -Dmapred.max.reduce.failures.percent=20 input.txt output.scoobie.3 -- scoobi verbose"

[info] Set current project to MyApplication (in build file:/home/dvasthimal/my-app/)

[info] Running WordCount -Dmapred.max.map.failures.percent=20 -Dmapred.max.reduce.failures.percent=20 input.txt output.scoobie.3 -- scoobi verbose

[WARN] NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[INFO] deprecation - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used

[INFO] deprecation - mapred.map.child.log.level is deprecated. Instead, use mapreduce.map.log.level

[INFO] deprecation - mapred.reduce.child.log.level is deprecated. Instead, use mapreduce.reduce.log.level

[INFO] deprecation - mapred.max.map.failures.percent is deprecated. Instead, use mapreduce.map.failures.maxpercent

[INFO] deprecation - mapred.max.reduce.failures.percent is deprecated. Instead, use mapreduce.reduce.failures.maxpercent

[INFO] deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address

[INFO] HadoopMode - SCOOBI version: 0.8.5, commit: 7d0e802, timestamp: 08-07-2014 09:32:43 +1000

[INFO] Sink - Output path: output.scoobie.3

[INFO] Source - Input path: input.txt (1.63 KiB)

[INFO] HadoopMode - ======================================================================

[INFO] HadoopMode - ===== START OF SCOOBI JOB 'WordCount$-0928-215852-1929107525' ========

[INFO] HadoopMode - ======================================================================

[INFO] HadoopMode - Executing map reduce jobs

Mscr(1

inputs: + GbkInputChannel(Load (1)[String] (source 1) )

mappers

ParallelDo (17)[String,(String,Int),((Unit,Unit),Unit)] (bridge 7fca0)

last mappers

ParallelDo (17)[String,(String,Int),((Unit,Unit),Unit)] (bridge 7fca0)

outputs: + GbkOutputChannel(GroupByKey (18)[String,Int] (bridge 52d0e) , combiner = Combine (19)[String,Int] (bridge 93d66) [sinks: Some(output.scoobie.3)]))

[INFO] HadoopMode - ===== START OF MAP REDUCE JOB 1 of 1 (mscr id = 1) ======

[INFO] deprecation - mapred.jar is deprecated. Instead, use mapreduce.job.jar

[INFO] deprecation - mapred.output.value.groupfn.class is deprecated. Instead, use mapreduce.job.output.group.comparator.class

[INFO] deprecation - mapred.output.key.comparator.class is deprecated. Instead, use mapreduce.job.output.key.comparator.class

[INFO] deprecation - mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files

[INFO] deprecation - mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class

[INFO] deprecation - mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class

[INFO] deprecation - mapreduce.partitioner.class is deprecated. Instead, use mapreduce.job.partitioner.class

[INFO] deprecation - mapred.job.name is deprecated. Instead, use mapreduce.job.name

[INFO] deprecation - mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class

[INFO] deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

[INFO] deprecation - mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir

[INFO] deprecation - fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir

[INFO] deprecation - dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir

[INFO] deprecation - fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir

[INFO] deprecation - mapred.temp.dir is deprecated. Instead, use mapreduce.cluster.temp.dir

[INFO] deprecation - dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir

[INFO] deprecation - mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir

[INFO] deprecation - mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class

[INFO] deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

[INFO] deprecation - mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class

[INFO] deprecation - dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir

[INFO] MapReduceJob - Total input size: 1.63 KiB

[INFO] MapReduceJob - Number of reducers: 1

[INFO] deprecation - session.id is deprecated. Instead, use dfs.metrics.session-id

[INFO] JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=

[INFO] deprecation - mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir

[INFO] FileInputFormat - Total input paths to process : 1

[INFO] JobSubmitter - number of splits:1

[INFO] deprecation - user.name is deprecated. Instead, use mapreduce.job.user.name

[INFO] deprecation - mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes

[INFO] deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

[INFO] deprecation - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

[INFO] deprecation - mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class

[INFO] deprecation - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir

[INFO] deprecation - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

[INFO] deprecation - mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps

[INFO] deprecation - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir

[INFO] JobSubmitter - Submitting tokens for job: job_local842016996_0001

[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737052/7b100dac-bcb3-4c06-a4c8-020b2093f246 <- /home/dvasthimal/my-app/7b100dac-bcb3-4c06-a4c8-020b2093f246

[INFO] LocalDistributedCacheManager - Localized file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/env/7b100dac-bcb3-4c06-a4c8-020b2093f246 as file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737052/7b100dac-bcb3-4c06-a4c8-020b2093f246

[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737053/scoobi.metadata.TK20 <- /home/dvasthimal/my-app/scoobi.metadata.TK20

[INFO] LocalDistributedCacheManager - Localized file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TK20 as file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737053/scoobi.metadata.TK20

[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737054/scoobi.metadata.TV20 <- /home/dvasthimal/my-app/scoobi.metadata.TV20

[INFO] LocalDistributedCacheManager - Localized file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TV20 as file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737054/scoobi.metadata.TV20

[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737055/scoobi.metadata.TP20 <- /home/dvasthimal/my-app/scoobi.metadata.TP20

[INFO] LocalDistributedCacheManager - Localized file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TP20 as file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737055/scoobi.metadata.TP20

[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737056/scoobi.metadata.TG20 <- /home/dvasthimal/my-app/scoobi.metadata.TG20

[INFO] LocalDistributedCacheManager - Localized file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TG20 as file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737056/scoobi.metadata.TG20

[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737057/scoobi.mappers-step_1_of_1 <- /home/dvasthimal/my-app/scoobi.mappers-step_1_of_1

[INFO] LocalDistributedCacheManager - Localized file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.mappers-step_1_of_1 as file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737057/scoobi.mappers-step_1_of_1

[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737058/scoobi.combiners-step_1_of_1 <- /home/dvasthimal/my-app/scoobi.combiners-step_1_of_1

[INFO] LocalDistributedCacheManager - Localized file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.combiners-step_1_of_1 as file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737058/scoobi.combiners-step_1_of_1

[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737059/scoobi.reducers-step_1_of_1 <- /home/dvasthimal/my-app/scoobi.reducers-step_1_of_1

[INFO] LocalDistributedCacheManager - Localized file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.reducers-step_1_of_1 as file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737059/scoobi.reducers-step_1_of_1

[INFO] deprecation - mapred.cache.localFiles is deprecated. Instead, use mapreduce.job.cache.local.files

[INFO] Job - The url to track the job: http://localhost:8080/

[INFO] LocalJobRunner - OutputCommitter set in config null

[INFO] MapReduceJob - MapReduce job 'job_local842016996_0001' submitted. Please see http://localhost:8080/ for more info.

[INFO] LocalJobRunner - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter

[INFO] LocalJobRunner - Waiting for map tasks

[INFO] LocalJobRunner - Starting task: attempt_local842016996_0001_m_000000_0

[INFO] Task - Using ResourceCalculatorProcessTree : [ ]

[INFO] MapTask - Processing split: file:/home/dvasthimal/my-app/input.txt:0+1664 (on channel:1)

[INFO] MapTask - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer

[INFO] MapTask - (EQUATOR) 0 kvi 26214396(104857584)

[INFO] MapTask - mapreduce.task.io.sort.mb: 100

[INFO] MapTask - soft limit at 83886080

[INFO] MapTask - bufstart = 0; bufvoid = 104857600

[INFO] MapTask - kvstart = 26214396; length = 6553600

[INFO] MapTask - the URL of Java (evidenced with the java.lang.String class) is jar:file:/home/dvasthimal/invisio/jdk1.7.0_67/jre/lib/rt.jar!/java/lang/String.class

[INFO] MapTask - the URL of Scala (evidenced with the scala.collection.immutable.Range class) is jar:file:/home/dvasthimal/.sbt/boot/scala-2.10.3/lib/scala-library.jar!/scala/collection/immutable/Range.class

[INFO] MapTask - the URL of Hadoop (evidenced with the org.apache.hadoop.io.Writable class) is jar:file:/home/dvasthimal/.ivy2/cache/org.apache.hadoop/hadoop-common/jars/hadoop-common-2.2.0.jar!/org/apache/hadoop/io/Writable.class

[INFO] MapTask - the URL of Avro (evidenced with the org.apache.avro.Schema class) is jar:file:/home/dvasthimal/.ivy2/cache/org.apache.avro/avro/jars/avro-1.7.4.jar!/org/apache/avro/Schema.class

[INFO] MapTask - the URL of Kiama (evidenced with the org.kiama.rewriting.Rewriter class) is jar:file:/home/dvasthimal/.ivy2/cache/com.googlecode.kiama/kiama_2.10/jars/kiama_2.10-1.6.0.jar!/org/kiama/rewriting/Rewriter.class

[INFO] MapTask - the URL of Scoobi (evidenced with the com.nicta.scoobi.core.ScoobiConfiguration class) is jar:file:/home/dvasthimal/.ivy2/cache/com.nicta/scoobi_2.10/jars/scoobi_2.10-0.8.5.jar!/com/nicta/scoobi/core/ScoobiConfiguration.class

[INFO] deprecation - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used

[INFO] DistCache - trying to pull an object from the cache at path: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.mappers-step_1_of_1 (memoise=false)

[INFO] DistCache - trying to open: file://file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737057/scoobi.mappers-step_1_of_1

[INFO] DistCache - trying to open: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.mappers-step_1_of_1

[INFO] DistCache - successfully opened: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.mappers-step_1_of_1

[INFO] MapTask - Starting on invisio-365818

[INFO] MapTask - Input is file:/home/dvasthimal/my-app/input.txt:0+1664 (on channel:1)

[INFO] DistCache - trying to pull an object from the cache at path: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/env/7b100dac-bcb3-4c06-a4c8-020b2093f246 (memoise=false)

[INFO] DistCache - trying to open: file://file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737052/7b100dac-bcb3-4c06-a4c8-020b2093f246

[INFO] DistCache - trying to open: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/env/7b100dac-bcb3-4c06-a4c8-020b2093f246

[INFO] DistCache - successfully opened: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/env/7b100dac-bcb3-4c06-a4c8-020b2093f246

[INFO] DistCache - trying to pull an object from the cache at path: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TK20 (memoise=false)

[INFO] DistCache - trying to open: file://file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737053/scoobi.metadata.TK20

[INFO] DistCache - trying to open: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TK20

[INFO] DistCache - successfully opened: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TK20

[INFO] DistCache - trying to pull an object from the cache at path: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TV20 (memoise=false)

[INFO] DistCache - trying to open: file://file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737054/scoobi.metadata.TV20

[INFO] DistCache - trying to open: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TV20

[INFO] DistCache - successfully opened: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TV20

[INFO] LocalJobRunner -

[INFO] MapTask - Starting flush of map output

[INFO] MapTask - Spilling map output

[INFO] MapTask - bufstart = 0; bufend = 3183; bufvoid = 104857600

[INFO] MapTask - kvstart = 26214396(104857584); kvend = 26213524(104854096); length = 873/6553600

[INFO] DistCache - trying to pull an object from the cache at path: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.combiners-step_1_of_1 (memoise=true)

[INFO] DistCache - trying to open: file://file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737058/scoobi.combiners-step_1_of_1

[INFO] DistCache - trying to open: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.combiners-step_1_of_1

[INFO] DistCache - successfully opened: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.combiners-step_1_of_1

[INFO] MapTask - Finished spill 0

[INFO] Task - Task:attempt_local842016996_0001_m_000000_0 is done. And is in the process of committing

[INFO] LocalJobRunner - map

[INFO] Task - Task 'attempt_local842016996_0001_m_000000_0' done.

[INFO] LocalJobRunner - Finishing task: attempt_local842016996_0001_m_000000_0

[INFO] LocalJobRunner - Map task executor complete.

[INFO] MapReduceJob - Map 100% Reduce 0%

[INFO] Task - Using ResourceCalculatorProcessTree : [ ]

[INFO] Merger - Merging 1 sorted segments

[INFO] Merger - Down to the last merge-pass, with 1 segments left of total size: 2186 bytes

[INFO] LocalJobRunner -

[INFO] deprecation - mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords

[INFO] DistCache - trying to pull an object from the cache at path: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.reducers-step_1_of_1 (memoise=false)

[INFO] DistCache - trying to open: file://file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737059/scoobi.reducers-step_1_of_1

[INFO] DistCache - trying to open: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.reducers-step_1_of_1

[INFO] DistCache - successfully opened: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.reducers-step_1_of_1

[INFO] ReduceTask - Starting on invisio-365818

[INFO] OutputChannel - Outputs are Some(output.scoobie.3)

[INFO] DistCache - trying to pull an object from the cache at path: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TG20 (memoise=false)

[INFO] DistCache - trying to open: file://file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/step_1_of_1/1411966737056/scoobi.metadata.TG20

[INFO] DistCache - trying to open: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TG20

[INFO] DistCache - successfully opened: file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/dist-objs/scoobi.metadata.TG20

[INFO] deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress

[INFO] Task - Task:attempt_local842016996_0001_r_000000_0 is done. And is in the process of committing

[INFO] LocalJobRunner -

[INFO] Task - Task attempt_local842016996_0001_r_000000_0 is allowed to commit now

[INFO] FileOutputCommitter - Saved output of task 'attempt_local842016996_0001_r_000000_0' to file:/tmp/scoobi-dvasthimal/WordCount$-0928-215852-1929107525/tmp-out-step_1_of_1/_temporary/0/task_local842016996_0001_r_000000

[INFO] LocalJobRunner - reduce > reduce

[INFO] Task - Task 'attempt_local842016996_0001_r_000000_0' done.

[INFO] MapReduceJob - Map 100% Reduce 100%

[INFO] HadoopMode - Map reduce job sinks: Vector(TextFileSink: output.scoobie.3)

[INFO] HadoopMode - ===== END OF MAP REDUCE JOB 1 of 1 (mscr id = 1, Scoobi job = WordCount$-0928-215852-1929107525) ======

[INFO] HadoopMode - ======================================================================

[INFO] HadoopMode - ===== END OF SCOOBI JOB 'WordCount$-0928-215852-1929107525' ========

[INFO] HadoopMode - ======================================================================

Not interrupting system thread Thread[process reaper,10,system]

[success] Total time: 8 s, completed Sep 28, 2014 9:58:58 PM

[dvasthimal@invisio-365818 my-app]$

2) Unable to run the scoobi configurations.

[-- scoobi VALUE1.VALUE2.VALUE3]

What does this syntax mean? Generally, configs are of form Key=Value.

There are multiple scoobi configs as defined here: http://nicta.github.io/scoobi/guide/com.nicta.scoobi.guide.Application.html#Scoobi+Configuration+Arguments

One example is inmemory, if i want to use it, do i specify as inmemory=true? inmemory.true or true.inmemory . All three did not work and the job always ran in local hadoop mode instead of scala collection.

Regards,

Deepak

Deepak Jain

unread,

Sep 29, 2014, 1:04:46 AM9/29/14

to scoob...@googlegroups.com

Is this the right forum for scoobi user ? I did not find scoobi-user group.

Eric Torreborre

unread,

Sep 29, 2014, 6:15:19 PM9/29/14

to scoob...@googlegroups.com

Hi,

1. You can use the HADOOP_CONF_DIR variable to point to the directory containing the cluster configuration.

2. the Scoobi arguments are indeed unusual. If you want to run inmemory you just write

... -- scoobi inmemory

and if you want "verbose" as well:

... -- scoobi inmemory.verbose

Eric.

PS: the forum for scoobi users is here.

Deepak Jain

unread,

Sep 30, 2014, 1:42:04 AM9/30/14

to scoob...@googlegroups.com

It did not work. I still see hadoop job submitted in local mode instead of cluster. I verified the config dir and its correct.

$ export HADOOP_CONF_DIR=/etc/hadoop/conf.empty

[dvasthimal@invisio-365818 my-app]$ rm -rf output.scoobie*; sbt "run-main WordCount -Dmapred.max.map.failures.percent=20 -Dmapred.max.reduce.failures.percent=20 input.txt output.scoobie.3 -- scoobi verbose"