scoobi app runs in local mode
59 views
Skip to first unread message
samira elham
Jun 13, 2014, 4:45:18 AM6/13/14
to scoobi...@googlegroups.com
hi,
I wrote a program which count the IP addresses in a text file. when I run my program in local mode (by using sbt "run input output") everything is right. but when I run the jar file on hadoop2.2.0 (I created the file using sbt assembly) the job runs successfully but the result does not show on hdfs.
any Ideas what happened to the results?
my code is:
package mypackage.myapp
import com.nicta.scoobi.Scoobi._
import Reduction._
import scala.util.matching.Regex
import java.util.regex.Pattern
object merge extends ScoobiApp{
val wordPattern="(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)".r
def run(){
val lines:DList[String]=
fromTextFile(args(0))
val keyValuePair1:DList[(String,Int)] = lines mapFlatten {_.split("[,()]+")} map { w => if(w.matches(wordPattern.toString))(w,1) else null}
val keyValuePair:DList[(String,Int)]=keyValuePair1 filter{_!=null}
val grouped: DList[(String,Iterable[Int])] = keyValuePair.groupByKey
val combined:DList[(String,Int)]=grouped.combine (Sum.int)
persist(toTextFile(combined,args(1),overwrite=true))
}
}
the sbt file is:
import AssemblyKeys._
assemblySettings
name := "MyApplication"
version := "1.0"
scalaVersion := "2.10.2"
scalacOptions ++= Seq("-Dependent-method-types", "-deprecation")
libraryDependencies ++= Seq("com.nicta" %% "scoobi" % "0.7.2" )
resolvers ++= Seq(Resolver.sonatypeRepo("releases"),
Resolver.sonatypeRepo("snaspshots"),
"snapshots" at "http://oss.sonatype.org/content/repositories/snapshots",
"extra stuff" at "http://nicta.github.com/scoobi/releases/")
mergeStrategy in assembly <<= (mergeStrategy in assembly) { mergeStrategy => {
case entry => {
val strategy = mergeStrategy(entry)
if (strategy == MergeStrategy.deduplicate) MergeStrategy.first
else strategy
}
}}
the files on hdfs after run this job is:
Found 1 items
-rw-r--r-- 3 root supergroup 0 2014-06-13 01:19 /ou/_SUCCESS
I wrote a program which count the IP addresses in a text file. when I run my program in local mode (by using sbt "run input output") everything is right. but when I run the jar file on hadoop2.2.0 (I created the file using sbt assembly) the job runs successfully but the result does not show on hdfs.
any Ideas what happened to the results?
my code is:
package mypackage.myapp
import com.nicta.scoobi.Scoobi._
import Reduction._
import scala.util.matching.Regex
import java.util.regex.Pattern
object merge extends ScoobiApp{
val wordPattern="(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)".r
def run(){
val lines:DList[String]=
fromTextFile(args(0))
val keyValuePair1:DList[(String,Int)] = lines mapFlatten {_.split("[,()]+")} map { w => if(w.matches(wordPattern.toString))(w,1) else null}
val keyValuePair:DList[(String,Int)]=keyValuePair1 filter{_!=null}
val grouped: DList[(String,Iterable[Int])] = keyValuePair.groupByKey
val combined:DList[(String,Int)]=grouped.combine (Sum.int)
persist(toTextFile(combined,args(1),overwrite=true))
}
}
the sbt file is:
import AssemblyKeys._
assemblySettings
name := "MyApplication"
version := "1.0"
scalaVersion := "2.10.2"
scalacOptions ++= Seq("-Dependent-method-types", "-deprecation")
libraryDependencies ++= Seq("com.nicta" %% "scoobi" % "0.7.2" )
resolvers ++= Seq(Resolver.sonatypeRepo("releases"),
Resolver.sonatypeRepo("snaspshots"),
"snapshots" at "http://oss.sonatype.org/content/repositories/snapshots",
"extra stuff" at "http://nicta.github.com/scoobi/releases/")
mergeStrategy in assembly <<= (mergeStrategy in assembly) { mergeStrategy => {
case entry => {
val strategy = mergeStrategy(entry)
if (strategy == MergeStrategy.deduplicate) MergeStrategy.first
else strategy
}
}}
the files on hdfs after run this job is:
Found 1 items
-rw-r--r-- 3 root supergroup 0 2014-06-13 01:19 /ou/_SUCCESS
samira elham
Jun 13, 2014, 4:50:52 AM6/13/14
to scoobi...@googlegroups.com
I forgot to add these files:
hadoop jar ./target/scala-2.10/MyApplication-assembly-1.0.jar /result.txt /ou --scoobi cluster
14/06/13 02:08:01 INFO Configuration.deprecation: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
14/06/13 02:08:01 INFO Configuration.deprecation: mapred.map.child.log.level is deprecated. Instead, use mapreduce.map.log.level
14/06/13 02:08:01 INFO Configuration.deprecation: mapred.reduce.child.log.level is deprecated. Instead, use mapreduce.reduce.log.level
14/06/13 02:08:01 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/centos/hadoop/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/06/13 02:08:06 INFO Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
[INFO] ScoobiApp - the URL of Java (evidenced with the java.lang.String class) is jar:file:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55/jre/lib/rt.jar!/java/lang/String.class
[INFO] ScoobiApp - the URL of Scala (evidenced with the scala.collection.immutable.Range class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/scala/collection/immutable/Range.class
[INFO] ScoobiApp - the URL of Hadoop (evidenced with the org.apache.hadoop.io.Writable class) is jar:file:/home/centos/hadoop/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar!/org/apache/hadoop/io/Writable.class
[INFO] ScoobiApp - the URL of Avro (evidenced with the org.apache.avro.Schema class) is jar:file:/home/centos/hadoop/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar!/org/apache/avro/Schema.class
[INFO] ScoobiApp - the URL of Kiama (evidenced with the org.kiama.rewriting.Rewriter class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/org/kiama/rewriting/Rewriter.class
[INFO] ScoobiApp - the URL of Scoobi (evidenced with the com.nicta.scoobi.core.ScoobiConfiguration class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/com/nicta/scoobi/core/ScoobiConfiguration.class
[INFO] Sink - Output path: /ou
[INFO] Source - Input path: /result.txt (859.00 bytes)
[INFO] HadoopMode - Executing layers
Layer(1
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd) ,
GroupByKey (29)[String,Int] (bridge e1680) ,
Combine (30)[String,Int] (bridge 34740) [sinks: Some(/ou)])
Mscr(1
inputs: + GbkInputChannel(Load (1)[String] (TextSource(1)
/result.txt
) )
mappers
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd)
last mappers
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd)
outputs: + GbkOutputChannel(GroupByKey (29)[String,Int] (bridge e1680) , combiner = Combine (30)[String,Int] (bridge 34740) [sinks: Some(/ou)]))
[INFO] HadoopMode - Executing layer 1
Layer(1
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd) ,
GroupByKey (29)[String,Int] (bridge e1680) ,
Combine (30)[String,Int] (bridge 34740) [sinks: Some(/ou)])
[INFO] HadoopMode - executing mscrs
Mscr(1
inputs: + GbkInputChannel(Load (1)[String] (TextSource(1)
/result.txt
) )
mappers
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd)
last mappers
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd)
outputs: + GbkOutputChannel(GroupByKey (29)[String,Int] (bridge e1680) , combiner = Combine (30)[String,Int] (bridge 34740) [sinks: Some(/ou)]))
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.output.value.groupfn.class is deprecated. Instead, use mapreduce.job.output.group.comparator.class
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
14/06/13 02:08:09 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/06/13 02:08:09 INFO Configuration.deprecation: mapreduce.partitioner.class is deprecated. Instead, use mapreduce.job.partitioner.class
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
14/06/13 02:08:09 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
14/06/13 02:08:10 INFO Configuration.deprecation: fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
14/06/13 02:08:10 INFO Configuration.deprecation: fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
14/06/13 02:08:10 INFO Configuration.deprecation: mapred.temp.dir is deprecated. Instead, use mapreduce.cluster.temp.dir
14/06/13 02:08:10 INFO Configuration.deprecation: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
14/06/13 02:08:10 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/06/13 02:08:10 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
14/06/13 02:08:10 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/06/13 02:08:10 INFO Configuration.deprecation: dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
[INFO] MapReduceJob - Total input size: 859.00 bytes
[INFO] MapReduceJob - Number of reducers: 1
14/06/13 02:08:28 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
[INFO] JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
[INFO] FileInputFormat - Total input paths to process : 1
[INFO] JobSubmitter - number of splits:1
14/06/13 02:08:29 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes
14/06/13 02:08:29 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/06/13 02:08:29 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
[INFO] JobSubmitter - Submitting tokens for job: job_local1973612901_0001
14/06/13 02:08:29 WARN conf.Configuration: file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/staging/root1973612901/.staging/job_local1973612901_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/06/13 02:08:29 WARN conf.Configuration: file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/staging/root1973612901/.staging/job_local1973612901_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309453/c0c8deea-a368-4667-bb57-6a9592f932be <- /home/centos/myProject/myApp/c0c8deea-a368-4667-bb57-6a9592f932be
[INFO] LocalDistributedCacheManager - Localized hdfs://master:54311/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/env/c0c8deea-a368-4667-bb57-6a9592f932be as file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309453/c0c8deea-a368-4667-bb57-6a9592f932be
[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309454/scoobi.mappers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) <- /home/centos/myProject/myApp/scoobi.mappers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
[INFO] LocalDistributedCacheManager - Localized hdfs://master:54311/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/dist-objs/scoobi.mappers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) as file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309454/scoobi.mappers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309455/scoobi.combiners-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) <- /home/centos/myProject/myApp/scoobi.combiners-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
[INFO] LocalDistributedCacheManager - Localized hdfs://master:54311/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/dist-objs/scoobi.combiners-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) as file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309455/scoobi.combiners-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309456/scoobi.reducers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) <- /home/centos/myProject/myApp/scoobi.reducers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
[INFO] LocalDistributedCacheManager - Localized hdfs://master:54311/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/dist-objs/scoobi.reducers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) as file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309456/scoobi.reducers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.cache.localFiles is deprecated. Instead, use mapreduce.job.cache.local.files
14/06/13 02:08:29 WARN conf.Configuration: file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/localRunner/root/job_local1973612901_0001/job_local1973612901_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/06/13 02:08:29 WARN conf.Configuration: file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/localRunner/root/job_local1973612901_0001/job_local1973612901_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
[INFO] Job - The url to track the job: http://localhost:8080/
[INFO] MapReduceJob - MapReduce job 'job_local1973612901_0001' submitted. Please see http://localhost:8080/ for more info.
[INFO] LocalJobRunner - OutputCommitter set in config null
[INFO] LocalJobRunner - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
[INFO] LocalJobRunner - Waiting for map tasks
[INFO] LocalJobRunner - Starting task: attempt_local1973612901_0001_m_000000_0
[INFO] Task - Using ResourceCalculatorProcessTree : [ ]
[INFO] MapTask - Processing split: hdfs://master:54311/result.txt:0+859 (on channel:1)
[INFO] MapTask - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
[INFO] MapTask - (EQUATOR) 0 kvi 26214396(104857584)
[INFO] MapTask - mapreduce.task.io.sort.mb: 100
[INFO] MapTask - soft limit at 83886080
[INFO] MapTask - bufstart = 0; bufvoid = 104857600
[INFO] MapTask - kvstart = 26214396; length = 6553600
[INFO] MapTask - the URL of Java (evidenced with the java.lang.String class) is jar:file:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55/jre/lib/rt.jar!/java/lang/String.class
[INFO] MapTask - the URL of Scala (evidenced with the scala.collection.immutable.Range class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/scala/collection/immutable/Range.class
[INFO] MapTask - the URL of Hadoop (evidenced with the org.apache.hadoop.io.Writable class) is jar:file:/home/centos/hadoop/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar!/org/apache/hadoop/io/Writable.class
[INFO] MapTask - the URL of Avro (evidenced with the org.apache.avro.Schema class) is jar:file:/home/centos/hadoop/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar!/org/apache/avro/Schema.class
[INFO] MapTask - the URL of Kiama (evidenced with the org.kiama.rewriting.Rewriter class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/org/kiama/rewriting/Rewriter.class
[INFO] MapTask - the URL of Scoobi (evidenced with the com.nicta.scoobi.core.ScoobiConfiguration class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/com/nicta/scoobi/core/ScoobiConfiguration.class
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
[INFO] MapTask - Starting on master.cluster.lab
[INFO] MapTask - Input is hdfs://master:54311/result.txt:0+859 (on channel:1)
[INFO] LocalJobRunner -
[INFO] MapTask - Starting flush of map output
[INFO] Task - Task:attempt_local1973612901_0001_m_000000_0 is done. And is in the process of committing
[INFO] LocalJobRunner - map
[INFO] Task - Task 'attempt_local1973612901_0001_m_000000_0' done.
[INFO] LocalJobRunner - Finishing task: attempt_local1973612901_0001_m_000000_0
[INFO] LocalJobRunner - Map task executor complete.
[INFO] Task - Using ResourceCalculatorProcessTree : [ ]
[INFO] Merger - Merging 1 sorted segments
[INFO] Merger - Down to the last merge-pass, with 0 segments left of total size: 0 bytes
[INFO] LocalJobRunner -
14/06/13 02:08:30 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
[INFO] ReduceTask - Starting on master.cluster.lab
[INFO] Task - Task:attempt_local1973612901_0001_r_000000_0 is done. And is in the process of committing
[INFO] LocalJobRunner -
[INFO] Task - Task attempt_local1973612901_0001_r_000000_0 is allowed to commit now
[INFO] FileOutputCommitter - Saved output of task 'attempt_local1973612901_0001_r_000000_0' to hdfs://master:54311/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/tmp-out-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/_temporary/0/task_local1973612901_0001_r_000000
[INFO] LocalJobRunner - reduce > reduce
[INFO] Task - Task 'attempt_local1973612901_0001_r_000000_0' done.
[INFO] MapReduceJob - Map 100% Reduce 100%
[INFO] HadoopMode - ===== END OF MSCR 1 ======
[INFO] HadoopMode - Layer sinks: List(TextFileSink: /ou)
[INFO] HadoopMode - ===== END OF LAYER 1 ======
I also don't know why it wrote LocalJobRunner?
hadoop jar ./target/scala-2.10/MyApplication-assembly-1.0.jar /result.txt /ou --scoobi cluster
14/06/13 02:08:01 INFO Configuration.deprecation: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
14/06/13 02:08:01 INFO Configuration.deprecation: mapred.map.child.log.level is deprecated. Instead, use mapreduce.map.log.level
14/06/13 02:08:01 INFO Configuration.deprecation: mapred.reduce.child.log.level is deprecated. Instead, use mapreduce.reduce.log.level
14/06/13 02:08:01 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/centos/hadoop/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/06/13 02:08:06 INFO Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
[INFO] ScoobiApp - the URL of Java (evidenced with the java.lang.String class) is jar:file:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55/jre/lib/rt.jar!/java/lang/String.class
[INFO] ScoobiApp - the URL of Scala (evidenced with the scala.collection.immutable.Range class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/scala/collection/immutable/Range.class
[INFO] ScoobiApp - the URL of Hadoop (evidenced with the org.apache.hadoop.io.Writable class) is jar:file:/home/centos/hadoop/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar!/org/apache/hadoop/io/Writable.class
[INFO] ScoobiApp - the URL of Avro (evidenced with the org.apache.avro.Schema class) is jar:file:/home/centos/hadoop/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar!/org/apache/avro/Schema.class
[INFO] ScoobiApp - the URL of Kiama (evidenced with the org.kiama.rewriting.Rewriter class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/org/kiama/rewriting/Rewriter.class
[INFO] ScoobiApp - the URL of Scoobi (evidenced with the com.nicta.scoobi.core.ScoobiConfiguration class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/com/nicta/scoobi/core/ScoobiConfiguration.class
[INFO] Sink - Output path: /ou
[INFO] Source - Input path: /result.txt (859.00 bytes)
[INFO] HadoopMode - Executing layers
Layer(1
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd) ,
GroupByKey (29)[String,Int] (bridge e1680) ,
Combine (30)[String,Int] (bridge 34740) [sinks: Some(/ou)])
Mscr(1
inputs: + GbkInputChannel(Load (1)[String] (TextSource(1)
/result.txt
) )
mappers
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd)
last mappers
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd)
outputs: + GbkOutputChannel(GroupByKey (29)[String,Int] (bridge e1680) , combiner = Combine (30)[String,Int] (bridge 34740) [sinks: Some(/ou)]))
[INFO] HadoopMode - Executing layer 1
Layer(1
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd) ,
GroupByKey (29)[String,Int] (bridge e1680) ,
Combine (30)[String,Int] (bridge 34740) [sinks: Some(/ou)])
[INFO] HadoopMode - executing mscrs
Mscr(1
inputs: + GbkInputChannel(Load (1)[String] (TextSource(1)
/result.txt
) )
mappers
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd)
last mappers
ParallelDo (28)[String,(String,Int),(((Unit,Unit),Unit),Unit)] (bridge 659bd)
outputs: + GbkOutputChannel(GroupByKey (29)[String,Int] (bridge e1680) , combiner = Combine (30)[String,Int] (bridge 34740) [sinks: Some(/ou)]))
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.output.value.groupfn.class is deprecated. Instead, use mapreduce.job.output.group.comparator.class
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
14/06/13 02:08:09 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/06/13 02:08:09 INFO Configuration.deprecation: mapreduce.partitioner.class is deprecated. Instead, use mapreduce.job.partitioner.class
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/06/13 02:08:09 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
14/06/13 02:08:09 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
14/06/13 02:08:10 INFO Configuration.deprecation: fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
14/06/13 02:08:10 INFO Configuration.deprecation: fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
14/06/13 02:08:10 INFO Configuration.deprecation: mapred.temp.dir is deprecated. Instead, use mapreduce.cluster.temp.dir
14/06/13 02:08:10 INFO Configuration.deprecation: mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir
14/06/13 02:08:10 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/06/13 02:08:10 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
14/06/13 02:08:10 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/06/13 02:08:10 INFO Configuration.deprecation: dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
[INFO] MapReduceJob - Total input size: 859.00 bytes
[INFO] MapReduceJob - Number of reducers: 1
14/06/13 02:08:28 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
[INFO] JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
[INFO] FileInputFormat - Total input paths to process : 1
[INFO] JobSubmitter - number of splits:1
14/06/13 02:08:29 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes
14/06/13 02:08:29 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/06/13 02:08:29 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
[INFO] JobSubmitter - Submitting tokens for job: job_local1973612901_0001
14/06/13 02:08:29 WARN conf.Configuration: file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/staging/root1973612901/.staging/job_local1973612901_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/06/13 02:08:29 WARN conf.Configuration: file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/staging/root1973612901/.staging/job_local1973612901_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309453/c0c8deea-a368-4667-bb57-6a9592f932be <- /home/centos/myProject/myApp/c0c8deea-a368-4667-bb57-6a9592f932be
[INFO] LocalDistributedCacheManager - Localized hdfs://master:54311/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/env/c0c8deea-a368-4667-bb57-6a9592f932be as file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309453/c0c8deea-a368-4667-bb57-6a9592f932be
[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309454/scoobi.mappers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) <- /home/centos/myProject/myApp/scoobi.mappers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
[INFO] LocalDistributedCacheManager - Localized hdfs://master:54311/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/dist-objs/scoobi.mappers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) as file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309454/scoobi.mappers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309455/scoobi.combiners-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) <- /home/centos/myProject/myApp/scoobi.combiners-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
[INFO] LocalDistributedCacheManager - Localized hdfs://master:54311/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/dist-objs/scoobi.combiners-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) as file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309455/scoobi.combiners-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
[INFO] LocalDistributedCacheManager - Creating symlink: /tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309456/scoobi.reducers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) <- /home/centos/myProject/myApp/scoobi.reducers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
[INFO] LocalDistributedCacheManager - Localized hdfs://master:54311/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/dist-objs/scoobi.reducers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1) as file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/1402643309456/scoobi.reducers-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.cache.localFiles is deprecated. Instead, use mapreduce.job.cache.local.files
14/06/13 02:08:29 WARN conf.Configuration: file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/localRunner/root/job_local1973612901_0001/job_local1973612901_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
14/06/13 02:08:29 WARN conf.Configuration: file:/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/localRunner/root/job_local1973612901_0001/job_local1973612901_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
[INFO] Job - The url to track the job: http://localhost:8080/
[INFO] MapReduceJob - MapReduce job 'job_local1973612901_0001' submitted. Please see http://localhost:8080/ for more info.
[INFO] LocalJobRunner - OutputCommitter set in config null
[INFO] LocalJobRunner - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
[INFO] LocalJobRunner - Waiting for map tasks
[INFO] LocalJobRunner - Starting task: attempt_local1973612901_0001_m_000000_0
[INFO] Task - Using ResourceCalculatorProcessTree : [ ]
[INFO] MapTask - Processing split: hdfs://master:54311/result.txt:0+859 (on channel:1)
[INFO] MapTask - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
[INFO] MapTask - (EQUATOR) 0 kvi 26214396(104857584)
[INFO] MapTask - mapreduce.task.io.sort.mb: 100
[INFO] MapTask - soft limit at 83886080
[INFO] MapTask - bufstart = 0; bufvoid = 104857600
[INFO] MapTask - kvstart = 26214396; length = 6553600
[INFO] MapTask - the URL of Java (evidenced with the java.lang.String class) is jar:file:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.55/jre/lib/rt.jar!/java/lang/String.class
[INFO] MapTask - the URL of Scala (evidenced with the scala.collection.immutable.Range class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/scala/collection/immutable/Range.class
[INFO] MapTask - the URL of Hadoop (evidenced with the org.apache.hadoop.io.Writable class) is jar:file:/home/centos/hadoop/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar!/org/apache/hadoop/io/Writable.class
[INFO] MapTask - the URL of Avro (evidenced with the org.apache.avro.Schema class) is jar:file:/home/centos/hadoop/hadoop-2.2.0/share/hadoop/common/lib/avro-1.7.4.jar!/org/apache/avro/Schema.class
[INFO] MapTask - the URL of Kiama (evidenced with the org.kiama.rewriting.Rewriter class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/org/kiama/rewriting/Rewriter.class
[INFO] MapTask - the URL of Scoobi (evidenced with the com.nicta.scoobi.core.ScoobiConfiguration class) is jar:file:/home/centos/myProject/myApp/target/scala-2.10/MyApplication-assembly-1.0.jar!/com/nicta/scoobi/core/ScoobiConfiguration.class
14/06/13 02:08:29 INFO Configuration.deprecation: mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
[INFO] MapTask - Starting on master.cluster.lab
[INFO] MapTask - Input is hdfs://master:54311/result.txt:0+859 (on channel:1)
[INFO] LocalJobRunner -
[INFO] MapTask - Starting flush of map output
[INFO] Task - Task:attempt_local1973612901_0001_m_000000_0 is done. And is in the process of committing
[INFO] LocalJobRunner - map
[INFO] Task - Task 'attempt_local1973612901_0001_m_000000_0' done.
[INFO] LocalJobRunner - Finishing task: attempt_local1973612901_0001_m_000000_0
[INFO] LocalJobRunner - Map task executor complete.
[INFO] Task - Using ResourceCalculatorProcessTree : [ ]
[INFO] Merger - Merging 1 sorted segments
[INFO] Merger - Down to the last merge-pass, with 0 segments left of total size: 0 bytes
[INFO] LocalJobRunner -
14/06/13 02:08:30 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
[INFO] ReduceTask - Starting on master.cluster.lab
[INFO] Task - Task:attempt_local1973612901_0001_r_000000_0 is done. And is in the process of committing
[INFO] LocalJobRunner -
[INFO] Task - Task attempt_local1973612901_0001_r_000000_0 is allowed to commit now
[INFO] FileOutputCommitter - Saved output of task 'attempt_local1973612901_0001_r_000000_0' to hdfs://master:54311/tmp/scoobi-root/scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5/tmp-out-scoobi-20140613-020801-merge$-ef3303e9-0f63-49d0-8f26-3f3e0c8a3ec5(Mscr-1)/_temporary/0/task_local1973612901_0001_r_000000
[INFO] LocalJobRunner - reduce > reduce
[INFO] Task - Task 'attempt_local1973612901_0001_r_000000_0' done.
[INFO] MapReduceJob - Map 100% Reduce 100%
[INFO] HadoopMode - ===== END OF MSCR 1 ======
[INFO] HadoopMode - Layer sinks: List(TextFileSink: /ou)
[INFO] HadoopMode - ===== END OF LAYER 1 ======
I also don't know why it wrote LocalJobRunner?
Reply all
Reply to author
Forward
0 new messages