simple filter gives me the following error: exited : cascading.tuple.TupleException: unable to read from input identifier:

325 views

Skip to first unread message

Ali

unread,

Aug 25, 2015, 4:53:48 PM8/25/15

to cascading-user

Hi,

So I'm trying to modify the first tutorial, to do a simple example. Here is my input file(I named it rain.txt, less changes the better):

000000|Jane Doe|Kohl's|20090106|61820|89.23

000000|Jane Doe|Kohl's|20090106|60005|43.21

I'm pretty sure the given command

Tap inTap = new Hfs( new TextDelimited( true, "|" ), inPath );

Parses the first line of my input into their respective fields.

Now what I want to do is filter out so I only output accID and name.

I was running into trouble so I tried to simply output the entire input file as an output file like this:

 Pipe LU = new Each( copyPipe, Fields.ALL, new Identity(), Fields.RESULTS );

The plan being to replace Fields.RESULTS with the lookupFields(below) :

Fields acctID = new Fields("acctID");
Fields name = new Fields("name");
Fields lookupFields = acctID.append( name );

So I want them to output AccID, and name without the rest of the fields

However, I'm getting the following error message, it's kind of cryptic and I wasn't able to figure it out through googling or reading the javadoc.

I've attached the full code, and full log below. I'm running this on a Cloudera 5.4 VM.

Thanks!,

Ali

* Project and contact information: http://www.cascading.org/

* This file is part of the Cascading project.

* Licensed under the Apache License, Version 2.0 (the "License");

* you may not use this file except in compliance with the License.

* You may obtain a copy of the License at

* http://www.apache.org/licenses/LICENSE-2.0

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

package impatient;


import java.util.Properties;


import cascading.flow.FlowDef;
import cascading.flow.hadoop2.Hadoop2MR1FlowConnector;
import cascading.pipe.Pipe;
import cascading.property.AppProps;
import cascading.scheme.hadoop.TextDelimited;
import cascading.tap.Tap;
import cascading.tap.hadoop.Hfs;
import cascading.pipe.Each;
import cascading.operation.filter.FilterNull;
import cascading.tuple.Fields;
import cascading.operation.Identity;


public class Main
  {
  public static void main( String[] args )
    {
    String inPath = args[ 0 ];
    String outPath = args[ 1 ];


    Properties properties = new Properties();
    AppProps.setApplicationJarClass( properties, Main.class );
    Hadoop2MR1FlowConnector flowConnector = new Hadoop2MR1FlowConnector( properties );


    // create the source tap
    Tap inTap = new Hfs( new TextDelimited( true, "|" ), inPath );


    // create the sink tap
    Tap outTap = new Hfs( new TextDelimited( true, "," ), outPath );
 
    Pipe copyPipe = new Pipe( "copy" );
 //define fields to keep.
 Fields acctID = new Fields("acctID");
 Fields name = new Fields("name");
 Fields lookupFields = acctID.append( name ); 
 


 Pipe LU = new Each( copyPipe, Fields.ALL, new Identity(), Fields.RESULTS );

    // connect the taps, pipes, etc., into a flow
    FlowDef flowDef = FlowDef.flowDef().addSource( copyPipe, inTap ).addTailSink( LU, outTap );
 //specify a pipe to transfer data between pipes.


    // run the flow
    flowConnector.connect( flowDef ).complete();
    }
  }

console output:

2015-08-25 13:50:42,111 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1440177002606_0009_000001

2015-08-25 13:50:42,446 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2015-08-25 13:50:42,464 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:

2015-08-25 13:50:42,464 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@5c329c80)

2015-08-25 13:50:43,240 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.

2015-08-25 13:50:43,367 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null

2015-08-25 13:50:43,369 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter

2015-08-25 13:50:43,418 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler

2015-08-25 13:50:43,419 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher

2015-08-25 13:50:43,420 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher

2015-08-25 13:50:43,421 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher

2015-08-25 13:50:43,422 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler

2015-08-25 13:50:43,428 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher

2015-08-25 13:50:43,428 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter

2015-08-25 13:50:43,429 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter

2015-08-25 13:50:43,468 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020]

2015-08-25 13:50:43,488 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020]

2015-08-25 13:50:43,508 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020]

2015-08-25 13:50:43,520 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled

2015-08-25 13:50:43,575 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler

2015-08-25 13:50:43,825 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties

2015-08-25 13:50:43,890 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).

2015-08-25 13:50:43,890 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started

2015-08-25 13:50:43,902 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1440177002606_0009 to jobTokenSecretManager

2015-08-25 13:50:44,040 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1440177002606_0009 because: not enabled;

2015-08-25 13:50:44,067 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1440177002606_0009 = 5063. Number of splits = 2

2015-08-25 13:50:44,067 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1440177002606_0009 = 0

2015-08-25 13:50:44,068 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1440177002606_0009Job Transitioned from NEW to INITED

2015-08-25 13:50:44,069 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1440177002606_0009.

2015-08-25 13:50:44,103 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue

2015-08-25 13:50:44,115 INFO [Socket Reader #1 for port 40309] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 40309

2015-08-25 13:50:44,137 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server

2015-08-25 13:50:44,139 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting

2015-08-25 13:50:44,139 INFO [IPC Server listener on 40309] org.apache.hadoop.ipc.Server: IPC Server listener on 40309: starting

2015-08-25 13:50:44,140 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at quickstart.cloudera/10.0.2.15:40309

2015-08-25 13:50:44,211 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog

2015-08-25 13:50:44,215 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined

2015-08-25 13:50:44,225 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)

2015-08-25 13:50:44,231 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce

2015-08-25 13:50:44,231 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static

2015-08-25 13:50:44,234 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*

2015-08-25 13:50:44,234 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*

2015-08-25 13:50:44,245 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 33340

2015-08-25 13:50:44,245 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4

2015-08-25 13:50:44,286 INFO [main] org.mortbay.log: Extract jar:file:/usr/jars/hadoop-yarn-common-2.6.0-cdh5.4.2.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_33340_mapreduce____.7lt8c6/webapp

2015-08-25 13:50:44,528 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConne...@0.0.0.0:33340

2015-08-25 13:50:44,528 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 33340

2015-08-25 13:50:44,910 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules

2015-08-25 13:50:44,914 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1440177002606_0009

2015-08-25 13:50:44,916 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue

2015-08-25 13:50:44,917 INFO [Socket Reader #1 for port 52321] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 52321

2015-08-25 13:50:44,922 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting

2015-08-25 13:50:44,923 INFO [IPC Server listener on 52321] org.apache.hadoop.ipc.Server: IPC Server listener on 52321: starting

2015-08-25 13:50:44,955 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true

2015-08-25 13:50:44,956 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3

2015-08-25 13:50:44,956 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33

2015-08-25 13:50:45,013 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030

2015-08-25 13:50:45,115 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:8192, vCores:8>

2015-08-25 13:50:45,115 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.cloudera

2015-08-25 13:50:45,120 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500

2015-08-25 13:50:45,122 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0

2015-08-25 13:50:45,132 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1440177002606_0009Job Transitioned from INITED to SETUP

2015-08-25 13:50:45,136 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP

2015-08-25 13:50:45,139 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1

2015-08-25 13:50:45,149 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1440177002606_0009Job Transitioned from SETUP to RUNNING

2015-08-25 13:50:45,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved quickstart.cloudera to /default-rack

2015-08-25 13:50:45,181 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1440177002606_0009_m_000000 Task Transitioned from NEW to SCHEDULED

2015-08-25 13:50:45,181 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved quickstart.cloudera to /default-rack

2015-08-25 13:50:45,181 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1440177002606_0009_m_000001 Task Transitioned from NEW to SCHEDULED

2015-08-25 13:50:45,182 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1440177002606_0009_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED

2015-08-25 13:50:45,182 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1440177002606_0009_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED

2015-08-25 13:50:45,183 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>

2015-08-25 13:50:45,217 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1440177002606_0009, File: hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/cloudera/.staging/job_1440177002606_0009/job_1440177002606_0009_1.jhist

2015-08-25 13:50:45,480 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://quickstart.cloudera:8020]

2015-08-25 13:50:46,120 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0

2015-08-25 13:50:46,160 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1440177002606_0009: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:6144, vCores:7> knownNMs=1

2015-08-25 13:50:47,177 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1

2015-08-25 13:50:47,179 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1440177002606_0009_01_000002 to attempt_1440177002606_0009_m_000000_0

2015-08-25 13:50:47,180 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:1 RackLocal:0

2015-08-25 13:50:47,223 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved quickstart.cloudera to /default-rack

2015-08-25 13:50:47,243 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://quickstart.cloudera:8020/tmp/hadoop-yarn/staging/cloudera/.staging/job_1440177002606_0009/job.jar

2015-08-25 13:50:47,247 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /tmp/hadoop-yarn/staging/cloudera/.staging/job_1440177002606_0009/job.xml

2015-08-25 13:50:47,248 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container

2015-08-25 13:50:47,248 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1

2015-08-25 13:50:47,248 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData

2015-08-25 13:50:47,275 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m

2015-08-25 13:50:47,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1440177002606_0009_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED

2015-08-25 13:50:47,284 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1440177002606_0009_01_000002 taskAttempt attempt_1440177002606_0009_m_000000_0

2015-08-25 13:50:47,286 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1440177002606_0009_m_000000_0

2015-08-25 13:50:47,287 INFO [ContainerLauncher #0] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : quickstart.cloudera:36019

2015-08-25 13:50:47,359 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1440177002606_0009_m_000000_0 : 13562

2015-08-25 13:50:47,361 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1440177002606_0009_m_000000_0] using containerId: [container_1440177002606_0009_01_000002 on NM: [quickstart.cloudera:36019]

2015-08-25 13:50:47,365 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1440177002606_0009_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING

2015-08-25 13:50:47,365 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: ATTEMPT_START task_1440177002606_0009_m_000000

2015-08-25 13:50:47,365 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1440177002606_0009_m_000000 Task Transitioned from SCHEDULED to RUNNING

2015-08-25 13:50:48,184 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1440177002606_0009: ask=3 release= 0 newContainers=1 finishedContainers=0 resourcelimit=<memory:4096, vCores:5> knownNMs=1

2015-08-25 13:50:48,184 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1

2015-08-25 13:50:48,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1440177002606_0009_01_000003 to attempt_1440177002606_0009_m_000001_0

2015-08-25 13:50:48,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved quickstart.cloudera to /default-rack

2015-08-25 13:50:48,185 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:2 RackLocal:0

2015-08-25 13:50:48,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m

2015-08-25 13:50:48,186 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1440177002606_0009_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED

2015-08-25 13:50:48,187 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1440177002606_0009_01_000003 taskAttempt attempt_1440177002606_0009_m_000001_0

2015-08-25 13:50:48,187 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1440177002606_0009_m_000001_0

2015-08-25 13:50:48,187 INFO [ContainerLauncher #1] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : quickstart.cloudera:36019

2015-08-25 13:50:48,207 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1440177002606_0009_m_000001_0 : 13562

2015-08-25 13:50:48,208 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1440177002606_0009_m_000001_0] using containerId: [container_1440177002606_0009_01_000003 on NM: [quickstart.cloudera:36019]

2015-08-25 13:50:48,208 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1440177002606_0009_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING

2015-08-25 13:50:48,209 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: ATTEMPT_START task_1440177002606_0009_m_000001

2015-08-25 13:50:48,209 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1440177002606_0009_m_000001 Task Transitioned from SCHEDULED to RUNNING

2015-08-25 13:50:48,832 INFO [Socket Reader #1 for port 52321] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1440177002606_0009 (auth:SIMPLE)

2015-08-25 13:50:48,857 INFO [IPC Server handler 1 on 52321] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1440177002606_0009_m_000002 asked for a task

2015-08-25 13:50:48,857 INFO [IPC Server handler 1 on 52321] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1440177002606_0009_m_000002 given task: attempt_1440177002606_0009_m_000000_0

2015-08-25 13:50:49,190 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1440177002606_0009: ask=3 release= 0 newContainers=1 finishedContainers=0 resourcelimit=<memory:3072, vCores:4> knownNMs=1

2015-08-25 13:50:49,190 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1

2015-08-25 13:50:49,190 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Cannot assign container Container: [ContainerId: container_1440177002606_0009_01_000004, NodeId: quickstart.cloudera:36019, NodeHttpAddress: quickstart.cloudera:8042, Resource: <memory:1024, vCores:1>, Priority: 20, Token: Token { kind: ContainerToken, service: 10.0.2.15:36019 }, ] for a map as either container memory less than required <memory:1024, vCores:1> or no pending map tasks - maps.isEmpty=true

2015-08-25 13:50:49,190 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:3 ContRel:1 HostLocal:2 RackLocal:0

2015-08-25 13:50:49,818 INFO [Socket Reader #1 for port 52321] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1440177002606_0009 (auth:SIMPLE)

2015-08-25 13:50:49,835 INFO [IPC Server handler 1 on 52321] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1440177002606_0009_m_000003 asked for a task

2015-08-25 13:50:49,835 INFO [IPC Server handler 1 on 52321] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1440177002606_0009_m_000003 given task: attempt_1440177002606_0009_m_000001_0

2015-08-25 13:50:50,199 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1440177002606_0009: ask=0 release= 1 newContainers=0 finishedContainers=1 resourcelimit=<memory:4096, vCores:5> knownNMs=1

2015-08-25 13:50:50,199 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1440177002606_0009_01_000004

2015-08-25 13:50:50,200 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container id container_1440177002606_0009_01_000004

2015-08-25 13:50:50,783 INFO [IPC Server handler 1 on 52321] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1440177002606_0009_m_000000_0 is : 0.0

2015-08-25 13:50:50,797 FATAL [IPC Server handler 2 on 52321] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1440177002606_0009_m_000000_0 - exited : cascading.tuple.TupleException: unable to read from input identifier: hdfs://quickstart.cloudera:8020/user/cloudera/data/rain.txt

at cascading.tuple.TupleEntrySchemeIterator.hasNext(TupleEntrySchemeIterator.java:152)

at cascading.flow.stream.element.SourceStage.map(SourceStage.java:84)

at cascading.flow.stream.element.SourceStage.run(SourceStage.java:66)

at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:142)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Caused by: cascading.tap.TapException: did not parse correct number of values from input data, expected: 6, got: 1:

at cascading.scheme.util.DelimitedParser.onlyParseLine(DelimitedParser.java:404)

at cascading.scheme.util.DelimitedParser.parseLine(DelimitedParser.java:341)

at cascading.scheme.hadoop.TextDelimited.source(TextDelimited.java:1015)

at cascading.tuple.TupleEntrySchemeIterator.getNext(TupleEntrySchemeIterator.java:166)

at cascading.tuple.TupleEntrySchemeIterator.hasNext(TupleEntrySchemeIterator.java:139)

... 10 more

2015-08-25 13:50:50,797 INFO [IPC Server handler 2 on 52321] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1440177002606_0009_m_000000_0: Error: cascading.tuple.TupleException: unable to read from input identifier: hdfs://quickstart.cloudera:8020/user/cloudera/data/rain.txt