/mapreduce/mapperCallback being called excessively

[Christopher]

Recently in the past 10 hours this url has been hit (though task
queues) 100k times eating through the free quota, and now pretty much
any other request bombs out because of this. It looks like every call
is having problems split on a single entity. Here's info output from
the logs:

2011-08-05 12:25:26.125
com.google.appengine.tools.mapreduce.DatastoreInputSplit readFields:
Initialized DatastoreInputSplit CloudAccount(652783) null
2011-08-05 12:09:20.061
com.google.appengine.tools.mapreduce.MapReduceServlet processMapper:
Out of mapper quota. Aborting request until quota is replenished.
Consider increasing mapreduce.mapper.inputprocessingrate (default
1000) if you would like your mapper job to complete faster.

The note about the input processing rate is currently set as the
default (1000ms). We only have 113 CloudAccount entities to map over
in the app which this occurred on. This mapper has been running for
the past 7 days without a problem. No other exceptions have popped
up, except for recently since we've hit the Quota:

/mapreduce/mapperCallback
com.google.apphosting.api.ApiProxy$OverQuotaException: The API call
taskqueue.BulkAdd() required more quota than is available.
at com.google.apphosting.runtime.ApiProxyImpl
$AsyncApiFuture.rpcFinished(ApiProxyImpl.java:418)
at com.google.net.rpc.RpcStub$RpcCallbackDispatcher
$1.runInContext(RpcStub.java:1050)
at com.google.tracing.TraceContext$TraceContextRunnable
$1.run(TraceContext.java:449)
at com.google.tracing.TraceContext.runInContext(TraceContext.java:
689)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:
327)
at com.google.tracing.TraceContext
$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:
319)
at com.google.tracing.TraceContext
$TraceContextRunnable.run(TraceContext.java:447)
at com.google.net.rpc.RpcStub
$RpcCallbackDispatcher.rpcFinished(RpcStub.java:1071)
at com.google.net.rpc.RPC.internalFinish(RPC.java:2205)
at com.google.net.rpc.impl.RpcNetChannel.finishRpc(RpcNetChannel.java:
2338)
at
com.google.net.rpc.impl.RpcNetChannel.messageReceived(RpcNetChannel.java:
1267)
at
com.google.net.rpc.impl.RpcConnection.parseMessages(RpcConnection.java:
328)
at
com.google.net.rpc.impl.RpcConnection.dataReceived(RpcConnection.java:
299)
at com.google.net.async.Connection.handleReadEvent(Connection.java:
474)
at
com.google.net.async.EventDispatcher.processNetworkEvents(EventDispatcher.java:
848)
at
com.google.net.async.EventDispatcher.internalLoop(EventDispatcher.java:
220)
at com.google.net.async.EventDispatcher.loop(EventDispatcher.java:
116)
at com.google.net.async.GlobalEventRegistry
$2.runLoop(GlobalEventRegistry.java:95)
at com.google.net.async.LoopingEventDispatcher
$EventDispatcherThread.run(LoopingEventDispatcher.java:385)