Minor patch: how unresponsive instances are replaced
shlomo....@gmail.com
Here is a minor patch to PoolManager.java. It changes the way
unresponsive servers are replaced: instead of first terminating the
unresponsive instance and then launching a replacement, the patch has
the replacement launch first and then the termination happen second.
This is useful because it allows the PoolMonitor to differentiate
between the following two cases:
a: the pool has naturally shrunk to zero size in the normal way
b: the only remaining instance in the pool is unresponsive and is
being replaced with a new instance.
Before the patch, case (b) would look the same to the PoolMonitor as
case (a): the number of instances in the pool would be zero.
After the patch, case (b) does not have the pool size decrease to zero.
I want to differentiate between these two cases because I have written
a custom PoolMonitor that terminates itself when the pool size returns
to zero.
I launch the pool manager after I have finished ingesting all my
workloads, and the pool manager shuts itself down when the pool size
returns to zero. I settled on this behavior because my workflows are
ingested at a much slower rate than they are consumed. Therefore, if
the pool manager is active during ingestion, service instances will
sit idle most of the time. Now, I launch the pool manager after all
ingestion is complete, and the instances have all the workloads
waiting on the queue for them to process. The instances can work at
maximum efficiency, for the least amount of billable hours. When they
are done, the pool manager exits, resetting the system for the next
batch job.
.. Shlomo
Index: PoolManager.java
===================================================================
--- PoolManager.java (revision 1185)
+++ PoolManager.java (working copy)
@@ -356,8 +356,8 @@
// if more than N minutes have gone by without a report, do the needful
if (i.lastReportTime < (System.currentTimeMillis()-laggardLimit)) {
logger.error("Instance "+i.id+" is being replaced");
- terminateInstances(new Instance [] {i}, true);
launchInstances(1);
+ terminateInstances(new Instance [] {i}, true);
}
}
