com.bigdata.service
Class LoadBalancerService.UpdateTask

java.lang.Object
  com.bigdata.service.LoadBalancerService.UpdateTask

All Implemented Interfaces:

Runnable

Enclosing class:

LoadBalancerService

protected class LoadBalancerService.UpdateTask
extends Object
implements Runnable

Computes and updates the ServiceScores based on an examination of aggregated performance counters.

Version:

$Id: LoadBalancerService.java 6194 2012-03-27 14:14:52Z thompsonbry $

Author:

Bryan Thompson

See Also:

QueryUtil.getRequiredPerformanceCountersFilter()

TODO:

There could be a score for the last minute, hour, and day or the last minute, five minutes, and ten minutes., For starters, we can just run some hand-coded rules. Consider special transition states for new hosts and services., The scoring logic should be pluggable so that people can reply on the data that they have for their platform(s) that seems to best support decision-making and can apply rules for their platforms, environment, and applications which provide the best overall QOS., The logic to choose the under- and over-utilized services based on the services scores should be configurable (this is different from the logic to compute those scores)., if a client does not ILoadBalancerService#notify(String, byte[]) for 120 seconds then presume dead? this requires that we compute the age of the last reported counter value. e.g., do a counter scan for the service and report the largest value for lastModified() on any counter for that service.

Field Summary

protected org.apache.log4j.Logger

log Note: The logger is named for this class, but since it is an inner class the name uses a "$" delimiter (vs a ".") between the outer and the inner class names.

Constructor Summary

LoadBalancerService.UpdateTask()

Method Summary

protected ServiceScore	computeScore(HostScore hostScore, UUID serviceUUID, ICounterSet hostCounterSet, ICounterSet serviceCounterSet) Compute the score for a service.
protected HostScore	computeScore(String hostname, ICounterSet hostCounterSet) Compute the score for a host.
protected double	getAverageValueForMinutes(ICounterSet counterSet, String path, double defaultValue, int minutes) Return the average of the counter having the given path over the last minutes minutes.
protected double	getCurrentValue(ICounterSet counterSet, String path, double defaultValue)
protected void	logCounters() Writes the counters on a file.
void	run() Note: Don't throw anything here since we don't want to have the task suppressed!
protected void	setupCounters() Sets up reporting for the computed per-host and per-service scores.
protected void	updateHostScores() (Re-)compute the utilization score for each active host.
protected void	updateServiceScores() (Re-)compute the utilization score for each active service.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected final transient org.apache.log4j.Logger log

Note: The logger is named for this class, but since it is an inner class the name uses a "$" delimiter (vs a ".") between the outer and the inner class names.

Constructor Detail

LoadBalancerService.UpdateTask

public LoadBalancerService.UpdateTask()

Method Detail

run

public void run()

Note: Don't throw anything here since we don't want to have the task suppressed!

Specified by:

run in interface Runnable

updateHostScores

protected void updateHostScores()

(Re-)compute the utilization score for each active host.

updateServiceScores

protected void updateServiceScores()

(Re-)compute the utilization score for each active service.

Note: There is a dependency on AbstractFederation.getServiceCounterPathPrefix(UUID, Class, String). This method assumes that the service UUID is found in a specific place in the constructed path.

computeScore

protected HostScore computeScore(String hostname,
                                 ICounterSet hostCounterSet)

Compute the score for a host.

The host scores MUST reflect critical resource exhaustion, especially DISK free space, which can take down all services on the host, and SWAPPING, which can bring the effective throughput of the host to a halt. All other resources fail soft, by causing the response time to increase.

Note: DISK exhaustion can lead to immediate failure of all services on the same host. A host that is nearing DISK exhaustion SHOULD get heavily dinged and an admin SHOULD be alerted.

The correct response for heavy swapping is to alert an admin to shutdown one or more processes on that host. If you do not have failover provisioned for your data services then DO NOT shutdown data services or you WILL loose data!

Note: If we are not getting critical counters for some host then we are assuming a reasonable values for the missing data and computing the utilization based on those assumptions. Note that a value of zero (0) may be interpreted as either critically high utilization or no utilization depending on the performance counter involved and that the impact of the different counters can vary depending on the formula used to compute the utilization score.

Parameters:

hostname - The fully qualified hostname.

hostCounterSet - The performance counters for that host.

serviceCounterSet - The performance counters for that service.

Returns:

The computed host score.

computeScore

protected ServiceScore computeScore(HostScore hostScore,
                                    UUID serviceUUID,
                                    ICounterSet hostCounterSet,
                                    ICounterSet serviceCounterSet)

Compute the score for a service.

Note: utilization is defined in terms of transient system resources : CPU, IO (DISK and NET), RAM. A host with enough CPU/RAM/IO/DISK can support more than one data service. Therefore it is important to look at not just host utilization but also at process utilization.

Parameters:

hostScore - The pre-computed score for the host on which the service is running.

serviceUUID - The service UUID.

hostCounterSet - The performance counters for that host (in case you need anything that is not already in the HostScore).

serviceCounterSet - The performance counters for that service.

Returns:

The computed score for that service.

getCurrentValue

protected double getCurrentValue(ICounterSet counterSet,
                                 String path,
                                 double defaultValue)

getAverageValueForMinutes

protected double getAverageValueForMinutes(ICounterSet counterSet,
                                           String path,
                                           double defaultValue,
                                           int minutes)

Return the average of the counter having the given path over the last minutes minutes.

Parameters:

counterSet -

path -

defaultValue -

minutes -

Returns:

FIXME should be a weighted average, right?

setupCounters

protected void setupCounters()

Sets up reporting for the computed per-host and per-service scores. These counters are reported under the service UUID for the LoadBalancerService itself. This makes it easy to consult the scores for the various hosts and services.

Note: The host and service scores will not appear until the LoadBalancerService.UpdateTask has executed and those scores have been computed.

See Also:

LoadBalancerService.Options.UPDATE_DELAY

TODO:

counters for service scores should be eventually removed after the service leaves. Likewise for host scores. However, these counters SHOULD remain available for a while for post-mortem of the service/host, e.g., at least 2-3 days. This would be fixed with a persistence model for the scores.

logCounters

protected void logCounters()

Writes the counters on a file.

See Also:

LoadBalancerService.Options