com.bigdata.service
Class LoadBalancerService

java.lang.Object
  com.bigdata.service.AbstractService
      com.bigdata.service.LoadBalancerService

All Implemented Interfaces:

IEventReceivingService, IEventReportingService, ILoadBalancerService, IService, IServiceShutdown, Remote

Direct Known Subclasses:

LoadBalancerServer.AdministrableLoadBalancer

public abstract class LoadBalancerService
extends AbstractService
implements ILoadBalancerService, IServiceShutdown, IEventReportingService

The LoadBalancerService collects a variety of performance counters from hosts and services, identifies over- and under- utilized hosts and services based on the collected data and reports those to DataService s so that they can auto-balance, and acts as a clearing house for WARN and URGENT alerts for hosts and services.

While the LoadBalancerService MAY observe service start/stop events, it does NOT get directly informed of actions that change the load distribution, such as index partition moves or reading from a failover service. Instead, DataServices determine whether or not they are overloaded and, if so, query the LoadBalancerService for the identity of under-utilized services. If under-utilized DataServices are reported by the LoadBalancerService then the DataService will self-identify index partitions to be shed and move them onto the identified under-utilized DataServices. The LoadBalancerService learns of these actions solely through their effect on host and service load as self- reported by various services.

Note: utilization should be defined in terms of transient system resources : CPU, IO (DISK and NET), RAM. DISK exhaustion on the other hand is the basis for WARN or URGENT alerts since it can lead to immediate failure of all services on the same host.

Note: When new services are made available, either on new hosts or on the existing hardware, and service utilization discrepancies should become rapidly apparent (within a few minutes). Once we have collected performance counters for the new hosts / services, a subsequent overflow event(s) on existing DataService(s) will cause index partition moves to be nominated targeting the new hosts and services. The amount of time that it takes to re-balance the load on the services will depend in part on the write rate, since writes drive overflow events and index partition splits, both of which lead to pre-conditions for index partition moves.

Note: If a host is suffering high IOWAIT then it is probably "hot for read" (writes are heavily buffered and purely sequential and therefore unlikely to cause high IOWAIT where as reads are typically random on journals even through a key range scan is sequential on index segments). Therefore a "hot for read" condition should be addressed by increasing the replication count for those service(s) which are being swamped by read requests on a host suffering from high IOWAIT.

See Also:

IRequiredHostCounters, A core set of variables to support decision-making., http://www.google.com/search?hl=en&q=load+balancing+jini

TODO:

we could significantly accelerate overflow events when new hardware is made available by setting the forceOverflow flag on the highly utilized data services. we probably don't want to do this for all highly utilized services at once since that could cause a lot of perturbation., Work out high-level alerting for resource exhaustion and failure to maintain QOS on individual machines, indices, and across the federation., All clients (IBigdataClient, DataService, etc) should issue WARN and URGENT notices. The client-side rules for those alerts should be configurable / pluggable / declarative. It would be great if the WARN and URGENT notices were able to carry some information about the nature of the emergency., logging on this log can provide a single point for administrators to configure email or other alerts., refactor so that we can use the same basic infrastructure for load balancing of other service classes as well, e.g., map/reduce services., ping hosts?, SNMP

http://en.wikipedia.org/wiki/Simple_Network_Management_Protocol

http://www.snmp4j.org/

http://sourceforge.net/projects/joesnmp/ (used by jboss)

http://net-snmp.sourceforge.net/ (Un*x and Windows SNMP support)., Consider an SNMP adaptor for the clients so that they can report to SNMP aware applications. In this context we could report both the original counters (as averaged) and the massaged metrics on which we plan to make decisions., Would it make sense to replace the counters XML mechanism with MXBeans specific to bigdata and additional MXBeans for performance counters for the operating system?

Nested Class Summary

static interface	LoadBalancerService.Options Options understood by the LoadBalancerService.
protected class	LoadBalancerService.RoundRobinServiceLoadHelper Integration with the LoadBalancerService.
protected class	LoadBalancerService.ServiceLoadHelperWithoutScores Integration with the LoadBalancerService.
protected class	LoadBalancerService.ServiceLoadHelperWithScores Integration with the LoadBalancerService.
protected class	LoadBalancerService.UpdateTask Computes and updates the ServiceScores based on an examination of aggregated performance counters.

Field Summary

protected ConcurrentHashMap<UUID,ServiceScore>	activeDataServices The set of active services.
protected ConcurrentHashMap<String,HostScore>	activeHosts The active hosts (one or more services).
protected EventReceiver	eventReceiver
protected Journal	eventStore Used to persist the logged events.
protected int	historyMinutes The #of minutes of history that will be smoothed into an average when LoadBalancerService.UpdateTask updates the HostScores and the ServiceScores.
protected AtomicReference<HostScore[]>	hostScores Scores for the hosts in ascending order (least utilized to most utilized).
protected long	initialRoundRobinUpdateCount The #of updates during which getUnderUtilizedDataServices(int, int, UUID) will apply a round robin policy.
protected boolean	isTransient true iff the LBS will refrain from writing state on the disk.
protected Condition	joined Used to await a service join when there are no services.
protected ReentrantLock	lock Lock is used to control access to data structures that are not thread-safe.
protected static org.apache.log4j.Logger	log
protected File	logDir The directory in which the service will log the CounterSets and Events.
protected long	nupdates The #of LoadBalancerService.UpdateTasks which have run so far.
protected String	ps
protected long	serviceJoinTimeout Service join timeout in milliseconds - used when we need to wait for a service to join before we can recommend an under-utilized service.
protected AtomicReference<ServiceScore[]>	serviceScores Scores for the services in ascending order (least utilized to most utilized).
protected ScheduledExecutorService	updateService Runs a periodic LoadBalancerService.UpdateTask.

Constructor Summary

LoadBalancerService(Properties properties)
Note: The load balancer MUST NOT collect host statistics unless it is the only service running on that host.

Method Summary

void	destroy() Destroy the service.
protected void	finalized()
protected abstract String	getClientHostname() Return the canonical hostname of the client in the context of a RMI request.
Properties	getProperties() An object wrapping the properties provided to the constructor.
Class	getServiceIface() Returns ILoadBalancerService.
UUID	getUnderUtilizedDataService() Return the UUID of an under-utilized data service.
UUID[]	getUnderUtilizedDataServices(int minCount, int maxCount, UUID exclude) Return up to limit IDataService UUIDs that are currently under-utilized.
protected boolean	isHighlyUtilizedDataService(ServiceScore score, ServiceScore[] scores)
boolean	isHighlyUtilizedDataService(UUID serviceUUID) Return true if the service is considered to be "highly utilized".
boolean	isOpen() Return true iff the service is running.
protected boolean	isUnderUtilizedDataService(ServiceScore score, ServiceScore[] scores)
boolean	isUnderUtilizedDataService(UUID serviceUUID) Return true if the service is considered to be "under-utilized".
void	join(UUID serviceUUID, Class serviceIface, String hostname) Notify the LoadBalancerService that a new service is available.
void	leave(UUID serviceUUID) Notify the LoadBalancerService that a service is no longer available.
void	logCounters() Logs the counters on a file created using File.createTempFile(String, String, File) in the log directory.
protected void	logCounters(File file) Writes the counters on a file.
protected void	logCounters(String basename) Writes the counters on a file.
void	notify(UUID serviceUUID, byte[] data) Send performance counters.
void	notifyEvent(Event e) Accepts the event, either updates the existing event with the same UUID or adds the event to the set of recent events, and then prunes the set of recent events so that all completed events older than #eventHistoryMillis are discarded.
long	rangeCount(long fromTime, long toTime) Reports the #of completed events that start in the given interval.
Iterator<Event>	rangeIterator(long fromTime, long toTime) Visits completed events that start in the given interval in order by their start time.
protected void	setHostScores(HostScore[] a) Normalizes the ServiceScores and set them in place.
protected void	setServiceScores(ServiceScore[] a) Normalizes the ServiceScores and set them in place.
void	shutdown() The service will no longer accept new requests, but existing requests will be processed (sychronous).
void	shutdownNow() The service will no longer accept new requests and will make a best effort attempt to terminate all existing requests and return ASAP.
void	sighup() Logs counters to a temp file.
LoadBalancerService	start() Starts the AbstractService.
void	urgent(String msg, UUID serviceUUID) An urgent warning issued the caller is in immediate danger of depleting its resources with a consequence of immediate service and/or host failure(s).
void	warn(String msg, UUID serviceUUID) A warning issued by a client when it is in danger of depleting its resources.

Methods inherited from class com.bigdata.service.AbstractService

clearLoggingContext, getFederation, getHostname, getServiceName, getServiceUUID, setServiceUUID, setupLoggingContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.bigdata.service.IService

getHostname, getServiceName, getServiceUUID

Field Detail

log

protected static final org.apache.log4j.Logger log

ps

protected final String ps

See Also:

Constant Field Values

serviceJoinTimeout

protected final long serviceJoinTimeout

Service join timeout in milliseconds - used when we need to wait for a service to join before we can recommend an under-utilized service.

See Also:

LoadBalancerService.Options.SERVICE_JOIN_TIMEOUT

lock

protected final ReentrantLock lock

Lock is used to control access to data structures that are not thread-safe.

joined

protected final Condition joined

Used to await a service join when there are no services.

activeHosts

protected ConcurrentHashMap<String,HostScore> activeHosts

The active hosts (one or more services).

TODO:

get rid of hosts that are no longer active. e.g., we no longer receive #notify(String, byte[]) events from the host and the host can not be pinged. this will require tracking the #of services on the host which we do not do directly right now.

activeDataServices

protected ConcurrentHashMap<UUID,ServiceScore> activeDataServices

The set of active services.

hostScores

protected AtomicReference<HostScore[]> hostScores

Scores for the hosts in ascending order (least utilized to most utilized).

This array is initially null and gets updated periodically by the LoadBalancerService.UpdateTask. The main consumer of this information is the logic in LoadBalancerService.UpdateTask that computes the service utilization.

serviceScores

protected AtomicReference<ServiceScore[]> serviceScores

Scores for the services in ascending order (least utilized to most utilized).

This array is initially null and gets updated periodically by the LoadBalancerService.UpdateTask. The methods that report service utilization and under-utilized services are all based on the data in this array. Since services can leave at any time, that logic MUST also test for existence of the service in activeDataServices before assuming that the service is still live.

nupdates

protected long nupdates

The #of LoadBalancerService.UpdateTasks which have run so far.

See Also:

LoadBalancerService.Options.INITIAL_ROUND_ROBIN_UPDATE_COUNT, getUnderUtilizedDataServices(int, int, UUID)

initialRoundRobinUpdateCount

protected final long initialRoundRobinUpdateCount

The #of updates during which getUnderUtilizedDataServices(int, int, UUID) will apply a round robin policy.

See Also:

LoadBalancerService.Options.INITIAL_ROUND_ROBIN_UPDATE_COUNT

logDir

protected final File logDir

The directory in which the service will log the CounterSets and Events.

See Also:

LoadBalancerService.Options.LOG_DIR

isTransient

protected final boolean isTransient

true iff the LBS will refrain from writing state on the disk. This option causes the LBS to use an in memory eventStore. In addition, it will refuse to write counter snapshots when this option is specified.

See Also:

LoadBalancerService.Options.TRANSIENT

updateService

protected final ScheduledExecutorService updateService

Runs a periodic LoadBalancerService.UpdateTask.

historyMinutes

protected final int historyMinutes

The #of minutes of history that will be smoothed into an average when LoadBalancerService.UpdateTask updates the HostScores and the ServiceScores.

See Also:

LoadBalancerService.Options.HISTORY_MINUTES

eventStore

protected final Journal eventStore

Used to persist the logged events.

eventReceiver

protected final EventReceiver eventReceiver

Constructor Detail

LoadBalancerService

public LoadBalancerService(Properties properties)

Note: The load balancer MUST NOT collect host statistics unless it is the only service running on that host. Normally it relies on another service running on the same host to collect statistics for that host and those statistics are then reported to the load balancer and aggregated along with the rest of the performance counters reported by the other services in the federation. However, if the load balancer itself collects host statistics then it will only know about and report the current (last 60 seconds) statistics for the host rather than having the historical data for the host.

Parameters:

properties - See LoadBalancerService.Options

Method Detail

getProperties

public Properties getProperties()

An object wrapping the properties provided to the constructor.

getClientHostname

protected abstract String getClientHostname()

Return the canonical hostname of the client in the context of a RMI request. If the request is not remote then return the canonical hostname for this host.

start

public LoadBalancerService start()

Description copied from class: AbstractService

Starts the AbstractService.

Note: A AbstractService.start() is required in order to give subclasses an opportunity to be fully initialized before they are required to begin operations. It is impossible to encapsulate the startup logic cleanly without this ctor() + start() pattern. Those familiar with Objective-C will recognized this.

Specified by:

start in class AbstractService

Returns:

this (the return type should be strengthened by the concrete implementation to return the actual type).

isOpen

public boolean isOpen()

Description copied from interface: IServiceShutdown

Return true iff the service is running.

Specified by:

isOpen in interface IServiceShutdown

finalized

protected void finalized()
                  throws Throwable

Throws:

Throwable

shutdown

public void shutdown()

Description copied from interface: IServiceShutdown

The service will no longer accept new requests, but existing requests will be processed (sychronous). This method should await the termination of pending requests, but no longer than the timeout specified by IServiceShutdown.Options.SHUTDOWN_TIMEOUT. Implementations SHOULD be synchronized. If the service is aleady shutdown, then this method should be a NOP.

Specified by:

shutdown in interface IServiceShutdown

Overrides:

shutdown in class AbstractService

shutdownNow

public void shutdownNow()

Description copied from interface: IServiceShutdown

The service will no longer accept new requests and will make a best effort attempt to terminate all existing requests and return ASAP. This method should terminate any asynchronous processing, release all resources and return immediately. Implementations SHOULD be synchronized. If the service is aleady shutdown, then this method should be a NOP.

Specified by:

shutdownNow in interface IServiceShutdown

Overrides:

shutdownNow in class AbstractService

destroy

public void destroy()

Description copied from interface: IService

Destroy the service. If the service is running, it is shutdown immediately and then destroyed. This method has the same signature as DestroyAdmin.destroy().

Specified by:

destroy in interface IService

Overrides:

destroy in class AbstractService

getServiceIface

public final Class getServiceIface()

Returns ILoadBalancerService.

Specified by:

getServiceIface in interface IService

Specified by:

getServiceIface in class AbstractService

setHostScores

protected void setHostScores(HostScore[] a)

Normalizes the ServiceScores and set them in place.

Parameters:

a - The new service scores.

setServiceScores

protected void setServiceScores(ServiceScore[] a)

Normalizes the ServiceScores and set them in place.

Parameters:

a - The new service scores.

logCounters

protected void logCounters(String basename)

Writes the counters on a file.

Parameters:

basename - The basename of the file. The file will be written in the logDir.

logCounters

protected void logCounters(File file)

Writes the counters on a file.

Parameters:

file - The file. If the file exists it will be overwritten.

logCounters

public void logCounters()
                 throws IOException

Logs the counters on a file created using File.createTempFile(String, String, File) in the log directory.

Throws:

IOException

TODO:

this method is not exposed to RMI (it is not on any Remote interface) but it could be.

sighup

public void sighup()
            throws IOException

Description copied from interface: ILoadBalancerService

Logs counters to a temp file. Replacement for sighup mechanism.

Specified by:

sighup in interface ILoadBalancerService

Throws:

IOException

join

public void join(UUID serviceUUID,
                 Class serviceIface,
                 String hostname)

Notify the LoadBalancerService that a new service is available.

Note: Embedded services must invoke this method directly when they start up.

Note: Distributed services implementations MUST discover services using a framework, such as jini, and invoke this method the first time a given service is discovered.

Parameters:

serviceUUID -

serviceIface -

hostname -

See Also:

IFederationDelegate.serviceJoin(IService, UUID), #leave(String, UUID)

leave

public void leave(UUID serviceUUID)

Notify the LoadBalancerService that a service is no longer available.

Note: Embedded services must invoke this method directly when they shut down.

Note: Distributed services implementations MUST discover services using a framework, such as jini, and invoke this method when a service is no longer registered.

Parameters:

serviceUUID - The service UUID.

See Also:

IFederationDelegate.serviceLeave(UUID), join(UUID, Class, String)

notifyEvent

public void notifyEvent(Event e)
                 throws IOException

Accepts the event, either updates the existing event with the same UUID or adds the event to the set of recent events, and then prunes the set of recent events so that all completed events older than #eventHistoryMillis are discarded.

Specified by:

notifyEvent in interface IEventReceivingService

Throws:

IOException

See Also:

EventReceiver

rangeIterator

public Iterator<Event> rangeIterator(long fromTime,
                                     long toTime)

Visits completed events that start in the given interval in order by their start time.

Specified by:

rangeIterator in interface IEventReportingService

rangeCount

public long rangeCount(long fromTime,
                       long toTime)

Reports the #of completed events that start in the given interval.

Specified by:

rangeCount in interface IEventReportingService

Parameters:

fromTime - The first start time to be included.

toTime - The first start time to be excluded.

Returns:

The #of events whose start time is in that interval.

notify

public void notify(UUID serviceUUID,
                   byte[] data)

Description copied from interface: ILoadBalancerService

Send performance counters. Clients SHOULD invoke this method no less than once every 60 seconds.

Specified by:

notify in interface ILoadBalancerService

Parameters:

serviceUUID - The service UUID that is self-reporting.

data - The serialized performance counter data.

warn

public void warn(String msg,
                 UUID serviceUUID)

Description copied from interface: ILoadBalancerService

A warning issued by a client when it is in danger of depleting its resources.

Specified by:

warn in interface ILoadBalancerService

Parameters:

msg - A message.

serviceUUID - The service UUID that is self-reporting.

urgent

public void urgent(String msg,
                   UUID serviceUUID)

Description copied from interface: ILoadBalancerService

An urgent warning issued the caller is in immediate danger of depleting its resources with a consequence of immediate service and/or host failure(s).

Specified by:

urgent in interface ILoadBalancerService

Parameters:

msg - A message.

serviceUUID - The service UUID that is self-reporting.

isHighlyUtilizedDataService

public boolean isHighlyUtilizedDataService(UUID serviceUUID)
                                    throws IOException

Description copied from interface: ILoadBalancerService

Return true if the service is considered to be "highly utilized".

Note: This is used mainly to decide when a service should attempt to shed index partitions. This implementation SHOULD reflect the relative rank of the service among all services as well as its absolute load.

Specified by:

isHighlyUtilizedDataService in interface ILoadBalancerService

Parameters:

serviceUUID - The service UUID.

Returns:

true if the service is considered to be "highly utilized".

Throws:

IOException

isUnderUtilizedDataService

public boolean isUnderUtilizedDataService(UUID serviceUUID)
                                   throws IOException

Description copied from interface: ILoadBalancerService

Return true if the service is considered to be "under-utilized".

Specified by:

isUnderUtilizedDataService in interface ILoadBalancerService

Parameters:

serviceUUID - The service UUID.

Returns:

true if the service is considered to be "under-utilized".

Throws:

IOException

isHighlyUtilizedDataService

protected boolean isHighlyUtilizedDataService(ServiceScore score,
                                              ServiceScore[] scores)

isUnderUtilizedDataService

protected boolean isUnderUtilizedDataService(ServiceScore score,
                                             ServiceScore[] scores)

getUnderUtilizedDataService

public UUID getUnderUtilizedDataService()
                                 throws IOException,
                                        TimeoutException,
                                        InterruptedException

Description copied from interface: ILoadBalancerService

Return the UUID of an under-utilized data service. If there is no under-utilized service, then return the UUID of the service with the least load.

Specified by:

getUnderUtilizedDataService in interface ILoadBalancerService

Throws:

TimeoutException - if there are no data services and a timeout occurs while awaiting a service join.

InterruptedException - if the request is interrupted.

IOException

getUnderUtilizedDataServices

public UUID[] getUnderUtilizedDataServices(int minCount,
                                           int maxCount,
                                           UUID exclude)
                                    throws IOException,
                                           TimeoutException,
                                           InterruptedException

Description copied from interface: ILoadBalancerService

Return up to limit IDataService UUIDs that are currently under-utilized.

When minCount is positive, this method will always return at least minCount service UUIDs, however the UUIDs returned MAY contain duplicates if the LoadBalancerService has a strong preference for allocating load to some services (or for NOT allocating load to other services). Further, the LoadBalancerService MAY choose (or be forced to choose) to return UUIDs for services that are within a nominal utilization range, or even UUIDs for services that are highly-utilized if it could otherwise not satisify the request.

Specified by:

getUnderUtilizedDataServices in interface ILoadBalancerService

Parameters:

minCount - The minimum #of services UUIDs to return -or- zero (0) if there is no minimum limit.

maxCount - The maximum #of services UUIDs to return -or- zero (0) if there is no maximum limit.

exclude - The optional UUID of a data service to be excluded from the returned set.

Returns:

Up to maxCount under-utilized services -or- null IFF no services are recommended at this time as needing additional load.

Throws:

TimeoutException - if there are no data services, or if there is only a single data service and it is excluded by the request, and a timeout occurs while awaiting a service join.

InterruptedException - if the request is interrupted.

IOException