com.bigdata.service
Class AbstractFederation<T>

java.lang.Object
  com.bigdata.service.AbstractFederation<T>

Type Parameters:

T - The generic type of the client or service.

All Implemented Interfaces:

ICounterSetAccess, IIndexManager, IIndexStore, IBigdataFederation<T>, IFederationDelegate<T>

Direct Known Subclasses:

AbstractScaleOutFederation

public abstract class AbstractFederation<T>
extends Object
implements IBigdataFederation<T>

Abstract base class for IBigdataFederation implementations.

Version:

$Id: AbstractFederation.java 6194 2012-03-27 14:14:52Z thompsonbry $

Author:

Bryan Thompson

TODO:

implement IServiceShutdown. When it is declared here it messes up the Options interface hierarchy. What appears to be happening is that the IServiceShutdown.Options interface is flattened into IServiceShutdown and it shadows the Options that are being used.

Nested Class Summary

static class	AbstractFederation.ReportTask Periodically report performance counter data to the ILoadBalancerService.
protected class	AbstractFederation.StartDeferredTasksTask This task starts an (optional) AbstractStatisticsCollector, an (optional) httpd service, and the (required) AbstractFederation.ReportTask.

Field Summary

protected static org.apache.log4j.Logger

log

Constructor Summary

protected

AbstractFederation(IBigdataClient<T> client)

Method Summary

ScheduledFuture<?>	addScheduledTask(Runnable task, long initialDelay, long delay, TimeUnit unit) Adds a task which will run until canceled, until it throws an exception, or until the service is shutdown.
protected void	assertOpen()
void	destroy() Destroys all discovered services belonging to the federation and their persistent data and disconnects from the federation.
void	didStart() Delegated.
void	dropIndex(String name) Drop a scale-out index.
AbstractClient<T>	getClient() Return the client object that was used to connect to the federation.
boolean	getCollectPlatformStatistics() true iff performance counters will be collected for the platform on which the client is running.
boolean	getCollectQueueStatistics() true iff statistics will be collected for work queues.
CounterSet	getCounters() The CounterSet which the client will use report its statistics to the ILoadBalancerService.
IDataService[]	getDataServices(UUID[] uuids) Return an array of IDataService references that is correlated with the given array of IDataService UUIDs.
ExecutorService	getExecutorService() A thread pool that may be used by clients to parallelize operations against the federation.
BigdataGangliaService	getGangliaService() The embedded ganglia peer.
BigdataFileSystem	getGlobalFileSystem() Return the global file system used to store block-structured files and their metadata and as a source and sink for map/reduce processing.
SparseRowStore	getGlobalRowStore() Return an unisolated view of the global SparseRowStore used to store named property sets.
SparseRowStore	getGlobalRowStore(long timestamp) Return a view of the global SparseRowStore used to store named property sets as of the specified timestamp.
CounterSet	getHostCounterSet() The node in IBigdataFederation.getCounters() corresponding to the root of the host on which the client or service is executing.
int	getHttpdPort() The port on which the optional httpd service will be run.
String	getHttpdURL() The URL that may be used to access the local httpd service for this client or service.
IClientIndex	getIndex(String name, long timestamp) Applies an AbstractIndexCache and strengthens the return type.
protected abstract AbstractIndexCache<? extends IClientIndex>	getIndexCache() Return the cache for IIndex objects.
ScaleOutIndexCounters	getIndexCounters(String name) Return the ScaleOutIndexCounters for the specified scale-out index for this client.
DefaultResourceLocator	getResourceLocator() Return the default locator for resources that are logical index containers (relations and relation containers).
ScheduledExecutorService	getScheduledExecutorService() A service which may be used to schedule performance counter sampling tasks.
T	getService() Delegated.
String	getServiceCounterPathPrefix() The path prefix under which all of the client or service's counters are located.
static String	getServiceCounterPathPrefix(UUID serviceUUID, Class serviceIface, String hostname) The path prefix under which all of the client or service's counters are located.
CounterSet	getServiceCounterSet() The node in IBigdataFederation.getCounters() corresponding to the root of the client or service connected using this federation.
Class	getServiceIface() Delegated.
String	getServiceName() Delegated.
UUID	getServiceUUID() Delegated.
TaskCounters	getTaskCounters() Return the TaskCounters which aggregate across all operations performed by the client against the connected federation.
TemporaryStore	getTempStore() A factory for TemporaryStores.
boolean	isOpen()
boolean	isServiceReady() Delegated.
AbstractHTTPD	newHttpd(int httpdPort, ICounterSetAccess accessor) Delegated.
void	reattachDynamicCounters() Delegated.
void	registerIndex(IndexMetadata metadata) Register a scale-out index.
UUID	registerIndex(IndexMetadata metadata, byte[][] separatorKeys, UUID[] dataServiceUUIDs) Register and statically partition a scale-out index.
UUID	registerIndex(IndexMetadata metadata, UUID dataServiceUUID) Register a scale-out index and assign the initial index partition to the specified data service.
void	reportCounters() Forces the immediate reporting of the CounterSet to the ILoadBalancerService.
protected void	sendEvent(Event e) Queues up an event to be sent to the ILoadBalancerService.
void	serviceJoin(IService service, UUID serviceUUID) Delegated.
void	serviceLeave(UUID serviceUUID) Delegated.
void	shutdown() Normal shutdown allows any existing client requests to federation services to complete but does not schedule new requests, disconnects from the federation, and then terminates any background processing that is being performed on the behalf of the client (service discovery, etc).
void	shutdownNow() Immediate shutdown terminates any client requests to federation services, disconnects from the federation, and then terminate any background processing that is being performed on the behalf of the client (service discovery, etc).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.bigdata.service.IBigdataFederation

getAnyDataService, getDataService, getDataServiceByName, getDataServiceUUIDs, getLastCommitTime, getLoadBalancerService, getMetadataIndex, getMetadataService, getTransactionService, isDistributed, isScaleOut, isStable

Methods inherited from interface com.bigdata.journal.IIndexStore

getResourceLockService

Field Detail

log

protected static final org.apache.log4j.Logger log

Constructor Detail

AbstractFederation

protected AbstractFederation(IBigdataClient<T> client)

Method Detail

getClient

public AbstractClient<T> getClient()

Description copied from interface: IBigdataFederation

Return the client object that was used to connect to the federation.

Specified by:

getClient in interface IBigdataFederation<T>

isOpen

public final boolean isOpen()

shutdown

public void shutdown()

Normal shutdown allows any existing client requests to federation services to complete but does not schedule new requests, disconnects from the federation, and then terminates any background processing that is being performed on the behalf of the client (service discovery, etc).

Note: concrete implementations MUST extend this method.

Note: Clients use IBigdataClient.disconnect(boolean) to disconnect from a federation. The federation implements that disconnect using either shutdown() or shutdownNow().

The implementation must be a NOP if the federation is already shutdown.

shutdownNow

public void shutdownNow()

Immediate shutdown terminates any client requests to federation services, disconnects from the federation, and then terminate any background processing that is being performed on the behalf of the client (service discovery, etc).

Note: concrete implementations MUST extend this method to either disconnect from the remote federation or close the embedded federation and then clear the #fed reference so that the client is no longer "connected" to the federation.

Note: Clients use IBigdataClient.disconnect(boolean) to disconnect from a federation. The federation implements that disconnect using either shutdown() or shutdownNow().

The implementation must be a NOP if the federation is already shutdown.

destroy

public void destroy()

Description copied from interface: IBigdataFederation

Destroys all discovered services belonging to the federation and their persistent data and disconnects from the federation.

Specified by:

destroy in interface IIndexStore

Specified by:

destroy in interface IBigdataFederation<T>

assertOpen

protected final void assertOpen()

Throws:

IllegalStateException - if the client has disconnected from the federation.

getScheduledExecutorService

public ScheduledExecutorService getScheduledExecutorService()

A service which may be used to schedule performance counter sampling tasks.

getCollectPlatformStatistics

public boolean getCollectPlatformStatistics()

true iff performance counters will be collected for the platform on which the client is running.

Specified by:

getCollectPlatformStatistics in interface IIndexStore

See Also:

IBigdataClient.Options.COLLECT_PLATFORM_STATISTICS

getCollectQueueStatistics

public boolean getCollectQueueStatistics()

true iff statistics will be collected for work queues.

Specified by:

getCollectQueueStatistics in interface IIndexStore

See Also:

IBigdataClient.Options.COLLECT_QUEUE_STATISTICS

getHttpdPort

public int getHttpdPort()

The port on which the optional httpd service will be run. The httpd service exposes statistics about the client, its work queues, the platform on which it is running, etc. If this is ZERO (0), then the port will be chosen randomly.

Specified by:

getHttpdPort in interface IIndexStore

See Also:

IBigdataClient.Options.HTTPD_PORT

getHttpdURL

public final String getHttpdURL()

Description copied from interface: IBigdataFederation

The URL that may be used to access the local httpd service for this client or service.

Specified by:

getHttpdURL in interface IBigdataFederation<T>

Returns:

The URL -or- null if the httpd service is not running.

getResourceLocator

public DefaultResourceLocator getResourceLocator()

Description copied from interface: IIndexStore

Return the default locator for resources that are logical index containers (relations and relation containers).

Specified by:

getResourceLocator in interface IIndexStore

getTaskCounters

public TaskCounters getTaskCounters()

Return the TaskCounters which aggregate across all operations performed by the client against the connected federation. These TaskCounters are sampled by a ThreadPoolExecutorStatisticsTask and the sampled data are reported by the client to the ILoadBalancerService.

getIndexCounters

public ScaleOutIndexCounters getIndexCounters(String name)

Return the ScaleOutIndexCounters for the specified scale-out index for this client. There is only a single instance per scale-out index and all operations by this client on that index are aggregated by that instance. These counters are reported by the client to the ILoadBalancerService.

Parameters:

name - The scale-out index name.

addScheduledTask

public ScheduledFuture<?> addScheduledTask(Runnable task,
                                           long initialDelay,
                                           long delay,
                                           TimeUnit unit)

Description copied from interface: IIndexStore

Adds a task which will run until canceled, until it throws an exception, or until the service is shutdown.

Specified by:

addScheduledTask in interface IIndexStore

Parameters:

task - The task.

initialDelay - The initial delay.

delay - The delay between invocations.

unit - The units for the delay parameters.

Returns:

The ScheduledFuture for that task.

getGangliaService

public final BigdataGangliaService getGangliaService()

The embedded ganglia peer. This can be used to obtain load balanced host reports, etc.

getCounters

public final CounterSet getCounters()

The CounterSet which the client will use report its statistics to the ILoadBalancerService.

Note: Applications MAY add their own counters (within a suitable namespace) to the returned CounterSet in order to report their own performance data to the ILoadBalancerService.

Note: This method must use stateful counters because the federation services all need to be able to report their history. If this were to return a new CounterSet each time, then the services would not remember any history (including the load balancer). Returning a new object every time here basically throws away the data we want.

Specified by:

getCounters in interface ICounterSetAccess

Specified by:

getCounters in interface IBigdataFederation<T>

See Also:

IBigdataFederation.getServiceCounterSet(), IBigdataFederation.getServiceCounterPathPrefix()

getHostCounterSet

public CounterSet getHostCounterSet()

Description copied from interface: IBigdataFederation

The node in IBigdataFederation.getCounters() corresponding to the root of the host on which the client or service is executing.

Specified by:

getHostCounterSet in interface IBigdataFederation<T>

getServiceCounterSet

public CounterSet getServiceCounterSet()

Description copied from interface: IBigdataFederation

The node in IBigdataFederation.getCounters() corresponding to the root of the client or service connected using this federation.

Specified by:

getServiceCounterSet in interface IBigdataFederation<T>

getServiceCounterPathPrefix

public String getServiceCounterPathPrefix()

Description copied from interface: IBigdataFederation

The path prefix under which all of the client or service's counters are located. The returned path prefix is terminated by an ICounterSet.pathSeparator.

Specified by:

getServiceCounterPathPrefix in interface IBigdataFederation<T>

See Also:

IBigdataFederation.getServiceCounterSet()

getServiceCounterPathPrefix

public static String getServiceCounterPathPrefix(UUID serviceUUID,
                                                 Class serviceIface,
                                                 String hostname)

The path prefix under which all of the client or service's counters are located. The returned path prefix is terminated by an ICounterSet.pathSeparator.

Parameters:

serviceUUID - The service UUID.

serviceIface - The primary interface or class for the service.

hostname - The fully qualified name of the host on which the service is running.

getExecutorService

public ExecutorService getExecutorService()

Description copied from interface: IBigdataFederation

A thread pool that may be used by clients to parallelize operations against the federation. This thread pool is automatically used by the ClientIndexView.

Specified by:

getExecutorService in interface IIndexStore

Specified by:

getExecutorService in interface IBigdataFederation<T>

registerIndex

public void registerIndex(IndexMetadata metadata)

Description copied from interface: IBigdataFederation

Register a scale-out index.

Specified by:

registerIndex in interface IIndexManager

Specified by:

registerIndex in interface IBigdataFederation<T>

Parameters:

metadata - The metadata template used to create component indices for BTrees this scale-out index (this also specifies the name of the scale-out index).

registerIndex

public UUID registerIndex(IndexMetadata metadata,
                          UUID dataServiceUUID)

Description copied from interface: IBigdataFederation

Register a scale-out index and assign the initial index partition to the specified data service.

Specified by:

registerIndex in interface IBigdataFederation<T>

Parameters:

metadata - The metadata template used to create component indices for BTrees this scale-out index (this also specifies the name of the scale-out index).

dataServiceUUID - The data service identifier (optional). When null, a data service will be selected automatically. If IndexMetadata.Options#INITIAL_DATA_SERVICE was specified, then the identified service will be used. Otherwise an underutilized service will be selected using the ILoadBalancerService.

Returns:

The UUID of the registered index.

See Also:

IndexMetadata.Options#INITIAL_DATA_SERVICE

registerIndex

public UUID registerIndex(IndexMetadata metadata,
                          byte[][] separatorKeys,
                          UUID[] dataServiceUUIDs)

Description copied from interface: IBigdataFederation

Register and statically partition a scale-out index.

Specified by:

registerIndex in interface IBigdataFederation<T>

Parameters:

metadata - The metadata template used to create component indices for BTrees this scale-out index (this also specifies the name of the scale-out index).

separatorKeys - The array of separator keys. Each separator key is interpreted as an unsigned byte[]. The first entry MUST be an empty byte[]. The entries MUST be in sorted order.

dataServiceUUIDs - The array of data services onto which each partition defined by a separator key will be mapped (optional). When given, the #of entries in this array MUST agree with the #of entries in the separatorKeys array and all entries must be non-null. When not given, the index partitions will be auto-assigned to the discovered data services.

Returns:

The UUID of the scale-out index.

getIndexCache

protected abstract AbstractIndexCache<? extends IClientIndex> getIndexCache()

Return the cache for IIndex objects.

getIndex

public IClientIndex getIndex(String name,
                             long timestamp)

Applies an AbstractIndexCache and strengthens the return type. Obtain a view on a partitioned index.

Specified by:

getIndex in interface IIndexStore

Specified by:

getIndex in interface IBigdataFederation<T>

Parameters:

name - The index name.

timestamp - A transaction identifier, ITx.UNISOLATED for the unisolated index view, ITx.READ_COMMITTED, or timestamp for a historical view no later than the specified timestamp.

Returns:

The index or null if the index does not exist.

dropIndex

public void dropIndex(String name)

Description copied from interface: IBigdataFederation

Drop a scale-out index.

Specified by:

dropIndex in interface IIndexManager

Specified by:

dropIndex in interface IBigdataFederation<T>

Parameters:

name - The index name.

getGlobalRowStore

public SparseRowStore getGlobalRowStore()

Description copied from interface: IIndexStore

Return an unisolated view of the global SparseRowStore used to store named property sets.

Specified by:

getGlobalRowStore in interface IIndexStore

See Also:

GlobalRowStoreSchema

getGlobalRowStore

public SparseRowStore getGlobalRowStore(long timestamp)

Description copied from interface: IIndexStore

Return a view of the global SparseRowStore used to store named property sets as of the specified timestamp.

The SparseRowStore only permits ITx.UNISOLATED writes, so you MUST specify ITx.UNISOLATED as the timestamp if you intend to write on the global row store!

You can request the most recent committed state of the global row store by specifying ITx.READ_COMMITTED.

Specified by:

getGlobalRowStore in interface IIndexStore

Parameters:

timestamp - The timestamp of the view.

Returns:

The global row store view -or- null if no view exists as of that timestamp.

getGlobalFileSystem

public BigdataFileSystem getGlobalFileSystem()

Description copied from interface: IIndexStore

Return the global file system used to store block-structured files and their metadata and as a source and sink for map/reduce processing.

Specified by:

getGlobalFileSystem in interface IIndexStore

See Also:

BigdataFileSystem

getTempStore

public TemporaryStore getTempStore()

Description copied from interface: IIndexStore

A factory for TemporaryStores. TemporaryStores are thread-safe and may be used by multiple processes at once. Old TemporaryStores are eventually retired by the factory and their storage is reclaimed once they are finalized (after they are no longer in use by any process). The decision to retire a TemporaryStore is either made implicitly, when it is no longer weakly reachable, or explicitly, when it has grown large enough that no new processes should begin using that TemporaryStore. In the latter case, the TemporaryStore will remain available to the process(es) using it and a new TemporaryStore will be allocated and made available to the caller.

It is important that processes do not hold a hard reference to a TemporaryStore beyond the end of the process as that will prevent the TemporaryStore from being finalized. Holding reference to an AbstractBTree created on a TemporaryStore is equivalent to holding a hard reference to the TemporaryStore itself since the AbstractBTree holds onto the backing IRawStore using a hard reference.

Specified by:

getTempStore in interface IIndexStore

Returns:

A TemporaryStore.

reportCounters

public void reportCounters()

Forces the immediate reporting of the CounterSet to the ILoadBalancerService. Any errors will be logged, not thrown.

getService

public T getService()

Delegated. Return the client or service.

Specified by:

getService in interface IFederationDelegate<T>

getServiceName

public String getServiceName()

Delegated. Return a name for the service. It is up to administrators to ensure that service names are unique.

Specified by:

getServiceName in interface IFederationDelegate<T>

Returns:

A name for the service.

getServiceIface

public Class getServiceIface()

Delegated. Return the class or interface that is the most interesting facet of the client and which will be used to identify this client in the performance counters reported to the ILoadBalancerService.

Specified by:

getServiceIface in interface IFederationDelegate<T>

Returns:

The class or interface and never null.

getServiceUUID

public UUID getServiceUUID()

Delegated. The UUID assigned to the IBigdataClient or AbstractService.

Specified by:

getServiceUUID in interface IFederationDelegate<T>

See Also:

AbstractService.setServiceUUID(UUID)

isServiceReady

public boolean isServiceReady()

Delegated. Return true iff the service is ready to start.

Specified by:

isServiceReady in interface IFederationDelegate<T>

reattachDynamicCounters

public void reattachDynamicCounters()

Delegated. Offers the service an opportunity to dynamically detach and re-attach performance counters. This can be invoked either in response to an http GET or the periodic reporting of performance counters to the ILoadBalancerService. In general, implementations should limit the frequency of update, e.g., to no more than once a second.

Note: For most purposes, this has been replaced by ICounterSetAccess which is now passed into CounterSetHTTPD. That provides the necessary indirection for periodic refresh of the performance counters. The CounterSetHTTPD now also handles the limitation on the update frequency for the materialized counters.

However, there are still some counters which need to be dynamically reattached. For example, any counter set which is dynamic in its structure, such as the DirectBufferPool.

Specified by:

reattachDynamicCounters in interface IFederationDelegate<T>

didStart

public void didStart()

Delegated. Invoked by the AbstractFederation once the deferred startup tasks are executed. Services may use this event to perform additional initialization.

Specified by:

didStart in interface IFederationDelegate<T>

newHttpd

public AbstractHTTPD newHttpd(int httpdPort,
                              ICounterSetAccess accessor)
                       throws IOException

Delegated. Create a new AbstractHTTPD instance.

Specified by:

newHttpd in interface IFederationDelegate<T>

accessor - Used to materialize the CounterSet that will be served up.

Returns:

The httpd daemon.

Throws:

IOException

serviceJoin

public void serviceJoin(IService service,
                        UUID serviceUUID)

Delegated. Notice that the service has been discovered. This notice will be generated the first time the service is discovered by a given IBigdataClient.

Specified by:

serviceJoin in interface IFederationDelegate<T>

Parameters:

service - The service.

serviceUUID - The service UUID.

serviceLeave

public void serviceLeave(UUID serviceUUID)

Delegated. Notice that the service is no longer available. This notice will be generated once for a given IBigdataClient when the service is no longer available from any of its service registrars.

Specified by:

serviceLeave in interface IFederationDelegate<T>

Parameters:

serviceUUID - The service UUID.

getDataServices

public IDataService[] getDataServices(UUID[] uuids)

Description copied from interface: IBigdataFederation

Return an array of IDataService references that is correlated with the given array of IDataService UUIDs.

Note: This method will also resolve the UUID of an IMetadataService.

Specified by:

getDataServices in interface IBigdataFederation<T>

Returns:

The (meta)data service proxies.

TODO:

it may be possible to optimize this for the jini case.

sendEvent

protected void sendEvent(Event e)

Queues up an event to be sent to the ILoadBalancerService. Events are maintained on a non-blocking queue (no fixed capacity) and sent by a scheduled task.

Parameters:

e -

See Also:

SendEventsTask