Package com.bigdata.service

This package provides implementations of bigdata services (metadata service, data service, transaction manager service.

See:
Description

Interface Summary

AbstractScaleOutClient.Options	Options understood by the AbstractScaleOutClient.
AbstractTransactionService.Options	Options understood by this service.
ClientService.Options	Configuration options.
DataService.IDataServiceCounters	Interface defines and documents the counters and counter namespaces reported by the DataService and the various services which it uses.
DataService.Options	Options understood by the DataService.
DistributedTransactionService.Options	Options understood by this service.
IBigdataClient<T>	Interface for clients of a IBigdataFederation.
IBigdataClient.Options	Configuration options for IBigdataClients.
IBigdataFederation<T>	The client-facing interface to a bigdata federation.
IClientService	A service for distributing client Callables across the resources of an IBigdataFederation.
IDataService	The data service interface provides remote access to named indices, provides for both unisolated and isolated operations on those indices, and exposes the ITxCommitProtocol interface to the ITransactionManagerService service for the coordination of distributed transactions.
IDataServiceCallable	Interface for procedures that require access to the IDataService and or the federation.
IEventReceivingService	Remote interface for a service which can receive Events.
IEventReportingService	Extension of the common service interface to support event reporting.
IFederationCallable	Interface for Callables which require access to the IBigdataFederation when running on an IRemoteExecutor.
IFederationDelegate<T>	Interface allowing services to take over handling of events normally handled by the AbstractFederation.
ILoadBalancerService	Interface for collecting, reporting, and decision-making based on node and service utilization statistics.
IMetadataService	A metadata service for a named index.
IRemoteExecutor	Interface for running procedures on a remote service.
IService	Common service interface.
IServiceLoadHelper	Interface for decision making about the load imposed on services.
IServiceShutdown	Local API for service shutdown.
IServiceShutdown.Options	Options for IServiceShutdown implementations.
ISession	Non-remote interface exposing a transient property set associated with a service.
ITxCommitProtocol	Remote interface by which the ITransactionService manages the state of transactions on the distributed IDataServices.
LoadBalancerService.Options	Options understood by the LoadBalancerService.
MetadataService.Options	Options for the MetadataService.
Params	An interface designed to expose select fields for Event reporting.

Class Summary

AbstractClient<T>	Abstract base class for IBigdataClient implementations.
AbstractDistributedFederation<T>	Abstract base class for IBigdataFederation implementations where the services are distributed using RMI and are running, at least in principle, across more than one host/JVM.
AbstractFederation<T>	Abstract base class for IBigdataFederation implementations.
AbstractFederation.ReportTask	Periodically report performance counter data to the ILoadBalancerService.
AbstractIndexCache<T extends IRangeQuery>	Abstract base class providing caching for IIndex like objects.
AbstractRoundRobinServiceLoadHelper	A round robin implementation that may be used when there are no scores available.
AbstractScaleOutClient<T>	Client class for AbstractScaleOutFederations.
AbstractScaleOutFederation<T>	Abstract base class for federation implementations using the scale-out index architecture (federations that support key-range partitioned indices).
AbstractScaleOutFederation.ForceOverflowTask	Task forces immediate overflow of the specified data service, returning once both synchronous AND asynchronous overflow are complete.
AbstractScaleOutFederation.PurgeResourcesTask	Task directs a DataService to purge any unused resources and to optionally truncate the extent of the live journal.
AbstractService	Abstract base class defines protocols for setting the service UUID, etc.
AbstractServiceLoadHelper	Base class for abstract implementations with integration points for the LoadBalancerService.
AbstractServiceLoadHelperWithoutScores	Implementation that may be used when service scores are not yet available.
AbstractServiceLoadHelperWithScores	The default implementation used when scores are available.
AbstractTransactionService	Centralized transaction manager service.
CacheOnceMetadataIndex	Implementation caches all locators but does not allow stale locators.
CachingMetadataIndex	Implementation caches all locators and then updates them on demand as stale locators are discovered.
ClientService	A service for distributing application Callables across an IBigdataFederation.
ClientService.ClientServiceFederationDelegate	Extended to attach the various performance counters reported by the DistributedTransactionService.
CommitTimeIndex	BTree whose keys are commit times.
CommitTimeIndex.TupleSerializer	Encapsulates key and value formation.
DataService	An implementation of a network-capable IDataService.
DataService.DataServiceFederationDelegate	Delegate handles custom counters for the ResourceManager, local AbstractTransactionService and the ConcurrencyManager, dynamic re-attachment of counters, etc.
DataService.GetIndexMetadataTask	Retrieves the IndexMetadata for the named index as of the specified timestamp.
DataService.RangeIteratorTask	Task for running a rangeIterator operation.
DataService.ReadBlockCounters
DataServiceCallable<T>	Base class for IDataServiceCallable.
DefaultClientDelegate<T>	Default IFederationDelegate implementation used by a standard client.
DefaultServiceFederationDelegate<T extends AbstractService>	Basic delegate for services that need to override the service UUID and service interface reported to the ILoadBalancerService.
DistributedTransactionService	Implementation for an IBigdataFederation supporting both single-phase commits (for transactions that execute on a single IDataService) and distributed commits.
DistributedTransactionService.SnapshotHelper	A helper class for reading and writing snapshots of the commit time index.
Event	An event.
EventReceiver	Class capable of receiving Events from remote services.
EventReceiver.EventBTree	A BTree whose keys are event start times and whose values are the serialized Events.
EventReceiver.EventBTree.EventBTreeTupleSerializer	Encapsulates key and value formation for the EventReceiver.EventBTree.
EventResource	Semi-structured representation of the data service on which the event occurred, the name of the index, and the index partition identifier.
FederationCallable<T>	Base class for IFederationCallable.
HostScore	Per-host metadata and a score for that host which gets updated periodically by LoadBalancerService.UpdateTask.
IndexCache	Concrete implementation for IClientIndex views.
ListIndicesTask	Task returns an array of the named indices on the DataService to which it is submitted.
LoadBalancerService	The LoadBalancerService collects a variety of performance counters from hosts and services, identifies over- and under- utilized hosts and services based on the collected data and reports those to DataService s so that they can auto-balance, and acts as a clearing house for WARN and URGENT alerts for hosts and services.
ManagedResourceService	This class manages a pool of direct ByteBuffers.
MetadataIndexCache	Concrete implementation for IMetadataIndex views.
MetadataService	Implementation of a metadata service for a named scale-out index.
MetadataService.DropScaleOutIndexTask	Drops a scale-out index.
MetadataService.JoinIndexPartitionTask	Updates the MetadataIndex to reflect the join of 2 or more index partitions.
MetadataService.MoveIndexPartitionTask	Updates the MetadataIndex to reflect the move of an index partition.
MetadataService.NextPartitionIdTask	Task assigns the next partition identifier for a registered scale-out index in a restart-safe manner.
MetadataService.RegisterScaleOutIndexTask	Registers a metadata index for a named scale-out index and statically partition the index using the given separator keys and data services.
MetadataService.SplitIndexPartitionTask	Atomic operation removes the pre-existing entry for specified index partition and replaces it with N new entries giving the locators for the N new index partitions created when that index partition was split.
NoCacheMetadataIndexView	An implementation that performs NO caching.
ResourceService	A service which permits resources (managed files or buffers) identified by a UUID to be read by a remote service.
ResourceService.Counters	Performance counters for the ResourceService.
ResourceService.FetchResourceTask<S,T>	Client for a BufferService reads a single resource from the specified service, writing it into the local file system.
ResourceService.ReadBufferTask	Class sends a request for a remote ByteBuffer and then receives the data into a local ByteBuffer.
ResourceService.ReadResourceTask	Task sends a request for a file's data and then receives the data onto a local file.
ServiceScore	Per-service metadata and a score for that service which gets updated periodically by the LoadBalancerService.UpdateTask.
Session	A (transient) property set associated with some kinds of services.
Split	Describes a "split" of keys for a batch operation that are spanned by the same index partition.
TxId2CommitTimeIndex	BTree whose keys are commit times.
TxId2CommitTimeIndex.TupleSerializer	Encapsulates key and value formation.

Enum Summary

AbstractScaleOutClient.MetadataIndexCachePolicy	Policy options for caching PartitionLocators for an IMetadataIndex.
EventType	Type safe enum for Events.
ResourceService.ResourceTypeEnum	Type safe enumeration of the kinds of resources which can be served.
ResourceService.StatusEnum	Known status codes.
TxServiceRunState	Run states for the AbstractTransactionService.

Exception Summary

NoSuchService

Exception thrown when a service was requested but has not been discovered or is otherwise not available.

Package com.bigdata.service Description

This package provides implementations of bigdata services (metadata service, data service, transaction manager service.

A bigdata federation is comprised of the following essential services:

metadata service

The metadata service manages scale-out indices and is used to locate index partitions.

data service

The data service supports reads and writes on index partitions.

Clients are responsible for discovering the metadata service. Clients then use the metadata service to manage scale-out indices and to locate and cache leases for data service instances having data for key range partitions that the client will read or write. If a client will use transactions (vs ACID batch operations), then the client must also locate the transaction manager service. Service location is performed using JINI, but other service locator protocols are possible.

bigdata provides the following optional services. These are not considered essential since (a) they are not required by clients that make direct use of bigdata as a distributed database; and (b) they are themselves applications of the services briefly described above.

transaction manager service

Clients use the transaction manager service to start new transactions and to coordinate the distributed commit protocol. The transaction manager also provides a centralized timestamp factory.

map and reduce service

Manages the distributed execution of a functional program.

deployment

In order to deploy a bigdata federation, the following preconditions must be met (a standalone deployment can be realized by meeting these preconditions on a single host). This will go more smoothly if JINI is running before you start the various bigdata services since they will then be able to register themselves immediately on startup. Service discovery by default uses the local network and the "public" group. If you want to run more than one bigdata federation on the same network, then you MUST edit the configuration file.

Hosts:

• java 5 or better is installed. The use of a server VM is highly recommended. The following options are recommended for some well known VMs:

  Sun

  -server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=256M

  JRockit

  -Xms1g -Xmx1g

  TODO:

  review the use of -Xms (sets the initial heap size). How much of a difference does this make? Note that JRockit defaults to 75% of physical memory or 1G, which ever is less.

• the bigdata-server distribution is installed.
• the data service is running.

In addition, there must be at least one host on which the following preconditions are true (a different host could be used for each of these services, and more than one instance of these services may be running on the network). These services should be started during the boot procedure so that they are available whenever the host is up and running.

• jini 2.x is installed and running. This will be used to register and lookup services. It helps if jini is running before you try to start the bigdata services since they will be able to register themselves immediately in that case.
• a metadata server is running.
• one or more data server(s) are running.
• a transaction server is running (optional).
• one or more map and reduce servers are running (optional).

Clients:

• java 5 or better is installed. The use of a server VM is highly recommended
• the bigdata-client deployment package is installed. This includes bigdata-client.jar, but also support for generating unicode keys (ICU4J), etc.

unique identifiers

Data Service

The serviceID identifies the data service. If the service dies and then restarts on the same host with the same data, then it is still the same service instance. If the service dies and is recreated on another host with the same data, then it is still the same instance. What is important is that (a) the service has the same data (journals, index segments, and serviceID); and (b) that the data for that service has not become stale (or inconsistent).

The easiest way in which the data can become stale is for the service to fail. When failover is operating, clients will be redirected to one or more other services having consistent data for the same index partitions. If clients then write on _any_ of those index partitions and commit, then the old service now has stale (aka inconsistent) data. A service with inconsistent data MUST NOT be used.

In general, the majority of state for a service will live in its index segments. This can be hundreds or thousands of times more data in the index segments than there is in any single journal. This can make it worth while to make the service consistent again.

Journal

Each journal has a UUID that is generated when the journal is created (it is stored in the root block). This serves to uniquely identify the journal as a resource regardless of where it may be located on the host or in the file system.

Index Segment

Each index segment has a UUID that is generated when the index segment is created (it is stored in the fixed length index segment metadata record). This serves to uniquely identify the index segment as a resource regardless of where it may be located on the host or in the file system.

Index

Named indices are assigned UUIDs when they are created. This makes it possible to rename an index, since its UUID remains the same.

metadata service

transaction manager service

job scheduler service

downloaded code

While bigdata makes use of jini for service registration and discovery, it does not use downloaded code for executing its basic services (the necessary interfaces and implementation classes are already bundled in bigdata.jar). However, clients MAY use downloaded code in order to have data servers execute remote procedures or extension batch operations. In this scenario, the client will create and register a JINI service. The service should perform all of its operations locally. The data server will lookup the service using jini and then execute it locally.

In order for downloaded code to work correctly, clients must ensure that their classes are exposed by an HTTP server accessible to bigdata servers and declared to the VM in which the client starts using: -Djava.rmi.server.codebase=...

, consider this use case further. Are there other/better ways for executing remote code on the data server? At a minimum, we should supply a base class and ant script support for creating such remote services.