com.bigdata.jini.start
Class ServicesManagerServer

java.lang.Object
  com.bigdata.service.jini.AbstractServer
      com.bigdata.jini.start.ServicesManagerServer

All Implemented Interfaces:

Runnable, EventListener, net.jini.lease.LeaseListener, net.jini.lookup.ServiceIDListener

public class ServicesManagerServer
extends AbstractServer

A class for bootstrapping a JiniFederation across a cluster based on a Configuration file describing some basic configuration parameters and dynamically managing service configurations. The service configurations are stored in zookeeper and each activated service registers both a persistent znode (for restart) and a variety of transient znodes (to register its claim as an instance of a logical service, for master elections, etc).

Each host in a cluster runs ONE (1) of this server. The server will start other services on the local host. Those decisions are based on configuration requirements stored in zookeeper, including the target #of logical services of some service type, the replication count for a logical service, and IServiceConstraint that are used to decide which host will start the new service.

Services are broken down by service type (aka class), by logical service instance, and by physical instances of a logical service. Physical service instances may be transient or persistent. The configuration state required to re-start a persistent physical service is stored in a persistent znode representing that service. For transient services, that znode is ephemeral.

The initial configuration for the federation given in a Configuration file. That file is best located on a shared volume or volume image replicated on the hosts in the cluster, along with the various dependencies required to start and run the federation (java, jini, bigdata, log4j configuration, related JARs, etc). The information in the Configuration is used to discover and/or start jini and zookeeper instance(s). The ServicesManagerServer maintains a watch on the ServiceConfiguration znodes for the various kinds of services and will start or stop services as the state of those nodes changes.

Service Monitoring

The ServicesManagerServer maintains watches on znodes representing ServiceConfigurations, logical services, physical service instances, etc. If the number of logical services instances for some service type falls below the ServiceConfiguration.serviceCount then a new logical service will be created. Likewise, if the number of physical service instances for some logical service falls below the ServiceConfiguration.replicationCount then a new physical service instance will be started.

Physical services which are persistent are represented by persistent znodes within the BigdataZooDefs.PHYSICAL_SERVICES_CONTAINER for a logical service. This means that the #of physical service instances is NOT decreased when persistent physical service dies or is shutdown normally but ONLY when the physical service instance is destroyed (along with its persistent state). If a persistent physical service instance has died and can not be restarted, then you MUST delete its znode in the BigdataZooDefs.PHYSICAL_SERVICES_CONTAINER before a new physical service will be created for the corresponding logical service.

Physical services which are transient are represented by ephemeral znodes. As soon as one dies and zookeeper notices that its client has timed out the ephemeral znode will be deleted and a watch triggered which will result in a new instance being created on some machine running a ServicesManagerServer joined with the federation.

Most bigdata services have persistent state and may be safely shutdown and restarted. To shutdown a bigdata service instance use RemoteDestroyAdmin.shutdown() or zap it from the command line (not the preferred approach, but that will only take the service down, but not destroy its data).

Startup

Zookeeper service and jini instances are handled somewhat specially. Each is a self-contained system and provides its own means for peers to discover each other and for clients to discover the peers. The remainder of the services use jini to perform service discovery and zookeeper to register ephemeral znodes which represent their existence as part of a logical service and allow them to contend in the master election for a logical service. Only those services listed in the ServicesManagerConfiguration will be considered for start.

Zookeeper startup

A zookeeper service instance will be started if one was identified in the Configuration as running on this host in the ZookeeperServerConfiguration and none is found to be running at that time.

Jini startup

A jini server instance will be considered for start if one is specified for this host in the JiniCoreServicesConfiguration using a LookupLocator or if you are using multicast. In addition, the IServiceConstraints for the jini server in the Configuration must be satisified by this host.

Configuration push

During startup, the ServiceConfigurations will be pushed to zookeeper. This will trigger a variety of watches on the znodes for the service configurations both in this instance and in other ServicesManagerServers running on other hosts which are part of the same federation.

Since the ServiceConfigurations are pushed each time a ServicesManagerServer is started it is CRITICAL that all ServicesManagerServers use the same configuration. The Configuration may either be a file on a shared volume or a URL.

Persistent service restart

Once we are connected to both jini and zookeeper a scan will be performed of the persistent physical services registered in zookeeper. For each such service, if the service was started on this host and can not be discovered using jini, then an attempt will be made to restart the service. We don't want to do this on an ongoing basis because it would cause any service that was deliberately shutdown to be restarted as soon as we discover that it is no longer discoverable.

Service replication and failover

Jini and zookeeper each have their own failover models.

Jini

You can run a number of jini peers and any client can connect to any registrar. This is all transparent. You can even start a new jini instance after all have died. Services which have already been discovered will remain reachable (assuming that a network path exists) for some time after the last jini registrar has died.

Zookeeper

Zookeeper uses a quorum model. You always want to provision an odd number of zookeeper instances. ONE (1) instance gives you no failover (but the instance may be restarted if it dies). THREE (3) instances give you ONE (1) failure.

Persistent bigdata services

Replication and service failover have NOT been implemented for bigdata

When the physical service supports failover, the shutdown of a physical service instance does not take the logical service offline if there are remaining physical services instances for that logical service. However, the loss of all physical instances of a logical service will result in the loss of the persistent state for the logical service as a whole.

The two main services which require failover are the ITransactionService and the IDataService (which includes the IMetadataService as a special case). The loss of the ILoadBalancerService is normally not critical as only history about the system load over time is lost.

Transaction service

tdb

(Meta)data services

The basic design is for IDataService is to pipeline state changes from a master through a failover chain of secondaries. Write can proceed asynchronously on the pipeline until the next commit, at which point the secondaries must be synched with the master. Since all instances have the same state for any commit point, you can read from any instance but you always write on the master.

Asynchronous overflow operations are carried out by the master (could be a secondary, but that raises the complexity of the operation) and the generated IndexSegments are replicated to the secondaries before atomic updates which redefined index views.

Destroying services

Destroying an arbitrary service instance is dangerous - if it is not replicated then you can loose all your data!.

To destroy a bigdata service usie RemoteDestroyAdmin.destroy().

Federation shutdown sequence

See JiniFederation.distributedFederationShutdown(boolean)

Shutdown

Normal shutdown will kill child processes, using their normal shutdown whenever possible. Once the child processes are dead, the server will terminate as well. Use RemoteDestroyAdmin.shutdown() and RemoteDestroyAdmin.shutdownNow() for normal shutdown.

Signal handling

HUP

This executes the same logic that is describe above for startup. However, it will re-read the Configuration from whatever source was specified when the server was started. This signal may be used to update the zookeeper configuration, to restart any stopped persistent services, etc.

See ServicesManagerStartupTask.

TERM

All child processes are terminated, using normal shutdown whenever possible. Once the child processes are dead, the server will terminate as well. (This is normal shutdown.)

KILL

This signal is NOT trapped. The server will terminate immediately. Any child processes will continue to execute (this is the behavior on at least Windows and linux platforms).

Please note that Java has limited support for signal handlers, so these behaviors might not be available on your deployment platform. When in doubt, TEST FIRST!

Version:

$Id$

Author:

Bryan Thompson

See Also:

BigdataZooDefs

TODO:

document dependencies for performance counter reporting and supported platforms. perhaps config options for which counters are collected and which are reported to the LBS., It is possible to create locks and queues using javaspaces in a manner similar to zookeeper. Java spaces supports a concept similar to a watch and supports transactions over operations on the space. Gigaspaces has defined a variety of extensions that provide FIFO queues., The MetadataService, the ILoadBalancerService, and the ITransactionService MUST NOT have more than one logical instance in a federation. They can (eventually) have failover instances, but not peers. The DataService is the only one that already supports multiple logical service instances (lots!) (but not failover)., Operator alerts should be generated when persistent physical services die (this can be noticed either via jini or when the ephemeral znode for the master election queue goes away). FIXME remaining big issues are destroying logical and physical services (not implemented yet) and providing failover for the various bigdata services (not implemented yet).

Nested Class Summary

static class	ServicesManagerServer.AdministrableServicesManagerService Adds jini administration interfaces.
static interface	ServicesManagerServer.Options Configuration options.

Nested classes/interfaces inherited from class com.bigdata.service.jini.AbstractServer

AbstractServer.ConfigurationOptions, AbstractServer.MasterElectionTask

Field Summary

protected static org.apache.log4j.Logger

log

Fields inherited from class com.bigdata.service.jini.AbstractServer

config, impl, logicalServiceZPath, physicalServiceZPath, proxy

Constructor Summary

ServicesManagerServer(String[] args, com.sun.jini.start.LifeCycle lifeCycle)
Ctor for jini service activation.

Method Summary

static void	main(String[] args) Starts and maintains services based on the specified configuration file and/or an existing zookeeper ensemble.
protected ServicesManagerServer.AdministrableServicesManagerService	newService(Properties properties) This method is responsible for creating the remote service implementation object.

Methods inherited from class com.bigdata.service.jini.AbstractServer

destroy, fatal, getClient, getFileFilter, getHostName, getJoinManager, getProxy, getServiceID, getServiceName, isPersistent, isShuttingDown, notify, notifyServiceUUID, readServiceId, run, runDestroy, serviceIDNotify, setSecurityManager, shutdownNow, terminate, toString, unexport

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

log

protected static final org.apache.log4j.Logger log

Constructor Detail

ServicesManagerServer

public ServicesManagerServer(String[] args,
                             com.sun.jini.start.LifeCycle lifeCycle)

Ctor for jini service activation.

Parameters:

args - Either the command line arguments or the arguments from the ServiceDescriptor. Either way they identify the jini Configuration (you may specify either a file or URL) and optional overrides for that Configuration.

lifeCycle - The life cycle object. This is used if the server is started by the jini ServiceStarter. Otherwise specify a FakeLifeCycle.

Method Detail

main

public static void main(String[] args)

Starts and maintains services based on the specified configuration file and/or an existing zookeeper ensemble.

 java -Djava.security.policy=policy.all com.bigdata.jini.start.ServicesManager src/resources/config/bigdata.config
 

Parameters:

args - The command line arguments.

newService

protected ServicesManagerServer.AdministrableServicesManagerService newService(Properties properties)

Description copied from class: AbstractServer

This method is responsible for creating the remote service implementation object. This object MUST declare one or more interfaces that extent the Remote interface. The server will use JERI to create a proxy for the remote object and configure and manage the protocol for communications between the client (service proxy) and the remote object (the service implementation).

Note: You have to implement JoinAdmin in order to show up as an administerable service (blue folder) in the jini Service Browser.

Specified by:

newService in class AbstractServer

Parameters:

properties - The contents of the Properties file whose name was given by the propertyFile value in the Configuration identified to main by its command line arguments.