com.bigdata.resources
Class MoveTask

java.lang.Object
  com.bigdata.journal.AbstractTask<T>
      com.bigdata.resources.AbstractResourceManagerTask<T>
          com.bigdata.resources.AbstractPrepareTask<MoveResult>
              com.bigdata.resources.MoveTask

All Implemented Interfaces:

ITask<MoveResult>, Callable<MoveResult>

public class MoveTask
extends AbstractPrepareTask<MoveResult>

Task moves an index partition to another IDataService.

This task runs as a historical read operation and copy the view of the index partition as of the lastCommitTime of old journal to another IDataService. Once that historical view has been copied, this task then submits an AtomicUpdateMoveIndexPartitionTask. The atomic update is an ITx.UNISOLATED operation. It is responsible copying any writes buffered for the index partition on the live journal to the target IDataService and then updating the MetadataIndex. Once the atomic update task is finished, clients will discover that the source index partition does not exist. When they query the MetadataService they will discover that the key(-range) is now handled by the new index partition on the target IDataService.

Note: This task is run on the target IDataService and it copies the data from the source IDataService. This allows us to use standard IRangeQuery operations to copy the historical view. However, the AtomicUpdateMoveIndexPartitionTask is run on the source IDataService since it needs to obtain an exclusive lock on the index partition that is being moved in order to prevent concurrent writes during the atomic cutover. For the same reason, the AtomicUpdateMoveIndexPartitionTask can not use standard IRangeQuery operations. Instead, it initiates a series of data transfers while holding onto the exclusive lock until the target IDataService has the current state of the index partition. At that point it notifies the IMetadataService to perform the atomic cutover to the new index partition.

Note: This task does NOT cause any resources associated with the current view of the index partition to be released on the source IDataService. The reason is two-fold. First, the IndexSegment(s) associated with that view MAY be in used by historical views. Second, there MAY be historical commit points for the index partition on the live journal before the atomic cutover to the new IDataService - those historical commit points MUST be preserved until the release policy for those views has been satisfied.

Note: The MOVE task MUST be explicitly coordinated with the target IDataService. Failure to coordinate the move results in an error message reported by the MetadataService indicating that the wrong partition locator was found under the key. The cause is a MOVE operation during which the target data service undergoes concurrent synchronous (and then asynchronous) overflow. What happens is the MoveTask registers the new index partition on the target data service. One registered on the IDataService, the index partition it is visible during synchronous overflow BEFORE the MOVE is complete and BEFORE the index is registered with the MetadataService and hence discoverable to clients. If the target IDataService then undergoes synchronous and asynchronous overflow and chooses an action which would change the index partition definition (split, join, or move) WHILE the index partition is still being moved onto the target IDataService THEN the MOVE is not atomic and the definition of the index partition in the MetadataService will not coherently reflect either the MOVE or the action chosen by the target IDataService, depending on which one makes its atomic update first.

The target IDataService MAY undergo both synchronous and asynchronous overflow as IDataServices are designed to allow continued writes during those operations. Further, it MAY choose to copy, build, or compact the index partition while it is being moved. However, it MUST NOT choose any action (split, join, or move) that would change the index partition definition until the move is complete (whether it ends in success or failure).

This issue is addressed by the following protocol:

1. The MoveTask set the sourcePartitionId on the LocalPartitionMetadata when it registers the index partition on the target IDataService. When sourcePartitionId != -1. the target IDataService is restricted to for that index partition to overflows actions which do not change the index partition definition (copy, build, or merge). Further, any index partition found on restart whose by the target IDataService whose sourcePartitionId != -1 is deleted as it was never successfully put into play (this prevents partial moves from accumulating state which could not otherwise be released.)
2. The atomic update task causes the sourcePartitionId to be set to -1 as one of its last actions, thereby allowing the target IDataService to use operations that could re-define the index partition (split, join, move) and also preventing the target index partition from being deleted on restart.

FIXME javadoc

Note: There are only two entry points: a simple move and a move where the compacting merge has already been performed, e.g., by a split, and we just need to do the atomic update phase.

Author:

Bryan Thompson

Nested Class Summary

protected static class	MoveTask.AtomicUpdate Moves an index partition from this data service to another data service.
protected static class	MoveTask.ReceiveIndexPartitionTask Receives an index partition comprised of a historical index segment store and an index segment store containing the buffered writes and registers the index partition on the data service on which this procedure is executed.

Nested classes/interfaces inherited from class com.bigdata.journal.AbstractTask

AbstractTask.DelegateTask<T>, AbstractTask.InnerReadWriteTxServiceCallable, AbstractTask.InnerWriteServiceCallable<T>, AbstractTask.ResubmitException

Field Summary

Fields inherited from class com.bigdata.resources.AbstractResourceManagerTask

DEBUG, INFO, log, resourceManager

Fields inherited from class com.bigdata.journal.AbstractTask

checkpointNanoTime, concurrencyManager, isReadWriteTx, nanoTime_assignedWorker, nanoTime_beginWork, nanoTime_finishedWork, nanoTime_submitTask, readOnly, taskCounters, timestamp, transactionManager, tx

Constructor Summary

MoveTask(com.bigdata.resources.ViewMetadata vmd, UUID targetDataServiceUUID)

Method Summary

protected void	clearRefs() Method is responsible for clearing the SoftReferences held by ViewMetadata for the source view(s) on the old journal.
protected static MoveResult	doAtomicUpdate(ResourceManager resourceManager, String sourceIndexName, BuildResult historicalWritesBuildResult, UUID targetDataServiceUUID, int targetIndexPartitionId, Event parentEvent) Submits an MoveTask.AtomicUpdate and awaits and returns its outcome.
protected MoveResult	doTask() Builds a compact index segment from the historical view as of the last commit time on the old journal and then submits an atomic update operation to move the source index partition to the target data service.

Methods inherited from class com.bigdata.resources.AbstractResourceManagerTask

toString

Methods inherited from class com.bigdata.journal.AbstractTask

assertResource, assertRunning, assertUnisolated, call, clearLoggingContext, dropIndex, getCommitTime, getIndex, getJournal, getOnlyResource, getResource, getResourceManager, getTaskCounters, getTaskName, getTimestamp, isResource, registerIndex, setupLoggingContext, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

MoveTask

public MoveTask(com.bigdata.resources.ViewMetadata vmd,
                UUID targetDataServiceUUID)

Parameters:

vmd - Metadata for the source index partition view.

targetDataServiceUUID - The UUID for the target data service.

Method Detail

clearRefs

protected void clearRefs()

Description copied from class: AbstractPrepareTask

Method is responsible for clearing the SoftReferences held by ViewMetadata for the source view(s) on the old journal.

Note: This method MUST be invoked in order to permit those references to be cleared more eagerly than the end of the entire asynchronous overflow operation (which is when the task references would themselves go out of scope and become available for GC).

Specified by:

clearRefs in class AbstractPrepareTask<MoveResult>

doTask

protected MoveResult doTask()
                     throws Exception

Builds a compact index segment from the historical view as of the last commit time on the old journal and then submits an atomic update operation to move the source index partition to the target data service.

Specified by:

doTask in class AbstractTask<MoveResult>

Returns:

A MoveResult describing the move operation (this is returned mainly for historical reasons).

Throws:

Exception - The exception that will be thrown by AbstractTask.call() iff the operation fails.

InterruptedException - This exception SHOULD be thrown if Thread.interrupted() becomes true during execution.

doAtomicUpdate

protected static MoveResult doAtomicUpdate(ResourceManager resourceManager,
                                           String sourceIndexName,
                                           BuildResult historicalWritesBuildResult,
                                           UUID targetDataServiceUUID,
                                           int targetIndexPartitionId,
                                           Event parentEvent)
                                    throws InterruptedException,
                                           ExecutionException

Submits an MoveTask.AtomicUpdate and awaits and returns its outcome.

Parameters:

resourceManager - The resource manager.

sourceIndexName - The name of the source index partition.

historicalWritesBuildResult - An index segment containing all data for the source view as of the last commit time on the old journal. This index segment should be generated by a compacting merge or by an index partition split with the same semantics so that we will move the minimum amount of data.

targetDataServiceUUID - The UUID of the target data service.

targetIndexPartitionId - The partition identifier assigned to the target index partition.

parentEvent -

Throws:

ExecutionException

InterruptedException