com.bigdata.service.ndx
Interface IAsynchronousWriteBufferFactory

All Known Subinterfaces:

IScaleOutClientIndex

All Known Implementing Classes:

AbstractScaleOutClientIndexView, AbstractScaleOutClientIndexView2, AggregatorTask, ClientIndexView, ClientIndexViewRefactor

public interface IAsynchronousWriteBufferFactory

Interface for asynchronous writes on scale-out indices.

Version:

$Id: IAsynchronousWriteBufferFactory.java 2265 2009-10-26 12:51:06Z thompsonbry $

Author:

Bryan Thompson

Method Summary

<T extends IKeyArrayIndexProcedure,O,R,A> IRunnableBuffer<KVO<O>[]>

newWriteBuffer(IResultHandler<R,A> resultHandler, IDuplicateRemover<O> duplicateRemover, AbstractKeyArrayIndexProcedureConstructor<T> ctor) Asynchronous write API (streaming writes).

Method Detail

newWriteBuffer

<T extends IKeyArrayIndexProcedure,O,R,A> IRunnableBuffer<KVO<O>[]> newWriteBuffer(IResultHandler<R,A> resultHandler,
                                                                                   IDuplicateRemover<O> duplicateRemover,
                                                                                   AbstractKeyArrayIndexProcedureConstructor<T> ctor)

Asynchronous write API (streaming writes).

The returned buffer provides a streaming API which is highly efficient. The caller writes ordered KVO[] chunks onto the thread-safe BlockingBuffer. Those chunks are dynamically combined and then split into per-index partition chunks which are written on internally managed BlockingBuffers for each index partition which will be touched by a write operation. The splits are slices of ordered chunks for a specific index partition. The BlockingBuffer uses a merge sort when it combines ordered chunks so that the combined chunks remain fully ordered. Once a chunk is ready, it is re-shaped for the CTOR and sent to the target data service using RMI.

Since this API is asynchronous, you will not have synchronous access to values returned by asynchronous writes. However, patterns can be created using KVOC and KVOLatch which provide notification when application defined sets of results have become available. Such patterns are created by associated the KVOLatch with the set of results and using IResultHandler and the object reference on the KVOC to capture the side-effect of the write.

BlockingBuffer.getFuture() may be used to obtain the Future of the consumer. You can use Future.get() to await the completion of the consumer, to cancel the consumer, etc. The Future will not terminate (other than by error) until the buffer has been closed. The Future evaluates to an IndexAsyncWriteStats object. Those statistics are also reported to the ILoadBalancerService via the IBigdataFederation.

Each buffer returned by this method is independent, and writes onto independent sinks which write through to the index partitions. This is necessary in order for the caller to retain control over the life cycle of their write operations. The BlockingBuffer is thread-safe so it may be the target for concurrent producers can be can utilized to create very high throughput designs. While the returned buffers are independent, the performance counters for all asynchronous write buffers for a given client and scale-out index are aggregated by a single ScaleOutIndexCounters instance.

Type Parameters:

T - The generic type of the procedure used to write on the index.

O - The generic type for unserialized value objects.

R - The type of the result from applying the index procedure to a single Split of data.

A - The type of the aggregated result.

Parameters:

resultHandler - Used to aggregate results.

duplicateRemover - Used to filter out duplicates in an application specified manner (optional).

ctor - Used to create instances of the procedure that will execute a write on an individual index partition (this implies that insert and remove operations as well as custom index write operations must use separate buffers).

Returns:

A buffer on which the producer may write their data.

See Also:

IndexMetadata.getAsynchronousIndexWriteConfiguration(), AbstractFederation.getIndexCounters(String)

TODO:

The async API is only defined at this time for scale-out index views. An asynchronous write API could be defined for local B+Trees. It would have to ensure locks using the UnisolatedReadWriteIndex. It would not use the same layering since writes could not be scattered. It could be written as a single blocking buffer which was drained by the CTOR for the operation. If we combine the two buffer capacity parameters into a single parameter, then this method signature could be used for both local and scale-out index views. This method could then be moved to an IAsynchronousIndexWriter interface.