com.bigdata.service.ndx
Class AbstractScaleOutClientIndexView

java.lang.Object
  extended by com.bigdata.service.ndx.AbstractScaleOutClientIndexView
All Implemented Interfaces:
IAutoboxBTree, IIndex, IRangeQuery, ISimpleBTree, IAsynchronousWriteBufferFactory, IClientIndex, IScaleOutClientIndex, ISplitter
Direct Known Subclasses:
AbstractScaleOutClientIndexView2

public abstract class AbstractScaleOutClientIndexView
extends Object
implements IScaleOutClientIndex

Abstract base class for the IScaleOutClientIndex implementation(s).

Version:
$Id: AbstractScaleOutClientIndexView.java 2265 2009-10-26 12:51:06Z thompsonbry $
Author:
Bryan Thompson
TODO:
If the index was dropped then that should cause the operation to abort (only possible for read committed or unisolated operations).

Likewise, if a transaction is aborted, then then index should refuse further operations., detect data service failure and coordinate cutover to the failover data services. ideally you can read on a failover data service at any time but it should not accept write operations unless it is the primary data service in the failover chain.

Offer policies for handling index partitions that are unavailable at the time of the request (continued operation during partial failure)., We should be able to transparently use either a hash mod N approach to distributed index partitions or a dynamic approach based on overflow. This could even be decided on a per-index basis. The different approaches would be hidden by appropriate implementations of this class.

A hash partitioned index will need to enforce optional read-consistent semantics. This can be done by choosing a recent broadcast commitTime for the read or by re-issuing queries that come in with a different commitTime.


Field Summary
protected  boolean batchOnly
          This may be used to disable the non-batch API, which is quite convenient for locating code that needs to be re-written to use IIndexProcedures.
protected static String ERR_ABORT_TX
          Error message used if we were unable to abort a transaction that we started in order to provide read-consistent semantics for an ITx.READ_COMMITTED view or for a read-only operation on an ITx.UNISOLATED view.
protected static String ERR_NEW_TX
          Error message used if we were unable to start a new transaction in order to provide read-consistent semantics for an ITx.READ_COMMITTED view or for a read-only operation on an ITx.UNISOLATED view.
protected  AbstractScaleOutFederation fed
           
protected static org.apache.log4j.Logger log
          Note: Invocations of the non-batch API are logged at the WARN level since they result in an application that can not scale-out efficiently.
protected  String name
          The name of the scale-out index (from the ctor).
protected static String NON_BATCH_API
           
protected  boolean readConsistent
          true iff globally consistent read operations are desired for iterators or index procedures mapped across more than one index partition.
protected  long taskTimeout
          The timeout in milliseconds for tasks run on an IDataService.
protected  long timestamp
          The timestamp from the ctor.
protected  boolean WARN
          True iff the log level is WARN or less.
 
Fields inherited from interface com.bigdata.btree.IRangeQuery
ALL, CURSOR, DEFAULT, DELETED, FIXED_LENGTH_SUCCESSOR, KEYS, NONE, PARALLEL, READONLY, REMOVEALL, REVERSE, VALS
 
Constructor Summary
AbstractScaleOutClientIndexView(AbstractScaleOutFederation fed, String name, long timestamp, IMetadataIndex metadataIndex)
          Create a view on a scale-out index.
 
Method Summary
 boolean contains(byte[] key)
          Return true iff there is a (non-deleted) index entry for the key.
 boolean contains(Object key)
          Return true iff there is an entry for the key.
 ICounter getCounter()
          Counters are local to a specific index partition and are only available to unisolated procedures running inside of an IConcurrencyManager (which includes procedures run on an IDataService).
 ICounterSet getCounters()
          Return a new CounterSet backed by the ScaleOutIndexCounters for this scale-out index.
 IDataService getDataService(PartitionLocator pmd)
          Resolve the data service to which the index partition is mapped.
 AbstractScaleOutFederation getFederation()
          Return the object used to access the services in the connected federation.
 IndexMetadata getIndexMetadata()
          The metadata for the managed scale-out index.
protected  IMetadataIndex getMetadataIndex()
          Return a view of the metadata index for the scale-out index as of the timestamp associated with this index view.
 MetadataIndex.MetadataIndexMetadata getMetadataIndexMetadata()
          Metadata for the MetadataIndex that manages the scale-out index (cached).
protected  IMetadataService getMetadataService()
          Obtain the proxy for a metadata service.
 String getName()
          The name of the scale-out index.
 IResourceMetadata[] getResourceMetadata()
          This operation is not supported - the resource description of a scale-out index would include all "live" resources in the corresponding MetadataIndex.
protected  ThreadPoolExecutor getThreadPool()
          The thread pool exposed by IBigdataFederation.getExecutorService()
 long getTimestamp()
          Either the startTime of an active transaction, ITx.UNISOLATED for the current unisolated index view, ITx.READ_COMMITTED for a read-committed view, or the timestamp for a historical view no later than the specified timestamp.
protected  ITupleSerializer getTupleSerializer()
           
 byte[] insert(byte[] key, byte[] value)
          Insert or update a value under the key.
 Object insert(Object key, Object val)
          Insert with auto-magic handling of keys and value objects.
 Iterator<PartitionLocator> locatorScan(long ts, byte[] fromKey, byte[] toKey, boolean reverseScan)
          Returns an iterator that will visit the PartitionLocators for the specified scale-out index key range.
 byte[] lookup(byte[] key)
          Lookup a value for a key.
 Object lookup(Object key)
          Lookup a value for a key.
<T extends IKeyArrayIndexProcedure,O,R,A>
BlockingBuffer<KVO<O>[]>
newWriteBuffer(IResultHandler<R,A> resultHandler, IDuplicateRemover<O> duplicateRemover, AbstractKeyArrayIndexProcedureConstructor<T> ctor)
          Asynchronous write API (streaming writes).
 long rangeCount()
          Return the #of tuples in the index.
 long rangeCount(byte[] fromKey, byte[] toKey)
          Returns the sum of the range count for each index partition spanned by the key range.
 long rangeCountExact(byte[] fromKey, byte[] toKey)
          The exact range count is obtained by mapping a key-range scan over the index partitions.
 long rangeCountExactWithDeleted(byte[] fromKey, byte[] toKey)
          The exact range count of deleted and undeleted tuples is obtained by mapping a key-range scan over the index partitions.
 ITupleIterator rangeIterator()
          Visits all tuples in key order.
 ITupleIterator rangeIterator(byte[] fromKey, byte[] toKey)
          An ITupleIterator that kinds the use of a series of ResultSets to cover all index partitions spanned by the key range.
 ITupleIterator rangeIterator(byte[] fromKey, byte[] toKey, int capacity, int flags, IFilterConstructor filter)
          Identifies the index partition(s) that are spanned by the key range query and maps an iterator across each index partition.
 byte[] remove(byte[] key)
          Remove the key and its associated value.
 Object remove(Object key)
          Remove the key and its associated value.
 LinkedList<Split> splitKeys(long ts, int fromIndex, int toIndex, byte[][] keys)
          Utility method to split a set of ordered keys into partitions based the index partitions defined for a scale-out index.
 LinkedList<Split> splitKeys(long ts, int fromIndex, int toIndex, KVO[] a)
          Identify the Splits for an ordered KVO[] such that there is one Split per index partition spanned by the data.
 void staleLocator(long ts, PartitionLocator locator, StaleLocatorException cause)
          Notifies the client that a StaleLocatorException was received.
 void submit(byte[] fromKey, byte[] toKey, IKeyRangeIndexProcedure proc, IResultHandler resultHandler)
          Maps an IIndexProcedure across a key range by breaking it down into one task per index partition spanned by that key range.
 Object submit(byte[] key, ISimpleIndexProcedure proc)
          Submits an index procedure that operations on a single key to the appropriate index partition returning the result of that procedure.
 void submit(int fromIndex, int toIndex, byte[][] keys, byte[][] vals, AbstractKeyArrayIndexProcedureConstructor ctor, IResultHandler aggregator)
          The procedure will be transparently broken down and executed against each index partitions spanned by its keys.
protected abstract  void submit(long ts, byte[] fromKey, byte[] toKey, IKeyRangeIndexProcedure proc, IResultHandler resultHandler)
          Variant uses the caller's timestamp.
protected abstract  Object submit(long ts, byte[] key, ISimpleIndexProcedure proc)
          Variant uses the caller's timestamp.
protected abstract  void submit(long ts, int fromIndex, int toIndex, byte[][] keys, byte[][] vals, AbstractKeyArrayIndexProcedureConstructor ctor, IResultHandler aggregator)
          Variant uses the caller's timestamp.
 String toString()
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 
Methods inherited from interface com.bigdata.service.ndx.IScaleOutClientIndex
getRecursionDepth
 

Field Detail

log

protected static final transient org.apache.log4j.Logger log
Note: Invocations of the non-batch API are logged at the WARN level since they result in an application that can not scale-out efficiently.


WARN

protected final boolean WARN
True iff the log level is WARN or less.


ERR_NEW_TX

protected static final transient String ERR_NEW_TX
Error message used if we were unable to start a new transaction in order to provide read-consistent semantics for an ITx.READ_COMMITTED view or for a read-only operation on an ITx.UNISOLATED view.

See Also:
Constant Field Values

ERR_ABORT_TX

protected static final transient String ERR_ABORT_TX
Error message used if we were unable to abort a transaction that we started in order to provide read-consistent semantics for an ITx.READ_COMMITTED view or for a read-only operation on an ITx.UNISOLATED view.

See Also:
Constant Field Values

fed

protected final AbstractScaleOutFederation fed

taskTimeout

protected final long taskTimeout
The timeout in milliseconds for tasks run on an IDataService.

See Also:
IBigdataClient.Options.CLIENT_TASK_TIMEOUT

NON_BATCH_API

protected static final String NON_BATCH_API
See Also:
Constant Field Values

batchOnly

protected final boolean batchOnly
This may be used to disable the non-batch API, which is quite convenient for locating code that needs to be re-written to use IIndexProcedures.


timestamp

protected final long timestamp
The timestamp from the ctor.


name

protected final String name
The name of the scale-out index (from the ctor).


readConsistent

protected final boolean readConsistent
true iff globally consistent read operations are desired for iterators or index procedures mapped across more than one index partition. When true and the index is ITx.READ_COMMITTED or (if the index is ITx.UNISOLATED and the operation is read-only), IIndexStore.getLastCommitTime() is queried at the start of the operation and used as the timestamp for all requests made in support of that operation.

Note that StaleLocatorExceptions can not arise for read-consistent operations. Such operations use a read-consistent view of the IMetadataIndex and the locators therefore will not change during the operation.

See Also:
Constant Field Values
TODO:
make this a ctor argument or settable property?
Constructor Detail

AbstractScaleOutClientIndexView

public AbstractScaleOutClientIndexView(AbstractScaleOutFederation fed,
                                       String name,
                                       long timestamp,
                                       IMetadataIndex metadataIndex)
Create a view on a scale-out index.

Parameters:
fed - The federation containing the index.
name - The index name.
timestamp - A transaction identifier, ITx.UNISOLATED for the unisolated index view, ITx.READ_COMMITTED, or timestamp for a historical view no later than the specified timestamp.
metadataIndex - The IMetadataIndex for the named scale-out index as of that timestamp. Note that the IndexMetadata on this object contains the template IndexMetadata for the scale-out index partitions.
Method Detail

getFederation

public AbstractScaleOutFederation getFederation()
Description copied from interface: IScaleOutClientIndex
Return the object used to access the services in the connected federation.

Specified by:
getFederation in interface IScaleOutClientIndex

getThreadPool

protected ThreadPoolExecutor getThreadPool()
The thread pool exposed by IBigdataFederation.getExecutorService()


getTimestamp

public final long getTimestamp()
Description copied from interface: IClientIndex
Either the startTime of an active transaction, ITx.UNISOLATED for the current unisolated index view, ITx.READ_COMMITTED for a read-committed view, or the timestamp for a historical view no later than the specified timestamp.

Specified by:
getTimestamp in interface IClientIndex

getName

public final String getName()
Description copied from interface: IClientIndex
The name of the scale-out index.

Specified by:
getName in interface IClientIndex

getMetadataService

protected final IMetadataService getMetadataService()
Obtain the proxy for a metadata service. if this instance fails, then we can always ask for a new instance for the same federation (failover).


getMetadataIndex

protected final IMetadataIndex getMetadataIndex()
Return a view of the metadata index for the scale-out index as of the timestamp associated with this index view.

See Also:
IBigdataFederation.getMetadataIndex(String, long)
TODO:
This is a bit dangerous since most of the time when you want the metadata index you may have a timestamp in effect which is different from the timestamp of the view (e.g., a read-consistent transaction).

toString

public String toString()
Overrides:
toString in class Object

getMetadataIndexMetadata

public MetadataIndex.MetadataIndexMetadata getMetadataIndexMetadata()
Metadata for the MetadataIndex that manages the scale-out index (cached).


getIndexMetadata

public IndexMetadata getIndexMetadata()
The metadata for the managed scale-out index. Among other things, this gets used to determine how we serialize keys and values for IKeyArrayIndexProcedures when we serialize a procedure to be sent to a remote IDataService.

Specified by:
getIndexMetadata in interface IIndex

getDataService

public IDataService getDataService(PartitionLocator pmd)
Description copied from interface: IScaleOutClientIndex
Resolve the data service to which the index partition is mapped.

Specified by:
getDataService in interface IScaleOutClientIndex
Parameters:
pmd - The index partition locator.
Returns:
The data service and never null.

locatorScan

public Iterator<PartitionLocator> locatorScan(long ts,
                                              byte[] fromKey,
                                              byte[] toKey,
                                              boolean reverseScan)
Description copied from interface: IScaleOutClientIndex
Returns an iterator that will visit the PartitionLocators for the specified scale-out index key range.

Specified by:
locatorScan in interface IScaleOutClientIndex
Parameters:
ts - The timestamp that will be used to visit the locators.
fromKey - The scale-out index first key that will be visited (inclusive). When null there is no lower bound.
toKey - The first scale-out index key that will NOT be visited (exclusive). When null there is no upper bound.
reverseScan - true if you need to visit the index partitions in reverse key order (this is done when the partitioned iterator is scanning backwards).
Returns:
The iterator. The value returned by ITuple.getValue() will be a serialized PartitionLocator object.
See Also:
AbstractScaleOutFederation.locatorScan(String, long, byte[], byte[], boolean)

getResourceMetadata

public IResourceMetadata[] getResourceMetadata()
This operation is not supported - the resource description of a scale-out index would include all "live" resources in the corresponding MetadataIndex.

Specified by:
getResourceMetadata in interface IIndex

getCounter

public ICounter getCounter()
Description copied from interface: IClientIndex
Counters are local to a specific index partition and are only available to unisolated procedures running inside of an IConcurrencyManager (which includes procedures run on an IDataService).

Specified by:
getCounter in interface IIndex
Specified by:
getCounter in interface IClientIndex

getTupleSerializer

protected ITupleSerializer getTupleSerializer()

contains

public boolean contains(Object key)
Description copied from interface: IAutoboxBTree
Return true iff there is an entry for the key.

Specified by:
contains in interface IAutoboxBTree
Parameters:
key - The key is implicitly converted to an unsigned byte[].
Returns:
True if the btree contains an entry for that key.

contains

public boolean contains(byte[] key)
Description copied from interface: ISimpleBTree
Return true iff there is a (non-deleted) index entry for the key. An index entry with a null value will cause this method to return true. A deleted index entry will cause this method to return false.

Specified by:
contains in interface ISimpleBTree
Parameters:
key - The key.
Returns:
true if the index contains an (un-deleted) entry for that key.

insert

public Object insert(Object key,
                     Object val)
Description copied from interface: IAutoboxBTree
Insert with auto-magic handling of keys and value objects.

Specified by:
insert in interface IAutoboxBTree
Parameters:
key - The key is implicitly converted to an unsigned byte[].
val - The value is implicitly converted to a byte[].
Returns:
The de-serialized old value -or- null if there was no value stored under that key.

insert

public byte[] insert(byte[] key,
                     byte[] value)
Description copied from interface: ISimpleBTree
Insert or update a value under the key.

Specified by:
insert in interface ISimpleBTree
Parameters:
key - The key.
value - The value (may be null).
Returns:
The previous value under that key or null if the key was not found or if the previous entry for that key was marked as deleted.

lookup

public Object lookup(Object key)
Description copied from interface: IAutoboxBTree
Lookup a value for a key.

Specified by:
lookup in interface IAutoboxBTree
Parameters:
key - The key is implicitly converted to an unsigned byte[].
Returns:
The de-serialized value or null if there is no entry for that key.

lookup

public byte[] lookup(byte[] key)
Description copied from interface: ISimpleBTree
Lookup a value for a key.

Specified by:
lookup in interface ISimpleBTree
Returns:
The value stored under that key or null if there is no entry for that key or if the entry under that key is marked as deleted.

remove

public Object remove(Object key)
Description copied from interface: IAutoboxBTree
Remove the key and its associated value.

Specified by:
remove in interface IAutoboxBTree
Parameters:
key - The key is implicitly converted to an unsigned byte[].
Returns:
The de-serialized value stored under that key or null if the key was not found.

remove

public byte[] remove(byte[] key)
Description copied from interface: ISimpleBTree
Remove the key and its associated value.

Specified by:
remove in interface ISimpleBTree
Parameters:
key - The key.
Returns:
The value stored under that key or null if the key was not found or if the previous entry under that key was marked as deleted.

rangeCount

public long rangeCount()
Description copied from interface: IRangeQuery
Return the #of tuples in the index.

Note: If the index supports deletion markers then the range count will be an upper bound and may double count tuples which have been overwritten, including the special case where the overwrite is a delete.

Specified by:
rangeCount in interface IRangeQuery
Returns:
The #of tuples in the index.

rangeCount

public long rangeCount(byte[] fromKey,
                       byte[] toKey)
Returns the sum of the range count for each index partition spanned by the key range.

Specified by:
rangeCount in interface IRangeQuery
Parameters:
fromKey - The lowest key that will be counted (inclusive). When null there is no lower bound.
toKey - The first key that will not be counted (exclusive). When null there is no upper bound.
Returns:
The #of tuples in the half-open key range.

rangeCountExact

public final long rangeCountExact(byte[] fromKey,
                                  byte[] toKey)
The exact range count is obtained by mapping a key-range scan over the index partitions. The operation is parallelized.

Specified by:
rangeCountExact in interface IRangeQuery
Parameters:
fromKey - The lowest key that will be counted (inclusive). When null there is no lower bound.
toKey - The first key that will not be counted (exclusive). When null there is no upper bound.
Returns:
The exact #of tuples in the half-open key range.

rangeCountExactWithDeleted

public final long rangeCountExactWithDeleted(byte[] fromKey,
                                             byte[] toKey)
The exact range count of deleted and undeleted tuples is obtained by mapping a key-range scan over the index partitions. The operation is parallelized.

Specified by:
rangeCountExactWithDeleted in interface IRangeQuery
Parameters:
fromKey - The lowest key that will be counted (inclusive). When null there is no lower bound.
toKey - The first key that will not be counted (exclusive). When null there is no upper bound.
Returns:
The exact #of deleted and undeleted tuples in the half-open key range.
See Also:
IRangeQuery.rangeCountExact(byte[], byte[])

rangeIterator

public final ITupleIterator rangeIterator()
Description copied from interface: IRangeQuery
Visits all tuples in key order. This is identical to
 rangeIterator(null, null)
 

Specified by:
rangeIterator in interface IRangeQuery
Returns:
An iterator that will visit all entries in key order.

rangeIterator

public ITupleIterator rangeIterator(byte[] fromKey,
                                    byte[] toKey)
An ITupleIterator that kinds the use of a series of ResultSets to cover all index partitions spanned by the key range.

Specified by:
rangeIterator in interface IRangeQuery
Parameters:
fromKey - The first key that will be visited (inclusive lower bound). When null there is no lower bound.
toKey - The first key that will NOT be visited (exclusive upper bound). When null there is no upper bound.
See Also:
SuccessorUtil, which may be used to compute the successor of a value before encoding it as a component of a key., BytesUtil#successor(byte[]), which may be used to compute the successor of an encoded key., EntryFilter, which may be used to filter the entries visited by the iterator.

rangeIterator

public ITupleIterator rangeIterator(byte[] fromKey,
                                    byte[] toKey,
                                    int capacity,
                                    int flags,
                                    IFilterConstructor filter)
Identifies the index partition(s) that are spanned by the key range query and maps an iterator across each index partition. The iterator buffers responses up to the specified capacity and a follow up iterator request is automatically issued if the iterator has not exhausted the key range on a given index partition. Once the iterator is exhausted on a given index partition it is then applied to the next index partition spanned by the key range.

Specified by:
rangeIterator in interface IRangeQuery
Parameters:
fromKey - The first key that will be visited (inclusive lower bound). When null there is no lower bound.
toKey - The first key that will NOT be visited (exclusive upper bound). When null there is no upper bound.
capacity - The #of entries to buffer at a time. This is a hint and MAY be zero (0) to use an implementation specific default capacity. A non-zero value may be used if you know that you want at most N results or if you want to override the default #of results to be buffered before sending them across a network interface. (Note that you can control the default value using IBigdataClient.Options#DEFAULT_CLIENT_RANGE_QUERY_CAPACITY).
flags - A bitwise OR of IRangeQuery.KEYS, IRangeQuery.VALS, etc.
filter - An optional object used to construct a stacked iterator. When IRangeQuery.CURSOR is specified in flags, the base iterator will implement ITupleCursor and the first filter in the stack can safely cast the source iterator to an ITupleCursor. If the outermost filter in the stack does not implement ITupleIterator, then it will be wrapped an ITupleIterator.
See Also:
SuccessorUtil, which may be used to compute the successor of a value before encoding it as a component of a key., BytesUtil#successor(byte[]), which may be used to compute the successor of an encoded key., IFilterConstructor, which may be used to construct an iterator stack performing filtering or other operations.
TODO:
If the return iterator implements ITupleCursor then this will need be modified to defer request of the initial result set until the caller uses first(), last(), seek(), hasNext(), or hasPrior().

splitKeys

public LinkedList<Split> splitKeys(long ts,
                                   int fromIndex,
                                   int toIndex,
                                   byte[][] keys)
Utility method to split a set of ordered keys into partitions based the index partitions defined for a scale-out index.

Find the partition for the first key. Check the last key, if it is in the same partition then then this is the simplest case and we can just send the data along.

Otherwise, perform a binary search on the remaining keys looking for the index of the first key GTE the right separator key for that partition. The batch for this partition is formed from all keys from the first key for that partition up to but excluding the index position identified by the binary search (if there is a match; if there is a miss, then the binary search result needs to be converted into a key index and that will be the last key for the current partition).

Examine the next key and repeat the process until all keys have been allocated to index partitions.

Note: Split points MUST respect the "row" identity for a sparse row store, but we get that constraint by maintaining the index partition boundaries in agreement with the split point constraints for the index.

Specified by:
splitKeys in interface ISplitter
Parameters:
ts - The timestamp for the IMetadataIndex view that will be applied to choose the Splits.
fromIndex - The index of the first key in keys to be processed (inclusive).
toIndex - The index of the last key in keys to be processed.
keys - An array of keys. Each key is an interpreted as an unsigned byte[]. All keys must be non-null. The keys must be in sorted order.
Returns:
The Splits that you can use to form requests based on the identified first/last key and partition identified by this process.
See Also:
Arrays.sort(Object[], int, int, java.util.Comparator), BytesUtil.compareBytes(byte[], byte[])
TODO:
Caching? This procedure performs the minimum #of lookups using IMetadataIndex.find(byte[]) since that operation will be an RMI in a distributed federation. The find(byte[] key) operation is difficult to cache since it locates the index partition that would span the key and many, many different keys could fit into that same index partition. The only effective cache technique may be an LRU that scans ~10 caches locators to see if any of them is a match before reaching out to the remote IMetadataService. Or perhaps the locators can be cached in a local BTree and a miss there would result in a read through to the remote IMetadataService but then we have the problem of figuring out when to release locators if the client is long-lived.

splitKeys

public LinkedList<Split> splitKeys(long ts,
                                   int fromIndex,
                                   int toIndex,
                                   KVO[] a)
Description copied from interface: ISplitter
Identify the Splits for an ordered KVO[] such that there is one Split per index partition spanned by the data.

Specified by:
splitKeys in interface ISplitter
Parameters:
ts - The timestamp for the IMetadataIndex view that will be applied to choose the Splits.
fromIndex - The index of the first key in keys to be processed (inclusive).
toIndex - The index of the last key in keys to be processed.
Returns:
The Splits that you can use to form requests based on the identified first/last key and partition identified by this process.

staleLocator

public void staleLocator(long ts,
                         PartitionLocator locator,
                         StaleLocatorException cause)
Description copied from interface: IScaleOutClientIndex
Notifies the client that a StaleLocatorException was received. The client will use this information to refresh the IMetadataIndex.

Specified by:
staleLocator in interface IScaleOutClientIndex
Parameters:
ts - The timestamp of the metadata index view from which the locator was obtained.
locator - The locator that was stale.
cause - The reason why the locator became stale (split, join, or move).

submit

public Object submit(byte[] key,
                     ISimpleIndexProcedure proc)
Description copied from interface: IIndex
Submits an index procedure that operations on a single key to the appropriate index partition returning the result of that procedure.

Specified by:
submit in interface IIndex
Parameters:
key - The key.
proc - The procedure.
Returns:
The value returned by IIndexProcedure.apply(IIndex)

submit

public void submit(byte[] fromKey,
                   byte[] toKey,
                   IKeyRangeIndexProcedure proc,
                   IResultHandler resultHandler)
Maps an IIndexProcedure across a key range by breaking it down into one task per index partition spanned by that key range.

Note: In order to avoid growing the task execution queue without bound, an upper bound of IBigdataClient.Options.CLIENT_MAX_PARALLEL_TASKS_PER_REQUEST tasks will be placed onto the queue at a time. More tasks will be submitted once those tasks finish until all tasks have been executed. When the task is not parallelizable the tasks will be submitted to the corresponding index partitions at a time and in key order.

Specified by:
submit in interface IIndex
Parameters:
fromKey - The lower bound (inclusive) -or- null if there is no lower bound.
toKey - The upper bound (exclusive) -or- null if there is no upper bound.
proc - The procedure. If the procedure implements the IParallelizableIndexProcedure marker interface then it MAY be executed in parallel against the relevant index partition(s).

submit

public void submit(int fromIndex,
                   int toIndex,
                   byte[][] keys,
                   byte[][] vals,
                   AbstractKeyArrayIndexProcedureConstructor ctor,
                   IResultHandler aggregator)
The procedure will be transparently broken down and executed against each index partitions spanned by its keys. If the ctor creates instances of IParallelizableIndexProcedure then the procedure will be mapped in parallel against the relevant index partitions.

Note: Unlike mapping an index procedure across a key range, this method is unable to introduce a truely enourmous burden on the client's task queue since the #of tasks arising is equal to the #of splits and bounded by n := toIndex - fromIndex.

Specified by:
submit in interface IIndex
Parameters:
fromIndex - The index of the first key to be used (inclusive).
toIndex - The index of the last key to be used (exclusive).
keys - The keys (required).
vals - The values (optional depending on the procedure).
ctor - An object that can create instances of the procedure.
aggregator - When defined, results from each procedure application will be reported to this object.

submit

protected abstract Object submit(long ts,
                                 byte[] key,
                                 ISimpleIndexProcedure proc)
Variant uses the caller's timestamp.

Parameters:
ts -
key -
proc -
Returns:

submit

protected abstract void submit(long ts,
                               byte[] fromKey,
                               byte[] toKey,
                               IKeyRangeIndexProcedure proc,
                               IResultHandler resultHandler)
Variant uses the caller's timestamp.

Parameters:
ts -
fromKey -
toKey -
proc -
resultHandler -

submit

protected abstract void submit(long ts,
                               int fromIndex,
                               int toIndex,
                               byte[][] keys,
                               byte[][] vals,
                               AbstractKeyArrayIndexProcedureConstructor ctor,
                               IResultHandler aggregator)
Variant uses the caller's timestamp.

Parameters:
ts -
fromIndex -
toIndex -
keys -
vals -
ctor -
aggregator -

newWriteBuffer

public <T extends IKeyArrayIndexProcedure,O,R,A> BlockingBuffer<KVO<O>[]> newWriteBuffer(IResultHandler<R,A> resultHandler,
                                                                                         IDuplicateRemover<O> duplicateRemover,
                                                                                         AbstractKeyArrayIndexProcedureConstructor<T> ctor)
Description copied from interface: IAsynchronousWriteBufferFactory
Asynchronous write API (streaming writes).

The returned buffer provides a streaming API which is highly efficient. The caller writes ordered KVO[] chunks onto the thread-safe BlockingBuffer. Those chunks are dynamically combined and then split into per-index partition chunks which are written on internally managed BlockingBuffers for each index partition which will be touched by a write operation. The splits are slices of ordered chunks for a specific index partition. The BlockingBuffer uses a merge sort when it combines ordered chunks so that the combined chunks remain fully ordered. Once a chunk is ready, it is re-shaped for the CTOR and sent to the target data service using RMI.

Since this API is asynchronous, you will not have synchronous access to values returned by asynchronous writes. However, patterns can be created using KVOC and KVOLatch which provide notification when application defined sets of results have become available. Such patterns are created by associated the KVOLatch with the set of results and using IResultHandler and the object reference on the KVOC to capture the side-effect of the write.

BlockingBuffer.getFuture() may be used to obtain the Future of the consumer. You can use Future.get() to await the completion of the consumer, to cancel the consumer, etc. The Future will not terminate (other than by error) until the buffer has been closed. The Future evaluates to an IndexAsyncWriteStats object. Those statistics are also reported to the ILoadBalancerService via the IBigdataFederation.

Each buffer returned by this method is independent, and writes onto independent sinks which write through to the index partitions. This is necessary in order for the caller to retain control over the life cycle of their write operations. The BlockingBuffer is thread-safe so it may be the target for concurrent producers can be can utilized to create very high throughput designs. While the returned buffers are independent, the performance counters for all asynchronous write buffers for a given client and scale-out index are aggregated by a single ScaleOutIndexCounters instance.

Specified by:
newWriteBuffer in interface IAsynchronousWriteBufferFactory
Type Parameters:
T - The generic type of the procedure used to write on the index.
O - The generic type for unserialized value objects.
R - The type of the result from applying the index procedure to a single Split of data.
A - The type of the aggregated result.
Parameters:
resultHandler - Used to aggregate results.
duplicateRemover - Used to filter out duplicates in an application specified manner (optional).
ctor - Used to create instances of the procedure that will execute a write on an individual index partition (this implies that insert and remove operations as well as custom index write operations must use separate buffers).
Returns:
A buffer on which the producer may write their data.
See Also:
IndexMetadata.getAsynchronousIndexWriteConfiguration(), AbstractFederation.getIndexCounters(String)

getCounters

public ICounterSet getCounters()
Return a new CounterSet backed by the ScaleOutIndexCounters for this scale-out index.

Specified by:
getCounters in interface IIndex


Copyright © 2006-2009 SYSTAP, LLC. All Rights Reserved.