com.bigdata.resources
Class IndexManager

java.lang.Object
  com.bigdata.resources.ResourceEvents
      com.bigdata.resources.StoreManager
          com.bigdata.resources.IndexManager

All Implemented Interfaces:

IResourceManager, IServiceShutdown

Direct Known Subclasses:

OverflowManager

public abstract class IndexManager
extends StoreManager

Class encapsulates logic and handshaking for tracking which indices (and their backing stores) are recently and currently referenced. This information is used to coordinate the close out of index resources (and their backing stores) on an LRU basis by the ResourceManager.

Version:

$Id: IndexManager.java 5894 2012-01-27 13:44:53Z thompsonbry $

Author:

Bryan Thompson

Nested Class Summary

static interface	IndexManager.IIndexManagerCounters Performance counters for the IndexManager.
class	IndexManager.IndexCache Extends the ConcurrentWeakValueCache to track the earliest timestamp from which any local IIndex view is reading.
static class	IndexManager.IndexSegmentStats Statistics about the IndexSegments open in the cache.
static interface	IndexManager.Options Options understood by the IndexManager.

Nested classes/interfaces inherited from class com.bigdata.resources.StoreManager

StoreManager.IStoreManagerCounters, StoreManager.ManagedJournal

Field Summary

protected ConcurrentHashMap<File,IndexSegmentBuilder>	buildTasks A map containing the concurrently executing index segment build tasks.
protected AtomicInteger	concurrentBuildTaskCount The #of build tasks which are executing concurrently.
protected AtomicInteger	concurrentMergeTaskCount The #of merge tasks which are executing concurrently.
protected static org.apache.log4j.Logger	log Logger.
protected LRUCache<String,StaleLocatorReason>	staleLocatorCache This cache is used to provide remote clients with an unambiguous indication that an index partition has been rather than simply not existing or having been dropped.

Fields inherited from class com.bigdata.resources.StoreManager

accelerateOverflowThreshold, bytesDeleted, bytesUnderManagement, dataDir, indexCacheLock, journalBytesUnderManagement, journalDeleteCount, journalReopenCount, journalsDir, lastCommitTimePreserved, lastOverflowTime, liveJournalRef, maximumJournalSizeAtOverflow, purgeResourcesMillis, segmentBytesUnderManagement, segmentsDir, segmentStoreDeleteCount, segmentStoreReopenCount, storeCache, tmpDir

Constructor Summary

protected

IndexManager(Properties properties)

Method Summary

BuildResult	buildIndexSegment(String indexPartitionName, ILocalBTreeView src, boolean compactingMerge, long commitTime, byte[] fromKey, byte[] toKey, Event parentEvent) Build an IndexSegment from an index partition.
void	disableWrites(String name) Declare that the named index will no longer accept writes (transient effect only).
void	enableWrites(String name) Declare that the named index will accept writes (default).
ILocalBTreeView	getIndex(String name, long timestamp) Note: An ITx.READ_COMMITTED view returned by this method WILL NOT update if there are intervening commits.
int	getIndexCacheCapacity() The configured capacity of the index cache.
int	getIndexCacheSize() The #of entries in the hard reference cache for IIndexs.
CounterSet	getIndexCounters() Return a CounterSet reflecting use of the named indices.
BTreeCounters	getIndexCounters(String name) Return the BTreeCounters for the named index.
AbstractBTree	getIndexOnStore(String name, long timestamp, IRawStore store) Return a reference to the named index as of the specified timestamp on the identified resource.
StaleLocatorReason	getIndexPartitionGone(String name) Note: this information is based on an LRU cache with a large fixed capacity.
protected long	getIndexRetentionTime() The earliest timestamp that MUST be retained for the read-historical indices in the cache and Long.MAX_VALUE if there are NO read-historical indices in the cache.
int	getIndexSegmentCacheCapacity() The configured capacity of the index segment cache.
int	getIndexSegmentCacheSize() The #of entries in the hard reference cache for IndexSegments.
AbstractBTree[]	getIndexSources(String name, long timestamp) Return the ordered AbstractBTree sources for an index or a view of an index partition.
AbstractBTree[]	getIndexSources(String name, long timestamp, BTree btree) Examine the partition metadata (if any) for the BTree.
protected int	getStaleLocatorCount() The #of entries in the stale locator LRU.
boolean	isDisabledWrites(String name) Return true if writes have been disabled for the named index.
String	listIndexPartitions(long timestamp) Dump index metadata as of the timestamp.
protected Map<String,BTreeCounters>	markAndGetDelta() Snapshots the index partition performance counters and returns a map containing the net change in the performance counters for each index partition since the last time this method was invoked (it is invoked by IResourceManager.overflow()).
protected void	setIndexPartitionGone(String name, StaleLocatorReason reason) Notify the ResourceManager that the named index partition was split, joined or moved.

Methods inherited from class com.bigdata.resources.StoreManager

addResource, assertNotOpen, assertOpen, assertRunning, awaitRunning, deleteResource, deleteResources, getBytesUnderManagement, getCommitTimeStrictlyGreaterThan, getConcurrencyManager, getDataDir, getDataDirFreeSpace, getIndexSegmentFile, getIndexSegmentFile, getJournal, getJournalBytesUnderManagement, getLiveJournal, getManagedJournalCount, getManagedSegmentCount, getProperties, getReleaseTime, getResourceService, getResourcesForTimestamp, getSegmentBytesUnderManagement, getStoreCacheSize, getStoreCounters, getTempDirFreeSpace, getTmpDir, isOpen, isRunning, isStarting, isTransient, munge, newFileFilter, nextTimestamp, openStore, overrideJournalExtent, purgeOldResources, purgeOldResources, retentionSetAdd, retentionSetRemove, setConcurrencyManager, setReleaseTime, shutdown, shutdownNow

Methods inherited from class com.bigdata.resources.ResourceEvents

closeJournal, closeTx, closeUnisolatedBTree, deleteJournal, dropUnisolatedBTree, extendJournal, isolateIndex, openJournal, openTx, openUnisolatedBTree

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.bigdata.journal.IResourceManager

getCounters, getDataService, getDataServiceUUID, getFederation, isOverflowEnabled, overflow, shouldOverflow

Field Detail

log

protected static final org.apache.log4j.Logger log

Logger.

staleLocatorCache

protected final LRUCache<String,StaleLocatorReason> staleLocatorCache

This cache is used to provide remote clients with an unambiguous indication that an index partition has been rather than simply not existing or having been dropped.

The keys are the name of an index partitions that has been split, joined, or moved. Such index partitions are no longer available and have been replaced by one or more new index partitions (having a distinct partition identifier) either on the same or on another data service. The value is a reason, e.g., "split", "join", or "move".

buildTasks

protected final ConcurrentHashMap<File,IndexSegmentBuilder> buildTasks

A map containing the concurrently executing index segment build tasks. This is used to report those tasks out via the performance counters interface.

concurrentBuildTaskCount

protected final AtomicInteger concurrentBuildTaskCount

The #of build tasks which are executing concurrently.

concurrentMergeTaskCount

protected final AtomicInteger concurrentMergeTaskCount

The #of merge tasks which are executing concurrently.

Constructor Detail

IndexManager

protected IndexManager(Properties properties)

Method Detail

disableWrites

public void disableWrites(String name)

Declare that the named index will no longer accept writes (transient effect only).

enableWrites

public void enableWrites(String name)

Declare that the named index will accept writes (default).

isDisabledWrites

public boolean isDisabledWrites(String name)

Return true if writes have been disabled for the named index.

Parameters:

name - The index name.

Returns:

true if writes are disabled for that index.

getIndexRetentionTime

protected long getIndexRetentionTime()

The earliest timestamp that MUST be retained for the read-historical indices in the cache and Long.MAX_VALUE if there are NO read-historical indices in the cache.

Specified by:

getIndexRetentionTime in class StoreManager

See Also:

StoreManager.indexCacheLock

getIndexCacheSize

public int getIndexCacheSize()

The #of entries in the hard reference cache for IIndexs. There MAY be more IIndexs open than are reported by this method if there are hard references held by the application to those IIndexs. IIndexs that are not fixed by a hard reference will be quickly finalized by the JVM.

getIndexCacheCapacity

public int getIndexCacheCapacity()

The configured capacity of the index cache.

See Also:

IndexManager.Options.INDEX_CACHE_CAPACITY

getIndexSegmentCacheSize

public int getIndexSegmentCacheSize()

The #of entries in the hard reference cache for IndexSegments. There MAY be more IndexSegments open than are reported by this method if there are hard references held by the application to those IndexSegments. IndexSegments that are not fixed by a hard reference will be quickly finalized by the JVM.

getIndexSegmentCacheCapacity

public int getIndexSegmentCacheCapacity()

The configured capacity of the index segment cache.

See Also:

IndexManager.Options.INDEX_SEGMENT_CACHE_CAPACITY

getIndexPartitionGone

public StaleLocatorReason getIndexPartitionGone(String name)

Note: this information is based on an LRU cache with a large fixed capacity. It is expected that the cache size is sufficient to provide good information to clients having queued write tasks. If the index partition split/move/join changes somehow outpace the cache size then the client would see a NoSuchIndexException instead.

Parameters:

name - The name of an index partition.

Returns:

The reason (split, join, or move) -or- null iff the index partition is not known to be gone.

setIndexPartitionGone

protected void setIndexPartitionGone(String name,
                                     StaleLocatorReason reason)

Notify the ResourceManager that the named index partition was split, joined or moved. This effects only the unisolated view of that index partition. Historical views will continue to exist and reside as before.

Parameters:

name - The name of the index partition.

reason - The reason (split, join, or move). FIXME Should also include "deleted" and handle case where a scale-out index is deleted and then re-created so that we don't get the StaleLocatorException after the recreate.

getStaleLocatorCount

protected int getStaleLocatorCount()

The #of entries in the stale locator LRU.

getIndexOnStore

public AbstractBTree getIndexOnStore(String name,
                                     long timestamp,
                                     IRawStore store)

Return a reference to the named index as of the specified timestamp on the identified resource.

Note: AbstractTask handles the load of the ITx.UNISOLATED index from the live journal in such a manner as to provide ACID semantics for add/drop of indices.

Note: The returned index is NOT isolated. Isolation is handled by the Tx.

Parameters:

name - The index name.

timestamp - A transaction identifier, ITx.UNISOLATED for the unisolated index view, ITx.READ_COMMITTED, or timestamp for a historical view no later than the specified timestamp.

store - The store from which the index will be loaded.

Returns:

A reference to the index -or- null if the index was not registered on the resource as of the timestamp or if the store has no data for that timestamp.

TODO:

this might have to be private since we assume that the store is in StoreManager#openStores.

getIndexSources

public AbstractBTree[] getIndexSources(String name,
                                       long timestamp)

Description copied from interface: IResourceManager

Return the ordered AbstractBTree sources for an index or a view of an index partition. The AbstractBTrees are ordered from the most recent to the oldest and together comprise a coherent view of an index partition.

Note: Index sources loaded from a historical timestamp (vs the live unisolated index view) will always be read-only.

Parameters:

name - The name of the index.

timestamp - A transaction identifier, ITx.UNISOLATED for the unisolated index view, ITx.READ_COMMITTED, or timestamp for a historical view no later than the specified timestamp.

Returns:

The sources for the index view or null if the index was not defined as of the timestamp.

See Also:

FusedView

getIndexSources

public AbstractBTree[] getIndexSources(String name,
                                       long timestamp,
                                       BTree btree)

Description copied from interface: IResourceManager

Examine the partition metadata (if any) for the BTree. If the BTree is part of an index partition, then return all of the sources for a view of that index partition (the BTree will be the first element in the array and, if there are no more sources for the index partition, then it will also be the sole element of the array). Otherwise return an array consisting of a single element, which is the BTree.

btree - A BTree.

Returns:

The source(s) for the view associated with that BTree.

getIndex

public ILocalBTreeView getIndex(String name,
                                long timestamp)

Note: An ITx.READ_COMMITTED view returned by this method WILL NOT update if there are intervening commits. This decision was made based on the fact that views are requested from the IndexManager by an AbstractTask running on the ConcurrencyManager. Such tasks, and hence such views, have a relatively short life. However, the Journal implementation of this method is different and will return a ReadCommittedView precisely because objects are directly requested from a Journal by the application and the application can hold onto a read-committed view for an arbitrary length of time. This has the pragmatic effect of allowing us to cache read-committed views in the application and in the IBigdataClient. For the IBigdataClient, the view acquires its read-committed semantics because an IClientIndex generates AbstractTask(s) for each IIndex operation and submits those task(s) to the appropriate IDataService(s) for evaluation. The IDataService will resolve the index using this method, and it will always see the then-current read-committed view and the IClientIndex will appear to have read-committed semantics.

Parameters:

name - The index name.

timestamp - A transaction identifier, ITx.UNISOLATED for the unisolated index view, ITx.READ_COMMITTED, or timestamp for a historical view no later than the specified timestamp.

Returns:

The index or null iff there is no index registered with that name for that timestamp, including if the timestamp is a transaction identifier and the transaction is unknown or not active.

See Also:

Journal.getIndex(String, long)

listIndexPartitions

public String listIndexPartitions(long timestamp)

Dump index metadata as of the timestamp.

Parameters:

timestamp -

Returns:

The dump.

Throws:

IllegalArgumentException - if timestamp is positive (a transaction identifier).

IllegalStateException - if the live journal is closed when this method is invoked.

RuntimeException - if the live journal is closed asynchronously while this method is running.

buildIndexSegment

public BuildResult buildIndexSegment(String indexPartitionName,
                                     ILocalBTreeView src,
                                     boolean compactingMerge,
                                     long commitTime,
                                     byte[] fromKey,
                                     byte[] toKey,
                                     Event parentEvent)
                              throws Exception

Build an IndexSegment from an index partition. Delete markers are propagated to the IndexSegment unless compactingMerge is true.

Note: IndexSegments are registered with the StoreManager by this method but are also placed into a hard reference collection (the retentionSet) in order to prevent their being released before they are put to use by incorporating them into an index partition view. The caller MUST remove the IndexSegment from that hard reference collection once the index has been incorporated into an index partition view or is no longer required (e.g., has been MOVEd). However, the caller MUST NOT remove the IndexSegment from the hard reference collection until after the commit point for the task which incorporates it into the index partition view. In practice, this means that those tasks must be encapsulated with either a post-condition action or wrapped by a caller which provides the necessary after-action in a finally{} clause.

Parameters:

indexPartitionName - The name of the index partition (not the name of the scale-out index).

src - A view of the index partition as of the createTime. This may be a partial view of comprised from only the first N sources in the view, in which case compactingMerge := false.

compactingMerge - When true the caller asserts that src is a FusedView and deleted index entries WILL NOT be included in the generated IndexSegment. Otherwise, it is assumed that the only select component(s) of the index partition view are being exported onto an IndexSegment and deleted index entries will therefore be propagated to the new IndexSegment.

commitTime - The commit time associated with the view from which the IndexSegment is being generated. This value is written into IndexSegmentCheckpoint.commitTime.

fromKey - The lowest key that will be included (inclusive). When null there is no lower bound.

toKey - The first key that will not be included (exclusive). When null there is no upper bound.

Returns:

A BuildResult identifying the new IndexSegment and the source index.

Throws:

Exception - if any errors are encountered then the file (if it exists) will be deleted as a side-effect before returning control to the caller.

See Also:

StoreManager.purgeOldResources(long, boolean)

getIndexCounters

public BTreeCounters getIndexCounters(String name)

Description copied from interface: IResourceManager

Return the BTreeCounters for the named index. If none exist, then a new instance is atomically created and returned to the caller. This facilitates the reuse of the same BTreeCounters instance for all views of the named index.

Parameters:

name - The name of the index.

Returns:

The counters for that index and never null.

markAndGetDelta

protected Map<String,BTreeCounters> markAndGetDelta()

Snapshots the index partition performance counters and returns a map containing the net change in the performance counters for each index partition since the last time this method was invoked (it is invoked by IResourceManager.overflow()).

Note: This method has a side effect of setting a new mark. It SHOULD NOT be used except at overflow since the "mark" is used to determine the net change in the per-index partition performance counters. If used other than at overflow the net change will be under-reported.

Returns:

A map containing the net change in the index partition performance counters for each index partition.

getIndexCounters

public CounterSet getIndexCounters()

Return a CounterSet reflecting use of the named indices. When an index partition is in use, its CounterSet is reported under a path formed from name of the scale-out index and partition identifier.

Returns:

A new CounterSet reflecting the use of the named indices.