com.bigdata.resources
Class StoreManager

java.lang.Object
  com.bigdata.resources.ResourceEvents
      com.bigdata.resources.StoreManager

All Implemented Interfaces:

IResourceManager, IServiceShutdown

Direct Known Subclasses:

IndexManager

public abstract class StoreManager
extends ResourceEvents
implements IResourceManager

Class encapsulates logic for managing the store files (journals and index segments), including the logic to compute the effective release time for the managed resources and to release those resources by deleting them from the file system.

Version:

$Id: StoreManager.java 2265 2009-10-26 12:51:06Z thompsonbry $

Author:

Bryan Thompson

TODO:

There is neither a "CREATE_TEMP_DIR" and "DELETE_ON_CLOSE" does not remove all directories created during setup. One of the consequences is that you have to explicitly clean up after a unit test using a ResourceManager or it will leave its files around., BufferMode.Temporary is not supported (verify whether the Transient mode is supported)., If we approach the limit on free space for the dataDir then we need to shed index partitions to other data services or potentially become more aggressive in releasing old resources. See #getDataDirFreeSpace(File)

Nested Class Summary

static interface	StoreManager.IStoreManagerCounters Performance counters for the StoreManager.
class	StoreManager.ManagedJournal The StoreManager.ManagedJournal provides the backing store used to absorb writes and retain history for the scale-out architecture.
static interface	StoreManager.Options Options for the StoreManager.

Field Summary

protected long	accelerateOverflowThreshold
protected AtomicLong	bytesDeleted The #of bytes that have been deleted since startup.
protected AtomicLong	bytesUnderManagement The #of bytes currently under management EXCEPT those on the live journal.
protected File	dataDir The directory in which the data files reside.
protected ReentrantReadWriteLock	indexCacheLock In order to have atomic semantics and prevent a read-historical operation from starting concurrently that would have access to a view that is being purged, IndexManager.getIndex(String, long) and purgeOldResources() MUST contend for a shared lock.
protected AtomicLong	journalBytesUnderManagement
protected AtomicLong	journalDeleteCount The #of StoreManager.ManagedJournals that have been deleted to date.
protected AtomicLong	journalReopenCount The #of StoreManager.ManagedJournals that have been (re-)opened to date.
protected File	journalsDir Directory containing the journal resources.
protected long	lastCommitTimePreserved The last value computed by #getEffectiveReleaseTime() and ZERO(0) until a value has been calculated.
protected long	lastOverflowTime The last commit time corresponding to the last synchronous overflow event and ZERO (0L) until there has been a synchronous overflow event.
protected AtomicReference<StoreManager.ManagedJournal>	liveJournalRef A atomic hard reference to the live journal.
protected static org.apache.log4j.Logger	log Logger.
protected long	maximumJournalSizeAtOverflow The observed maximum size of a journal (its length in bytes) as measured at each synchronous overflow event.
protected long	purgeResourcesMillis The elapsed #of milliseconds in purgeOldResources()
protected ResourceService	resourceService
protected AtomicLong	segmentBytesUnderManagement
protected File	segmentsDir Directory containing the index segment resources.
protected AtomicLong	segmentStoreDeleteCount The #of IndexSegmentStores that have been deleted to date.
protected AtomicLong	segmentStoreReopenCount The #of IndexSegmentStores that have been (re-)opened to date.
protected ConcurrentWeakValueCacheWithTimeout<UUID,IRawStore>	storeCache A cache that is used by the to automatically close out unused IndexSegmentStores.
protected File	tmpDir The directory in which the temporary files will reside.

Constructor Summary

protected

StoreManager(Properties properties) Note: This constructor starts an asynchronous thread that scans the data directory for journals and index segments and creates the initial journal if no store files are found.

Method Summary

protected void	addResource(IResourceMetadata resourceMetadata, File file) Notify the resource manager of a new resource.
protected void	assertNotOpen()
protected void	assertOpen()
protected void	assertRunning() Succeeds if the StoreManager isOpen() and is NOT isStarting() (the test itself is NOT atomic).
boolean	awaitRunning() Return true iff the StoreManager is running.
protected void	deleteResource(UUID uuid, boolean isJournal) Delete the resource in the file system and remove it from the storeCache and resourceFiles and either journalIndex or segmentIndex as appropriate.
void	deleteResources() Deletes all resources.
long	getBytesUnderManagement() The #of bytes currently under management, including those written on the live journal.
protected long	getCommitTimeStrictlyGreaterThan(long releaseTime) Finds the journal spanning the first ICommitRecord that is strictly greater than the specified timestamp and returns the timestamp of that ICommitRecord.
abstract IConcurrencyManager	getConcurrencyManager() The object used to control access to the index resources.
File	getDataDir() Note: The returned File is in canonical form.
long	getDataDirFreeSpace() The #of bytes of free space remaining on the volume hosting the dataDir.
protected abstract long	getIndexRetentionTime()
File	getIndexSegmentFile(IndexMetadata indexMetadata) Return the file on which a new IndexSegment should be written.
File	getIndexSegmentFile(String scaleOutIndexName, UUID indexUUID, int partitionId) Return the file on which a new IndexSegment should be written.
AbstractJournal	getJournal(long timestamp) Return the reference to the journal which has the most current data for the given timestamp.
long	getJournalBytesUnderManagement() The #of bytes in StoreManager.ManagedJournals, including those written on the live journal.
StoreManager.ManagedJournal	getLiveJournal() The journal on which writes are made.
int	getManagedJournalCount() The #of journals on hand.
int	getManagedSegmentCount() The #of index segments on hand.
Properties	getProperties() An object wrapping the Properties given to the ctor.
long	getReleaseTime() Return the last value set with setReleaseTime(long).
int	getResourceServicePort() The port at which you can connect to the ResourceService.
protected Set<UUID>	getResourcesForTimestamp(long commitTimeToPreserve) Finds all resources used by any registered index as of the commitTimeToPreserve up to and including the lastCommitTime for the live journal.
long	getSegmentBytesUnderManagement() The #of bytes in managed IndexSegmentStores.
int	getStoreCacheSize() The #of entries in the hard reference cache for IRawStores, including both StoreManager.ManagedJournals and IndexSegment}s.
DiskOnlyStrategy.StoreCounters	getStoreCounters() The performance counters for the IBufferStrategy backing the live journal and any historical journals which are concurrently open with the live journal.
long	getTempDirFreeSpace() The #of bytes of free space remaining on the volume hosting the tmpDir.
File	getTmpDir() The directory for temporary files.
boolean	isOpen() false initially and remains false until #start() completes successfully.
boolean	isRunning() Return true iff the StoreManager is open and startup processing has been completed.
boolean	isStarting() true initially and until #start() completes successfully.
boolean	isTransient() Return true iff data can not be made restart-safe.
static String	munge(String s) Munge a name index so that it is suitable for use in a filesystem.
protected ResourceFileFilter	newFileFilter() Returns a filter that is used to recognize files that are managed by this class.
protected long	nextTimestamp() Report the next timestamp assigned by the ITransactionService.
IRawStore	openStore(UUID uuid) Opens an IRawStore.
protected void	overrideJournalExtent(Properties p) When the StoreManager is relatively new (as measured by the #of bytes under management) we discount the journal extent in order to trigger overflow earlier.
protected PurgeResult	purgeOldResources() Identify and delete resources no longer required by the index views from the current releaseTime up to the lastCommitTime.
boolean	purgeOldResources(long timeout, boolean truncateJournal) This attempts to obtain the exclusive lock for the WriteExecutorService.
protected void	retentionSetAdd(UUID uuid) Add an IndexSegment to the set of IndexSegments which have been generated but not yet incorporated into an index partition view and hence we must take special cautions to prevent their release.
protected void	retentionSetRemove(UUID uuid) Remove an IndexSegment from the retentionSet.
abstract void	setConcurrencyManager(IConcurrencyManager concurrencyManager)
void	setReleaseTime(long releaseTime) Updates the releaseTime.
void	shutdown() The service will no longer accept new requests, but existing requests will be processed (sychronous).
void	shutdownNow() The service will no longer accept new requests and will make a best effort attempt to terminate all existing requests and return ASAP.

Methods inherited from class com.bigdata.resources.ResourceEvents

closeJournal, closeTx, closeUnisolatedBTree, deleteJournal, dropUnisolatedBTree, extendJournal, isolateIndex, openJournal, openTx, openUnisolatedBTree

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.bigdata.journal.IResourceManager

getCounters, getDataService, getDataServiceUUID, getFederation, getIndex, getIndexPartitionGone, getIndexSources, getIndexSources, isOverflowEnabled, overflow, shouldOverflow

Field Detail

log

protected static final org.apache.log4j.Logger log

Logger.

dataDir

protected final File dataDir

The directory in which the data files reside.

Note: It is a hard requirement that each resource is located by the path relative to the dataDir.

See Also:

StoreManager.Options.DATA_DIR, IResourceMetadata.getFile()

journalsDir

protected final File journalsDir

Directory containing the journal resources.

segmentsDir

protected final File segmentsDir

Directory containing the index segment resources.

tmpDir

protected final File tmpDir

The directory in which the temporary files will reside.

See Also:

Options.TMP_DIR

storeCache

protected final ConcurrentWeakValueCacheWithTimeout<UUID,IRawStore> storeCache

A cache that is used by the to automatically close out unused IndexSegmentStores. An IndexSegment that is no longer used will have its reference cleared when it is swept by the garbage collector and will automatically release all of its buffers (node and leaf cache, etc). However, at that point the IndexSegmentStore is still open, and it can buffer a significant amount of data in addition to the file handle.

When the weak reference is cleared we know that there are no longer any hard references to the IndexSegment and hence the corresponding IndexSegmentStore should be closed. In fact, we can immediately remove the IndexSegmentStore from the cache of open stores and then close the store. At this point if the store is re-opened it will be a new object. This is easy enough to do since the UUID of the IndexSegmentStore is the key in our map!

See Also:

StoreManager.Options.STORE_CACHE_CAPACITY, StoreManager.Options.STORE_CACHE_TIMEOUT

liveJournalRef

protected final AtomicReference<StoreManager.ManagedJournal> liveJournalRef

A atomic hard reference to the live journal.

resourceService

protected ResourceService resourceService

accelerateOverflowThreshold

protected final long accelerateOverflowThreshold

See Also:

StoreManager.Options.ACCELERATE_OVERFLOW_THRESHOLD

purgeResourcesMillis

protected long purgeResourcesMillis

The elapsed #of milliseconds in purgeOldResources()

lastCommitTimePreserved

protected long lastCommitTimePreserved

The last value computed by #getEffectiveReleaseTime() and ZERO(0) until a value has been calculated.

lastOverflowTime

protected long lastOverflowTime

The last commit time corresponding to the last synchronous overflow event and ZERO (0L) until there has been a synchronous overflow event.

maximumJournalSizeAtOverflow

protected long maximumJournalSizeAtOverflow

The observed maximum size of a journal (its length in bytes) as measured at each synchronous overflow event.

journalReopenCount

protected final AtomicLong journalReopenCount

The #of StoreManager.ManagedJournals that have been (re-)opened to date.

segmentStoreReopenCount

protected final AtomicLong segmentStoreReopenCount

The #of IndexSegmentStores that have been (re-)opened to date.

journalDeleteCount

protected final AtomicLong journalDeleteCount

The #of StoreManager.ManagedJournals that have been deleted to date.

segmentStoreDeleteCount

protected final AtomicLong segmentStoreDeleteCount

The #of IndexSegmentStores that have been deleted to date.

bytesUnderManagement

protected final AtomicLong bytesUnderManagement

The #of bytes currently under management EXCEPT those on the live journal. This is incremented each time a new resource is added using addResource(IResourceMetadata, File) and decremented each time a resource is deleted.

journalBytesUnderManagement

protected final AtomicLong journalBytesUnderManagement

segmentBytesUnderManagement

protected final AtomicLong segmentBytesUnderManagement

bytesDeleted

protected final AtomicLong bytesDeleted

The #of bytes that have been deleted since startup.

indexCacheLock

protected final ReentrantReadWriteLock indexCacheLock

In order to have atomic semantics and prevent a read-historical operation from starting concurrently that would have access to a view that is being purged, IndexManager.getIndex(String, long) and purgeOldResources() MUST contend for a shared lock. This is a ReentrantReadWriteLock since concurrent getIndex() requests can proceed as long as purgeOldResources() is not running. Also note that contention is not required for ITx.UNISOLATED index views.

Constructor Detail

StoreManager

protected StoreManager(Properties properties)

Note: This constructor starts an asynchronous thread that scans the data directory for journals and index segments and creates the initial journal if no store files are found.

Note: The store files are NOT accessible until the asynchronous startup is finished. Caller's MUST verify that the isOpen() AND NOT submit tasks until isStarting() returns false.

Parameters:

properties - See StoreManager.Options.

See Also:

Startup

Method Detail

getStoreCounters

public final DiskOnlyStrategy.StoreCounters getStoreCounters()

The performance counters for the IBufferStrategy backing the live journal and any historical journals which are concurrently open with the live journal. A single instance of this object is used, and a hard reference to that instance is held here, so that we can track the cumulative performance counters across the live cycles of all journal instances used by the data service over time. The performance counters are not themselves persistent and do not survive a restart of the StoreManager.

retentionSetAdd

protected void retentionSetAdd(UUID uuid)

Add an IndexSegment to the set of IndexSegments which have been generated but not yet incorporated into an index partition view and hence we must take special cautions to prevent their release.

Parameters:

The - UUID of the IndexSegmentStore.

See Also:

retentionSetRemove(UUID), retentionSet

retentionSetRemove

protected void retentionSetRemove(UUID uuid)

Remove an IndexSegment from the retentionSet. DO NOT invoke this until the IndexSegment has been incorporated in a restart safe manner into an index partition view (that is, post-commit rather than during the task that incorporates it into the view) or is known to be no longer required (post MOVE, task failed, etc).

Parameters:

uuid - The UUID of the IndexSegmentStore.

See Also:

retentionSetAdd(UUID), retentionSet

getStoreCacheSize

public int getStoreCacheSize()

The #of entries in the hard reference cache for IRawStores, including both StoreManager.ManagedJournals and IndexSegment}s. There MAY be more IRawStores open than are reported by this method if there are hard references held by the application to those IRawStores. IRawStores that are not fixed by a hard reference will be quickly finalized by the JVM.

getResourceServicePort

public int getResourceServicePort()

The port at which you can connect to the ResourceService. This service provides remote access to resources hosted by the owning DataService. This is used for moving resources to other data services in the federation, including supporting service failover.

Returns:

The port used to connect to that service.

TODO:

this could also be used for remote backup. however, note that you can not read the live journal using this object.

assertRunning

protected void assertRunning()

Succeeds if the StoreManager isOpen() and is NOT isStarting() (the test itself is NOT atomic).

Throws:

IllegalStateException - unless open and not starting.

isRunning

public boolean isRunning()

Return true iff the StoreManager is open and startup processing has been completed.

assertOpen

protected void assertOpen()

Throws:

IllegalStateException - unless open.

assertNotOpen

protected void assertNotOpen()

Throws:

IllegalStateException - if open.

awaitRunning

public boolean awaitRunning()

Return true iff the StoreManager is running. If the StoreManager is currently starting up, then this will await the completion of the Startup task.

Returns:

true if the StoreManager is running and false if it is shutdown.

getBytesUnderManagement

public long getBytesUnderManagement()

The #of bytes currently under management, including those written on the live journal.

Throws:

IllegalStateException - during startup or if the StoreManager is closed.

getJournalBytesUnderManagement

public long getJournalBytesUnderManagement()

The #of bytes in StoreManager.ManagedJournals, including those written on the live journal.

Throws:

IllegalStateException - during startup or if the StoreManager is closed.

getSegmentBytesUnderManagement

public long getSegmentBytesUnderManagement()

The #of bytes in managed IndexSegmentStores.

Throws:

IllegalStateException - during startup or if the StoreManager is closed.

getDataDirFreeSpace

public long getDataDirFreeSpace()

The #of bytes of free space remaining on the volume hosting the dataDir.

Returns:

The #of bytes of free space remaining -or- -1L if the free space could not be determined.

getTempDirFreeSpace

public long getTempDirFreeSpace()

The #of bytes of free space remaining on the volume hosting the tmpDir.

Returns:

The #of bytes of free space remaining -or- -1L if the free space could not be determined.

getProperties

public Properties getProperties()

An object wrapping the Properties given to the ctor.

isTransient

public boolean isTransient()

Return true iff data can not be made restart-safe.

isStarting

public boolean isStarting()

true initially and until #start() completes successfully.

isOpen

public boolean isOpen()

false initially and remains false until #start() completes successfully. once true this remains true until either shutdown() or shutdownNow() is invoked.

Specified by:

isOpen in interface IServiceShutdown

shutdown

public void shutdown()

Description copied from interface: IServiceShutdown

The service will no longer accept new requests, but existing requests will be processed (sychronous). This method should await the termination of pending requests, but no longer than the timeout specified by IServiceShutdown.Options.SHUTDOWN_TIMEOUT. Implementations SHOULD be synchronized. If the service is aleady shutdown, then this method should be a NOP.

Specified by:

shutdown in interface IServiceShutdown

shutdownNow

public void shutdownNow()

Description copied from interface: IServiceShutdown

The service will no longer accept new requests and will make a best effort attempt to terminate all existing requests and return ASAP. This method should terminate any asynchronous processing, release all resources and return immediately. Implementations SHOULD be synchronized. If the service is aleady shutdown, then this method should be a NOP.

Specified by:

shutdownNow in interface IServiceShutdown

getTmpDir

public File getTmpDir()

Description copied from interface: IResourceManager

The directory for temporary files.

Specified by:

getTmpDir in interface IResourceManager

getDataDir

public File getDataDir()

Note: The returned File is in canonical form.

Specified by:

getDataDir in interface IResourceManager

getManagedJournalCount

public int getManagedJournalCount()

The #of journals on hand.

getManagedSegmentCount

public int getManagedSegmentCount()

The #of index segments on hand.

addResource

protected void addResource(IResourceMetadata resourceMetadata,
                           File file)

Notify the resource manager of a new resource. The resource is added to resourceFiles and to either journalIndex or segmentIndex as appropriate. As a post-condition, you can use openStore(UUID) to open the resource using the UUID specified by IResourceMetadata.getUUID().

Note: This also adds the size of the store in bytes as reported by the OS to bytesUnderManagement.

Note: Adding a resource to the store manager has no persistent effect other than the presumed presence of the specified file in the file system. However, error handling routines SHOULD invoke deleteResource(UUID, boolean) in order to remove a resource that was not built correctly or not incorporated into the view. Otherwise the mapping from the UUID to the File will be maintained in memory and the StoreManager will overreport the #of bytes under management.

Parameters:

resourceMetadata - The metadata describing that resource.

file - The file in the local file system which is the resource.

Throws:

RuntimeException - if the file does not exist.

RuntimeException - if there is already a resource registered with the same UUID as reported by IResourceMetadata.getUUID()

RuntimeException - if the journalIndex or segmentIndex already know about that resource.

RuntimeException - if openStore(UUID) already knows about that resource.

IllegalArgumentException - if the resourceMetadata is null.

IllegalArgumentException - if the file is null and isTransient is false.

See Also:

deleteResource(UUID, boolean), retentionSetAdd(UUID), retentionSetRemove(UUID)

newFileFilter

protected ResourceFileFilter newFileFilter()

Returns a filter that is used to recognize files that are managed by this class. The ResourceManager will log warnings if it sees an unexpected file and will NOT deleteResources() files that it does not recognize.

See Also:

ResourceFileFilter

TODO:

perhaps define setFileFilter and getFileFilter instead since subclassing this method is a bit difficult. The ResourceFileFilter would have to be a static class and we would have to pass in the IResourceManager so that it could get the dataDir.

getConcurrencyManager

public abstract IConcurrencyManager getConcurrencyManager()

The object used to control access to the index resources.

Throws:

IllegalStateException - if the object has not been set yet using setConcurrencyManager(IConcurrencyManager).

setConcurrencyManager

public abstract void setConcurrencyManager(IConcurrencyManager concurrencyManager)

getLiveJournal

public StoreManager.ManagedJournal getLiveJournal()

The journal on which writes are made.

Specified by:

getLiveJournal in interface IResourceManager

Throws:

IllegalStateException - if the StoreManager is not open.

IllegalStateException - if the StoreManager is still starting up.

getJournal

public AbstractJournal getJournal(long timestamp)

Description copied from interface: IResourceManager

Return the reference to the journal which has the most current data for the given timestamp. If necessary, the journal will be opened.

Specified by:

getJournal in interface IResourceManager

Parameters:

timestamp - A transaction identifier, ITx.UNISOLATED for the unisolated index view, ITx.READ_COMMITTED, or timestamp for a historical view no later than the specified timestamp.

Returns:

The corresponding journal for that timestamp -or- null if no journal has data for that timestamp, including when a historical journal with data for that timestamp has been deleted.

Throws:

IllegalStateException - if the StoreManager is not open.

IllegalStateException - if the StoreManager is still starting up.

TODO:

write tests for unisolated and read-committed. make sure that there is no fencepost for read committed immediately after an overflow (there should not be since we do a commit when we register the indices on the new store).

openStore

public IRawStore openStore(UUID uuid)

Opens an IRawStore.

Specified by:

openStore in interface IResourceManager

Parameters:

uuid - The UUID identifying that store file.

Returns:

The open IRawStore.

Throws:

IllegalStateException - if the StoreManager is not open.

IllegalStateException - if the StoreManager is still starting up.

IllegalArgumentException - if uuid is null.

NoSuchStoreException - if the UUID is not recognized.

NoSuchStoreException - if the resource for that UUID could not be found.

RuntimeException - if something else goes wrong.

TODO:

it seems that we always have the IResourceMetadata on hand when we need to (re-)open a store so it might be nice to pass that in as it would make for more informative error messages when something goes wrong (except that I was planning to drop the file name from that interface).

nextTimestamp

protected long nextTimestamp()

Report the next timestamp assigned by the ITransactionService.

deleteResources

public void deleteResources()

Description copied from interface: IResourceManager

Deletes all resources.

Specified by:

deleteResources in interface IResourceManager

setReleaseTime

public void setReleaseTime(long releaseTime)

Updates the releaseTime.

Data services MAY release data for views whose timestamp is less than or equal to the specified release time IFF that action would be in keeping with their local history retention policy (minReleaseAge) AND if the data is not required for the most current committed state (data for the most current committed state is not releasable regardless of the release time or the minReleaseAge).

See Also:

#purgeOldResources(), which is responsible for actually deleting the old resources.

getReleaseTime

public long getReleaseTime()

Return the last value set with setReleaseTime(long).

getIndexRetentionTime

protected abstract long getIndexRetentionTime()

See Also:

IndexManager.getIndexRetentionTime()

purgeOldResources

protected final PurgeResult purgeOldResources()

Identify and delete resources no longer required by the index views from the current releaseTime up to the lastCommitTime.

Note: The ability to read from a historical commit point requires the existence of the journals back until the one covering that historical commit point. This is because the distinct historical commit points for the indices are ONLY defined on the journals. The index segments carry forward the committed state of a specific index as of the commitTime of the index from which the segment was built. This means that you can substitute the index segment for the historical index state on older journals, but the index segment carries forward only a single commit point for the index so it can not be used to read from arbitrary historical commit points.

The caller MUST hold the exclusive lock on the WriteExecutorService.

Returns:

A summary of the work done -or- null if the preconditions for the purge operation were not satisfied.

See Also:

src/architecture/purgeResourceDecisionsMatrix.xls, purgeOldResources(long, boolean)

deleteResource

protected void deleteResource(UUID uuid,
                              boolean isJournal)
                       throws NoSuchStoreException

Delete the resource in the file system and remove it from the storeCache and resourceFiles and either journalIndex or segmentIndex as appropriate.

DO NOT delete resources that are in use! A resource that has not yet been incoporated into a view may be deleted without futher concern. However, once a resource has been incorporated into a view then you MUST arange for appropriate synchronization before the resource may be deleted. For example, purgeOldResources() imposes that constraint on the caller that they are responsible for synchronization and is generally invoked during synchronous overflow since we know that there are no active writers at that time.

Pre-conditions:

• The resource identified by that UUID exists and is not the live journal.
• The resource is not in use (not checked).
• The resource is found in resourceFiles.

Post-conditions:

• The resource is closed if it was open and is no longer found in the storeCache.
• The resource is no longer found in resourceFiles.
• The backing file for the resource has been deleted (the backing file is obtain from resourceFiles).
• Various counters maintained by the StoreManager have been updated (bytes delete, bytes under management, etc).
• The file has been removed from either the journalIndex or the segmentIndex as appropriate.

Parameters:

uuid - The UUID which identifies the resource.

isJournal - true if the resource is a journal.

Throws:

NoSuchStoreException

getCommitTimeStrictlyGreaterThan

protected long getCommitTimeStrictlyGreaterThan(long releaseTime)

Finds the journal spanning the first ICommitRecord that is strictly greater than the specified timestamp and returns the timestamp of that ICommitRecord.

Parameters:

releaseTime - A release time as set by setReleaseTime(long). Any resource as of this timestamp is available for release.

Returns:

The timestamp of the first commit record whose timestamp is strictly greater than releaseTime.

Throws:

IllegalArgumentException - If there is no commit point that is strictly greater than the releaseTime. This implies that the release time is either in the future or, if the releaseTime is equal to the last commitTime, that you are trying to release everything in the database.

getResourcesForTimestamp

protected Set<UUID> getResourcesForTimestamp(long commitTimeToPreserve)

Finds all resources used by any registered index as of the commitTimeToPreserve up to and including the lastCommitTime for the live journal.

Note: We include all dependencies for all commit points subsequent to the probe in order to ensure that we do not accidently release dependencies required for more current views of the index.

Note: This method solely considers the index views as defined at each commit point starting with the given commit point. It DOES NOT pay attention to the release time or to any other aspect of the state of the system.

Parameters:

commitTimeToPreserve - The commit time corresponding to the first commit point which must be preserved.

Returns:

The set of resource UUIDs required by at least one index for any commit time GTE the specified commit time.

munge

public static String munge(String s)

Munge a name index so that it is suitable for use in a filesystem. In particular, any non-word characters are converted to an underscore character ("_"). This gets rid of all punctuation characters and whitespace in the index name itself, but will not translate unicode characters.

Parameters:

s - The name of the scale-out index.

Returns:

A string suitable for inclusion in a filename.

getIndexSegmentFile

public File getIndexSegmentFile(IndexMetadata indexMetadata)

Description copied from interface: IResourceManager

Return the file on which a new IndexSegment should be written. The file will exist but will have zero length.

Specified by:

getIndexSegmentFile in interface IResourceManager

Parameters:

indexMetadata - The index metadata.

Returns:

The file.

getIndexSegmentFile

public File getIndexSegmentFile(String scaleOutIndexName,
                                UUID indexUUID,
                                int partitionId)

Return the file on which a new IndexSegment should be written. The file will exist but will have zero length. The file is created using the File.createTempFile(String, String, File) mechanism within the configured dataDir in the subdirectory for the specified scale-out index.

Note: The index name appears in the file path above the UUID of the scale-out index. Therefore it is not possible to have collisions arise in the file system when given indices whose scale-out names differ only in characters that are munged onto the same character since the files will always be stored in a directory specific to the scale-out index.

Parameters:

scaleOutIndexName - The name of the scale-out index.

indexUUID - The UUID of the scale-out index.

partitionId - The index partition identifier -or- -1 if the index is not partitioned (handles the MDS which does not use partitioned indices at this time).

Returns:

The File on which a IndexSegmentStore for that index partition may be written. The file will be unique and empty.

Throws:

IllegalArgumentException - if any argument is null

IllegalArgumentException - if the partitionId is negative and not -1

TODO:

should the filename be relative or absolute?

purgeOldResources

public boolean purgeOldResources(long timeout,
                                 boolean truncateJournal)
                          throws InterruptedException

This attempts to obtain the exclusive lock for the WriteExecutorService. If successful, it purges any resources that are no longer required based on StoreManager.Options#MIN_RELEASE_AGE and optionally truncates the live journal such that no free space remains in the journal.

Note: If there is heavy write activity on the service then the timeout may well expire before the exclusive write lock becomes available. Further, the acquisition of the exclusive write lock will throttle concurrent write activity and negatively impact write performance if the system is heavily loaded by write tasks.

Parameters:

timeout - The timeout (in milliseconds) that the method will await the pause of the write service.

truncateJournal - When true, the live journal will be truncated to its minimum extent (all writes will be preserved but there will be no free space left in the journal). This may be used to force the DataService to its minimum possible footprint for the configured history retention policy.

truncateJournal - When true the live journal will be truncated such that no free space remains in the journal. If writes are directed to the live journal after it has been truncated then it will transparently re-extended.

Returns:

true if successful and false if the write service could not be paused after the specified timeout.

Throws:

IOException

InterruptedException

IllegalStateException - if the StoreManager is not running.

overrideJournalExtent

protected void overrideJournalExtent(Properties p)

When the StoreManager is relatively new (as measured by the #of bytes under management) we discount the journal extent in order to trigger overflow earlier. Together with the discount applied to the split handler by the AsynchronousOverflowTask, this helps to break down new index partitions allocated on the new data service and re-distribute those index partitions (if there are other data services which have even less utilization).

Parameters:

p - The properties (modified as side-effect).