com.bigdata.service.ndx.pipeline
Interface IDuplicateRemover<O>

All Known Implementing Classes:

DefaultDuplicateRemover, NOPDuplicateRemover

public interface IDuplicateRemover<O>

Interface allowing application specific elimination of duplicates.

Version:

$Id: IDuplicateRemover.java 2265 2009-10-26 12:51:06Z thompsonbry $

Author:

Bryan Thompson

Method Summary

KVO<O>[]

filter(KVO<O>[] a) Accepts a dense array of key-value tuples and returns a dense array of key-value tuples in which duplicates have been eliminated.

Method Detail

filter

KVO<O>[] filter(KVO<O>[] a)

Accepts a dense array of key-value tuples and returns a dense array of key-value tuples in which duplicates have been eliminated. The elements of the given array MUST NOT be modified as a side-effect.

Implementations MUST be thread-safe.

Implementations MUST test for KVOList and obey its contract (this ensures that we can map KVO.done() over the eliminated duplicates once the original has been written successfully onto the database).

Parameters:

a - A sorted array of key-value tuples.

Returns:

The a dense array of key-value tuples in which any duplicates have been eliminated.

TODO:

The thread safe requirement is probably not necessary. At least as implemented, duplicates are eliminated by the IndexPartitionWriteTask within a single-threaded context.