com.bigdata.service.ndx.pipeline
Class DefaultDuplicateRemover<O>

java.lang.Object
  com.bigdata.service.ndx.pipeline.DefaultDuplicateRemover<O>

Type Parameters:

O - The generic type of the object associated with the key-value pair.

All Implemented Interfaces:

IDuplicateRemover<O>

public class DefaultDuplicateRemover<O>
extends Object
implements IDuplicateRemover<O>

Implementation which retains one instance of each tuple having the same unsigned byte[] key and the same byte[] value. For efficiency, you may specify that the presence of the same non-null object reference may be used to detect duplicates without requiring the comparison of the byte[] values.

When duplicates are eliminated, KVOCs identified as duplicates are arranged into a linked list.

Version:

$Id: DefaultDuplicateRemover.java 2265 2009-10-26 12:51:06Z thompsonbry $

Author:

Bryan Thompson

Field Summary

static IDuplicateRemover	KEY_REF_VAL Instance verifies the same unsigned byte[] key and will accept the same non-null object reference as indicating the same value.
static IDuplicateRemover	KEY_VAL Instance verifies the same unsigned byte[] key and the same byte[] value.,

Constructor Summary

DefaultDuplicateRemover(boolean testRefs)

Method Summary

KVO<O>[]	filter(KVO<O>[] src) Accepts a dense array of key-value tuples and returns a dense array of key-value tuples in which duplicates have been eliminated.
protected boolean	filterDuplicate(KVO<O> prior, KVO<O> other) Return true if the other instance is a duplicate and may be dropped.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

KEY_VAL

public static final transient IDuplicateRemover KEY_VAL

Instance verifies the same unsigned byte[] key and the same byte[] value.,

KEY_REF_VAL

public static final transient IDuplicateRemover KEY_REF_VAL

Instance verifies the same unsigned byte[] key and will accept the same non-null object reference as indicating the same value. If the object reference is null then it will compare the byte[] values.

Constructor Detail

DefaultDuplicateRemover

public DefaultDuplicateRemover(boolean testRefs)

Parameters:

testRefs - When true, KVOs having the same key and the same non-null object reference will be filtered without testing the byte[] values for equality.

Method Detail

filter

public KVO<O>[] filter(KVO<O>[] src)

Description copied from interface: IDuplicateRemover

Accepts a dense array of key-value tuples and returns a dense array of key-value tuples in which duplicates have been eliminated. The elements of the given array MUST NOT be modified as a side-effect.

Implementations MUST be thread-safe.

Implementations MUST test for KVOList and obey its contract (this ensures that we can map KVO.done() over the eliminated duplicates once the original has been written successfully onto the database).

Specified by:

filter in interface IDuplicateRemover<O>

Parameters:

src - A sorted array of key-value tuples.

Returns:

The a dense array of key-value tuples in which any duplicates have been eliminated.

filterDuplicate

protected boolean filterDuplicate(KVO<O> prior,
                                  KVO<O> other)

Return true if the other instance is a duplicate and may be dropped. (This implementation recognizes KVOList and handles it appropriately.)

Parameters:

prior - The previous KVO instance.

other - Another KVO instance.

Returns:

true if the other is a duplicate.