com.bigdata.relation.rule.eval
Class RuleStats

java.lang.Object
  com.bigdata.relation.rule.eval.RuleStats

public class RuleStats
extends Object

Statistics about what an IStep did when it was executed.

Program execution has the general form of either a set of ISteps executed, at least logically, in parallel, or a sequence of ISteps executed in sequence. An IStep may be a closure operation of one or more IRules, even when it is the top-level IStep. Inside of a closure operation, there are one or more rounds and each rule in the closure will be run in each round. There is no implicit order on the rules within a closure operation, but they may be forced to execute in a sequence if the total program execution context forces sequential execution.

In order to aggregate the data on rule execution, we want to roll up the data for the individual rules along the same lines as the program structure.

Version:

$Id: RuleStats.java 2265 2009-10-26 12:51:06Z thompsonbry $

Author:

mikep, Bryan Thompson

TODO:

Report as counters aggregated by the ILoadBalancerService?

Field Summary

long[]	chunkCount The #of chunks materialized for each predicate in the body of the rule (in the order in which they were declared, not the order in which they were evaluated).
int	closureRound The round is zero unless this is a closure operations and then it is an integer in [1:nrounds].
List<RuleStats>	detailStats When execution RuleStates are being aggregated, this will contain the individual RuleStats for each execution RuleState.
long	elapsed Time to compute the entailments (ms).
long[]	elementCount The #of elements considered for each predicate in the body of the rule (in the order in which they were declared, not the order in which they were evaluated).
int[]	evalOrder The order of execution of the predicates in the body of a rule (only available at the detail level of a single rule instance execution).
IKeyOrder[]	keyOrder An array of the IKeyOrder that was used for each predicate in the tail of the rule.
AtomicLong	mutationCount The #of elements that were actually added to (or removed from) the relation indices.
String	name The name of the rule.
int[]	nvars The #of unbound variables for the predicates in the tail of the Rule (only available at the detail level of a single rule instance execution).
int[]	permutation The permutation of the tail predicate index order for an IRule (only available when the IStep is an IRule) that corresponds to the evaluation order of the tail predicate in the IRule.
long[]	rangeCount The predicated range counts for each predicate in the body of the rule (in the order in which they were declared, not the order in which they were evaluated) as reported by the IRangeCountFactory.
IStep	rule The IStep that was executed.
AtomicLong	solutionCount The #of ISolutions computed by the rule regardless of whether or not they are written onto an IMutableRelation and regardless of whether or not they duplicate a solution already computed.
long	startTime The start time for the rule execution.
int[]	subqueryCount The #of subqueries examined for each predicate in the rule (in the order in which they were declared, not the order in which they were evaluated).

Constructor Summary

RuleStats(IRuleState ruleState)
Initializes statistics from an iRule and its IEvaluationPlan.

RuleStats(IStep step)
Initializes statistics for an IStep.

Method Summary

void	add(RuleStats o) Aggregates statistics.
String	getHeadings() Returns the headings.
String	toString() Reports aggregate and details.
protected String	toString(IPredicate pred) Return a human readable representation of the predicate.
String	toString(long minElapsed, boolean joinDetails)
String	toStringSimple(int depth, boolean titles, boolean joinDetails) Reports just the data for this record.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

name

public final String name

The name of the rule.

rule

public final IStep rule

The IStep that was executed.

closureRound

public int closureRound

The round is zero unless this is a closure operations and then it is an integer in [1:nrounds].

solutionCount

public AtomicLong solutionCount

The #of ISolutions computed by the rule regardless of whether or not they are written onto an IMutableRelation and regardless of whether or not they duplicate a solution already computed.

Note: this counter will be larger than the actual #of solutions generated when evaluating an ISlice since the pattern used is to invoke AtomicLong.incrementAndGet() and add the ISolution to the IBuffer iff the post-increment value is LT ISlice.getLast(). Since we are working with the post-increment value when handling an ISlice this is naturally one larger than the #of ISolutions actually added to the IBuffer if we halt processing because we have reached the limit on the ISlice.

mutationCount

public AtomicLong mutationCount

The #of elements that were actually added to (or removed from) the relation indices. This is updated based on the IMutableRelation API. Correct reporting by that API and correct aggregation here are critical to the correct termination of the fix point of some rule set.

Note: This value is ONLY incremented when the IBuffer is flushed.

Note: Each IRule will have its own RuleStats on which it reports its mutation count. However, since evaluation of IRules MAY proceed in parallel and since multiple IRules can write on the same IBuffer, the mutation counts can not be credited unambiguously to any given rule when rules run in parallel.

Note: IBuffer.flush() maintains a running mutationCount. Therefore only the end state of that counter should be set on mutationCount. Otherwise the reported mutationCount may overreport the actual mutation count.

See Also:

add(RuleStats)

startTime

public final long startTime

The start time for the rule execution. This is approximate (it is initialized when the RuleStats instance is created). The startTime is mainly intended for use when correlating collected performance counters with query execution.

elapsed

public long elapsed

Time to compute the entailments (ms).

nvars

public final int[] nvars

The #of unbound variables for the predicates in the tail of the Rule (only available at the detail level of a single rule instance execution). The array is correlated with the predicates index in the tail of the rule NOT with its evaluation order.

evalOrder

public final int[] evalOrder

The order of execution of the predicates in the body of a rule (only available at the detail level of a single rule instance execution). When aggregated, the evalOrder will always contain zeros since it can not be meaningfully combined across executions of either the same or different rules.

permutation

public final int[] permutation

The permutation of the tail predicate index order for an IRule (only available when the IStep is an IRule) that corresponds to the evaluation order of the tail predicate in the IRule. permutation[i] is the sequence in the evaluation order at which tail[i] was evaluated.

keyOrder

public final IKeyOrder[] keyOrder

An array of the IKeyOrder that was used for each predicate in the tail of the rule. The array is correlated with the predicates index in the tail of the rule NOT with its evaluation order.

rangeCount

public final long[] rangeCount

The predicated range counts for each predicate in the body of the rule (in the order in which they were declared, not the order in which they were evaluated) as reported by the IRangeCountFactory. The range counts are used by the IEvaluationPlan. You can compare the elementCounts with the rangeCounts to see the actual vs predicated #of elements visited per predicate.

chunkCount

public final long[] chunkCount

The #of chunks materialized for each predicate in the body of the rule (in the order in which they were declared, not the order in which they were evaluated).

elementCount

public final long[] elementCount

The #of elements considered for each predicate in the body of the rule (in the order in which they were declared, not the order in which they were evaluated).

subqueryCount

public final int[] subqueryCount

The #of subqueries examined for each predicate in the rule (in the order in which they were declared, not the order in which they were evaluated). While there are N indices for a rule with N predicates, we only evaluate a subquery for N-1 predicates so at least one index will always be zero(0).

detailStats

public List<RuleStats> detailStats

When execution RuleStates are being aggregated, this will contain the individual RuleStats for each execution RuleState.

Constructor Detail

RuleStats

public RuleStats(IStep step)

Initializes statistics for an IStep.

Note: This form is used when statistics will be aggregated across the execution of multiple ISteps, such as when computing the closure (fix point) of a set of IRules or a sequential IProgram.

Parameters:

step - The IStep.

RuleStats

public RuleStats(IRuleState ruleState)

Initializes statistics from an iRule and its IEvaluationPlan.

Note: This ctor variant makes available the order of execution and range count metadata from the IEvaluationPlan.

Parameters:

rule - The IRule.

plan - The IEvaluationPlan.

keyOrder - Identifies which index will be used at each step in the evaluation plan (the indices are correlated with the tail predicate index, not the evaluation order index).

Method Detail

getHeadings

public String getHeadings()

Returns the headings.

The following are present for every record.

startTime

Timestamp taken when the rule begins to execute (approximate). This is primarily used to correlate performance counters with query execution.

rule

The name of the rule.

elapsed

Elapsed execution time in milliseconds. When the rules are executed concurrently the individual times will not be additive (they will sum to more than the elapsed clock time owing to parallel execution).

solutionCount

The #of solutions computed.

solutions/sec

The #of solutions computed per second.

mutationCount

The #of solutions that resulted in mutations on a relation (i.e., the #of distinct and new solutions). This will be zero unless the rule is writing on a relation.

mutations/sec

The #of mutations per second.

The following are only present for individual IRule execution records. Each of these is an array containing one element per tail predicate in the IRule.

evalOrder

The evaluation order for the predicate(s) in the rule.

keyOrder

The IKeyOrder for the predicate(s) in the rule. Basically, this tells you which index was used for each predicate.

nvars

The #of variables that will be unbound in the predicate when it is evaluated (this is a function of the selected evaluation plan).

rangeCount

The #of elements predicated for each tail predicate in the rule by the IRangeCountFactory on behalf of the IEvaluationPlan.

chunkCount

The #of chunks that were generated for the left-hand side of the JOIN for each predicate in the tail of the rule.

elementCount

The #of elements that were actually visited for each tail predicate in the rule.

subqueryCount

The #of subqueries issued for the right-hand side of the JOIN for each predicate in the tail of the rule.

tailIndex

The index in which the tail predicate(s) for rule were declared for the rule. This information is present iff joinDetails == true was specified. Since the tail predicates will be written into the table in this order, this information is mainly of use if you want to resort the table while keeping the relationship between the predicate evaluation order and the declared predicate order.

tailPredicate

The tail predicate(s) for rule in the order in which they were declared for the rule. This information is present iff joinDetails == true was specified. When present, the tail predicate details will be found on their own rows in the table and will be aligned under the column whose details are being broken out.

TODO:

collect data on the size of the subquery results. A large #of subqueries with a small number of results each is the main reason to unroll the JOIN loops., we are actually aggregating the elapsed time, which is a no-no as pointed out above when there are rules executing concurrently.

toStringSimple

public String toStringSimple(int depth,
                             boolean titles,
                             boolean joinDetails)

Reports just the data for this record.

Parameters:

depth - The depth at which the record was encountered within some top-level aggregation.

titles - When true the titles will be displayed inline, e.g., foo=12 vs 12.

joinDetails - When true , presents a tabular display of the details for each JOIN IFF this RuleStats was collected for an IRule (rather than an aggregation of IRule).

toString

protected String toString(IPredicate pred)

Return a human readable representation of the predicate. Subclasses may be created that know how to externalize the predicate correctly for its relation. It is a good idea to strip commas from the representation so that the table can be more readily imported into worksheets that exhibit problems with quoted strings embedding commas. This can be done using String.replace(CharSequence, CharSequence).

Parameters:

pred - A predicate from the tail of an IRule.

Returns:

The representation of that predicate.

toString

public String toString()

Reports aggregate and details.

Overrides:

toString in class Object

toString

public String toString(long minElapsed,
                       boolean joinDetails)

Parameters:

minElapsed - The minimum elapsed time for which details will be shown.

joinDetails - When true, also presents a tabular display of the details for each JOIN in each IRule.

add

public void add(RuleStats o)

Aggregates statistics.

Note: since the mutation count as reported by each buffer is cumulative we DO NOT aggregate the mutation counts from sub-steps as it would cause double-counting when steps are executed in parallel.

Instead, once the total program is finished, the total mutation count is computed as the value reported by IBuffer.flush() for each buffer on which the program writes.

Parameters:

o - Statistics for another rule.

See Also:

ProgramTask.executeMutation(IStep), MutationTask#flushBuffers(IJoinNexus, RuleStats, java.util.Map)