ThroughputMaster (bigdata® v1.2.1)

Overview

Package

Class

Use

Tree

Deprecated

Index

Help

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

com.bigdata.service.jini.benchmark
Class ThroughputMaster

java.lang.Object
  com.bigdata.service.jini.master.TaskMaster<ThroughputMaster.JobState,com.bigdata.service.jini.benchmark.ThroughputMaster.ClientTask,Void>
      com.bigdata.service.jini.benchmark.ThroughputMaster

All Implemented Interfaces:: Callable<Void>

public class ThroughputMaster
extends TaskMaster<ThroughputMaster.JobState,com.bigdata.service.jini.benchmark.ThroughputMaster.ClientTask,Void>
extends TaskMaster<ThroughputMaster.JobState,com.bigdata.service.jini.benchmark.ThroughputMaster.ClientTask,Void>

Utility class for benchmarking index operations on a federation. This test harness can be parameterized for at least the following:

general purpose scale-out index performance tests: These tests should focus on bulk index read, write, range count, and key-range scan operations. general purpose scale-out indices do not constrain the choice of separator keys for index partitions so any given read or write MAY be split across one or more index partitions. The workload can be shaped by the #of clients, by the relative proportion of reads and writes (including inserts, updates, and deletes), by the distribution of the center and spread of each read and write operation, the key type, the value type, and by the frequency and key range of range count and key range scan operations.
sparse row store performance tests: These tests should excercise atomic row read, atomic row write, and atomic row scans. by design the sparse row store restricts any given logical row to a single index partition so each row read and row write operation will be directed to ONE index partition. The row scan is atomic for each row read, and may read multiple rows atomically. The workload can be shaped by the #of clients, the degree of scatter of the primary keys for the logical row, the #of non-null columns in each row, the #of distinct timestamped values for each column, the key type, the value types, the mixture of reads and writes (including inserts, updates, and deletes) (probably 90/10), and the frequency and range of row scans. It is sufficient to test with a single schema (aka column family) since that is realized as a simple prefix on the primary key. validation should ensure that row reads and row writes are never split across index partition boundaries. That can be done by a read on the metadata index in which we examine the separator keys or by asserts in the code for read, write and scan. It is the responsibility of the ISimpleSplitHandler to ensure that separator keys are chosen for index partitions that fall on logical row boundaries.
bigdata file system performance tests: These tests should exercise block append, read, and update and stream oriented read and write. the workload can be shaped by the #of clients, by the #of and distribution of files read and written, by the #of blocks in a file, etc.

These tests share several common features:

Operations are independent
Operations are UNISOLATED

Operations are local for the sparse row store and the bigdata file system, and MAY or MAY NOT be local for generalized scale-out indices depending on the workload. To the extent that the operations are independent and local the system should scale linearly. Applications that concentrate operations on a few key ranges will create hot spots for those key ranges which may or may not translate into bottlenecks. Likewise, cache may be defeated to the extent that operations rarely revisit recently visited parts of the key space.

Higher-level applications (such as the RDF DB) can introduce dependencies in the scale-out indices. For example, a write on the RDF DB must first write on TERM2ID, then on ID2TERM, and finally may write on the SPO, POS, and OSP indices in parallel.

You can examine the performance on the system at several levels:

Operating System: The AbstractFederation collects performance counters from the underlying operating system using vmstat or the equivalent and relays them to the LoadBalancerService. This is done (by default) for all machines participating in a federation, including those running clients.
Client: The AbstractFederation collects statistics on its work queues, including the moving average of the queue length and the moving average of the queueing time. This information is collected for all clients (including the DataServices) and is relayed to the LoadBalancerService. This mechanism may be extended by an application to report additional performance counters. Clients MAY also report their own Events.
Data services: The DataServers collects statistics on its internal work queues, including the moving average of the various queue lengths and the moving average of the queueing time for those queues. The most interesting queue is generally the "write service", which is where tasks which will write on an index are queued. The DataServer also reports a variety of Events to the LoadBalancerService, most of which deal with synchronous and asynchronous overflow handling.

The LoadBalancerService normally exposes an CounterSetHTTPD instance which may be used to examine the aggregated performance counters using a navigational model or as correlated timeseries and pivot tables suitable for import into a worksheet program. The CounterSetHTTPD also exposes an Event viewer. See XHTMLRenderer. In addition, any client or service MAY optionally expose a CounterSetHTTPD instance giving a "live" look at the performance counters for that host and service, but only the current value for each counter will be available (no timeseries). Finally, the LoadBalancerService logs both the performance counters and the Events. The performance counter logs are available for post-mortem analysis using CounterSetHTTPDServer. The Events are logged in a tab-delimited format and may be imported into a worksheet for further analysis.

While bigdata does not report metrics via JMX, you can enable remote JMX and examine JMX counters for JVMs throughout the federation. This can be done trivially if you are running one DataServer per host since you can pre-assign the port number for remote JMX in the Configuration file.

Version:: $Id$
Author:: Bryan Thompson

Nested Class Summary
`static interface`	`ThroughputMaster.ConfigurationOptions` `Configuration` options for the `ThroughputMaster`.
`static class`	`ThroughputMaster.JobState` State describing the job to be executed.
`static class`	`ThroughputMaster.Task` Run an unisolated operation.

Nested classes/interfaces inherited from class com.bigdata.service.jini.master.TaskMaster
`TaskMaster.DiscoveredServices`

Field Summary

Fields inherited from class com.bigdata.service.jini.master.TaskMaster
`fed, log`

Constructor Summary
`protected`	`ThroughputMaster(JiniFederation fed)`

Method Summary
`protected void`	`beginJob(ThroughputMaster.JobState jobState)` Extended to register a scale-out index in the specified namespace.
`static void`	`main(String[] args)` Runs the master.
`protected com.bigdata.service.jini.benchmark.ThroughputMaster.ClientTask`	`newClientTask(int clientNum)` Return a client to be executed on a remote data service.
`protected ThroughputMaster.JobState`	`newJobState(String component, net.jini.config.Configuration config)` Return a `TaskMaster.JobState`.

Methods inherited from class com.bigdata.service.jini.master.TaskMaster
`allDone, attachPerformanceCounters, awaitAll, call, cancelAll, detachPerformanceCounters, error, execute, forceOverflow, getFederation, getJobState, innerMain, notifyOutcome, runJob, setupJob, startClients, success, tearDownJob`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail