bigdata®

Coherent RDF

2010-03-10T06:07:00.001-05:00

Over the years I have come to realise that few ideas are successful for the reasons they were originally developed.

Sometimes you just have to ask the question "How did we get here?".

What makes XML a success? Is it down to a unique ability to represent structured data? I think not. It is more likely because it is a lot like HTML and there have developed a range of available tools to parse and process the format. Why is HTML a success? Motivationally because of its de-facto use in markup and the magical result of browsers rendering the documents, but also because it is really easy to do something really simple and then not much harder to do something a little more complicated.

So what about RDF? RDF takes a ride on XML acceptance and introduces a triple-based data model. Where XML provides hierarchical data modelling, RDF provides a simple list of triples (at least theoretically). With this simplicity comes the problem... interpretation. XML naturally encapsulates data whilst RDF enables expansion with assertion of disconnected triples. We must follow a set of rules to interpret the triples to form coherent data structures, and these rules must be understood whenever data is created or accessed.

With BigData we want to be able to store really large amounts of information with flexible representation. A triple store provides for the flexibility, but we must be careful not to create complex problems of interpretation.

My strong feeling is that too much debate has been focused on RDF syntax and its interpretation. Instead we should understand that a triple store can flexibly represent data structures and then discover how we are able to represent the data we want to use.

So, I am arguing for a general re-appraisal of the approach to building RDF-based applications. One that essentially defocuses from RDF but is able to leverage the underlying representation when appropriate.

Is it really the case that we want to make SPARQL queries against RDF data? Of course, if we do have an underlying RDF representation then such queries could be made, but is this really the goal? When a developer is tasked with displaying a list of products purchased by a customer, do they want to make a SPARQL query or do they just want the list products?

Many years ago someone coined the term "impedance mismatch" to describe the problem of transforming data between the format used by a programming language and that used to represent it externally, for example in a database. A figure that I heard repeated was that 90% of computer code was involved with data transformation. I suspect this figure is now much higher since we no longer have to worry only about data storage transformations but also other representations. Discussing this recently with respect to RDF the phrase "mother of all impedance mismatches" was coined. We're going to call this MAIM!

So what does this mean for BigData? Well, we are aiming to solve MAIM by providing a toolset (interfaces and metadata) that enables the easy creation of domain specific models. We will use an underlying triple representation augmented with indices to support the efficient access to domain data. The flexibility of the "schema-free" triple-based representation will enable data sharing between different models/ontologies, while the domain specific metadata will resolve issues of interpretation in defined contexts.

A key advantage of this approach is that the triple representation and custom indices are driven by the requirement to support access patterns for the domain model. So, along with solving MAIM, we also hope to save on time spent discussing which RDF representation should be used.

We're Hiring

2010-03-05T16:00:00.003-05:00

Want to get paid to work on bigdata? Know someone who does? We are hiring! Here is our job listing:

Senior software engineer for parallel semantic web database query optimization.

SYSTAP, LLC is seeking a senior software engineer with experience in the Semantic Web, parallel databases, and distributed systems to help develop bigdata®, a Java-based open-source parallel Semantic Web database capable of running on clusters of 100s or 1000s of machines. SYSTAP has been recognized as a top Semantic Web startup company, and this is an exciting opportunity to work on the core development team of a cutting-edge parallel Semantic Web database product.

We are looking for immediate help with query rewrite, optimization and distributed query planning. Significant experience is desired in some or all of the following areas: the Semantic Web stack (RDF, OWL), distributed database architectures, query optimization, performance analysis, distributed query planning, view maintenance and caching, datalog, DL reasoners, parallel file systems, and GIS. Programming skills must include Java. Math and/or logic background is a plus. Many core programming (GPU) skills a plus. Active security clearance a plus. Candidate must be self-motivated with good written and verbal communication skills and must able to work independently and from home. Occasional travel required to customer sites and conferences. Competitive salary and benefits. Send cover letter and resume to jobs@systap.com.

HA Whitepaper (RFC)

2010-02-22T13:36:00.005-05:00

We have posted a draft of the bigdata HA white paper online [1]. As part of this effort, we are going to simplify the deployment, configuration and administration of the distributed database.

We are planning an initial HA release in 3-4 months. This is a great time to get us your feedback on features which you would like to see in the HA design, e.g., hooks for monitoring frameworks, etc.

You can follow the design discussions on the bigdata-developers mailing list [2].

[1] Bigdata High Availability Whitepaper
[2] Bigdata developers mailing list

HA team

2010-02-18T06:44:00.002-05:00

We would like to welcome several new members to the bigdata project team, adding a rich mixture of expertise in RDF, distributed systems, storage, deployment and agile methods. Initially we will focus on developing and testing the High Availability architecture for bigdata, with the near-term goal of a very large scale operational deployment of the bigdata quad store with high availability.

Bigdata, HA, and the clouds

2009-11-20T12:21:00.004-05:00

Mike and I have been doing some heavy thinking about the High Availability (HA) architecture for bigdata, and we think that we have a really interesting new dimension to the total story. The basic idea is that we are going to decouple the index read/write service from the distributed file system. This is going to change things in a big way. Here are some for instances:

- Add or drop bigdata services based on demand, dynamically changing the throughput for data write. This is entirely in line with the notions of cloud and will allow resource sharing on private clouds or pay per use on public clouds.

- Many machines can serve from the same shard. This means that we can hit HPC levels of concurrency for query answering. We can even partition the machines so long running queries execute against a dedicated set of hosts without interfering with normal low latency query processing.

- Blindingly fast parallel closure. There are some new algorithms out there which describe simple techniques for computing the RDFS closure of each database partition independently by replicating the "ontology" onto each node. For bigdata, this works out as computing the closure for each POS index shard independently. Using just these techniques we can probably obtain a 10x improvement when computing the database closure. However, by decoupling the index read/write services from the distributed file system, we can actually put as much as an entire machine on each POS shard, computing the closure of an arbitrarily large graph in minutes.

There are a lot of ways to handle the distributed file system, ranging from refactoring our existing code to produce a custom API, to running over any of the various distributed file systems for commodity hardware, to running over a parallel file system for an HPC cluster, SAN or NAS. Our preference is a distributed file system for commodity hardware because that supports the most reuse across both bigdata and other cloud applications, but the distributed file system must support sync to disk.

Bigdata IO is already highly optimized for both write and read operations. When we open an index segment, we fully buffer the B+Tree nodes, which are conveniently arranged in a contiguous region in the file. Likewise, when we read on an index segment, we can go directly to the leaf since the nodes are in memory. The leaves are double-linked as well, and we have bloom filters in from of the B+Tree as well as record level caching.

Journal writes are append only, occur at group commit points, and are generally 1 or more megabytes in size. Still, it might make sense to use a hybrid design where we tie the journal to a data service and use a failover chain for master and secondary data service to absorb writes. Once we close out the journal, we can blast its 200MB extent directly onto the distributed file system. Likewise, when reading against a historical commit point, we could slurp the corresponding journal onto the machine paired with the shard eliminating any remote IO for small data records on the journal. Cold shards can simply be dropped. They can be reassigned to data services on an as needed basis and brought back online nearly instantly. This will allow the number of machines dedicated to bigdata to fluctuate with the demand, which is exactly in keeping with the cloud paradigm.

This really opens up the possibilities that we see for bigdata tremendously. By tossing out the idea that the CPU/RAM of at most a single machine was available to absorb writes or compute closure, we can gain tremendous leverage from pooled computational resources.

Parallel materialization of the RDFS closure

2009-10-28T10:03:00.003-04:00

There were two excellent presentations yesterday at ISWC 2009 on using parallel techniques to materialize the RDFS closure at extremely high rates (for example, the RDFS closure of U8000 in 15 minutes). This is something that we are going to try out as soon as possible. Unlike either of the systems described in these papers, bigdata using automatic dynamic sharding based on key-ranges of the data. These techniques can be adapted by mapping the computation onto the POS index shards, bringing them to fixed point, and then reusing our high-throughput data loader to quickly relocate the entailments onto the distributed indices. There is clearly a surge in parallel and distributed algorithms for the semantic web, which is extremely exciting.

[1] Jesse Weaver, James A. Hendler. Parallel Materialization of the Finite RDFS Closure for Hundreds of Millions of Triples, In Proceedings of the 8th International Semantic Web Conference, pp. 682--697, 2009.

[2] Jacopo Urbani, Spyros Kotoulas, Eyal Oren, and Frank van Harmelen. Department of Computer Science, Vrije Universiteit Amsterdam, the Netherlands, Scalable Distributed Reasoning using MapReduce, In Proceedings of the 8th International Semantic Web Conference, 2009.

Release 0.81b

2009-10-27T09:35:00.004-04:00

We've just released a new version of bigdata. This release is capable of loading 1B triples in under one hour on a 15 node cluster and has been used to load up to 13B triples on the same cluster. JDK 1.6 is required. See [1] for instructions on installing bigdata(R), [2] for the javadoc and [3] and [4] for news, questions, and the latest developments.

Please note that we recommend checking out the code from SVN using the tag for this release. The code will build automatically under eclipse. You can also build the code using the ant script. The cluster installer requires the use of the ant script. You can checkout this release from the following URL:

https://bigdata.svn.sourceforge.net/svnroot/bigdata/tags/RELEASE_0.81b

New features:

Support for quads.
The B+Tree data record is now the same on the disk and in memory, eliminating de-serialization costs for immutable B+Tree records.
Shared LRU for all B+Tree instances in the same JVM. This provides competition across those B+Tree instances for RAM. The LRU is configured by default to use 10% of the JVM heap, but that value may be increased to 20% even on very heavy workloads with large heaps.
Parallel iterator for distributed access paths.
3x improvement in distributed query performance. We have only just begun to optimize distributed query. This performance improvement is mainly due to the parallel access path iterator, the shared LRU, and the selection of a better chunk size for distributed query. We expect substantial improvements in query performance over the next several months.
Some query hotspot elimination.

The roadmap for the next release includes:

Full transactions for the SAIL.
Record level compression.
Query optimizations.

For more information, please see the following links:

[1] http://bigdata.wiki.sourceforge.net/GettingStarted
[2] http://www.bigdata.com/bigdata/docs/api/
[3] http://sourceforge.net/projects/bigdata/
[4] http://www.bigdata.com/blog

About bigdata:

Bigdata® is a horizontally-scaled, general purpose storage and computing fabric for ordered data (B+Trees), designed to operate on either a single server or a cluster of commodity hardware. Bigdata® uses dynamically partitioned key-range shards in order to remove any realistic scaling limits - in principle, bigdata® may be deployed on 10s, 100s, or even thousands of machines and new capacity may be added incrementally without requiring the full reload of all data. The bigdata® RDF database supports RDFS and OWL Lite reasoning, high-level query (SPARQL), and datum level provenance.

ISWC 2009 : The Semantic Web at Web Scale

2009-10-18T19:25:00.004-04:00

We will be presenting on Thurs, 10/29, 11:20am - 11:50am at ISWC 2009 in Washington DC. The talk will cover the bigdata architecture, present performance results on a 15 node cluster, and the bigdata roadmap. Come on out!

Quad support for Sesame 2.x

2009-10-01T10:20:00.004-04:00

We've recently added quad support to bigdata and integrated that support with Sesame 2.x. There are now three major database modes for RDF data: plain triples, plain triples with provenance (statement identifiers), and quads. Inference is not supported yet for quads. We plan to introduce query time inference support instead of eager closure.

The support for quads is in the development branch [1], which you can check out from SVN. This branch is fairly solid and we plan to create a release from it soon [2].

[1] https://bigdata.svn.sourceforge.net/svnroot/bigdata/branches/BTREE_BUFFER_BRANCH

[2] https://sourceforge.net/apps/trac/bigdata/roadmap

B+Tree compression and buffering

2009-08-19T19:33:00.005-04:00

We've been working on big performance improvements for the B+Tree. The goal is to drastically increase the amount of data that bigdata(R) can buffer in memory so we can really take advantage of those 64-bit JVMs while making search faster, reducing the in-memory footprint and GC churn, and reducing the network size of RMI payloads.

First, we have a new binary image for the B+Tree nodes and leaves with pluggable compression schemes that support search and value retrieval directly on the coded (compressed) representation. This will reduce the footprint for data on the disk, in memory, and on the wire and should boost search performance. We will roll out with prefix-compression (front-coding) for keys and with canonical huffman encoding options for keys and values. These compression schemes all based on fastutils[1] and dsiutils[2], which are fantastic libraries.

Second, we are moving away from a per-B+Tree buffer for persistent leaves and nodes to a shared buffer using a mixture of a LRU and a hash map with weak reference values. Combined with the smaller in-memory footprint for leaves and nodes, this will allow bigdata(R) to buffer vastly more data in RAM.

Finally, we are examining options for record-level compression to reduce the on disk footprint even further. Our thinking here is to offer a pluggable scheme, with initial support for Deflate, zip, gzip and bmz[3]. We are already coding (compressing) the keys and values, but there is an additional win to be had from a fast compression pass over the entire B+Tree node or leaf. Most compression libraries are accessed via JNI and will generally do better on larger records, so we may wind up increasing the default branching factor for the index segments to compensate. You should be able to choose a record level compression provider for the Journal, another for the index segments, and have the option to specify the compression provider separately for all shards for a specific index. Different compression schemes make sense for the journal and the index segments because the journal is geared towards fast write absoption while the index segments are geared towards read-only data. Likewise, some indices may derive more benefit from specific compression schemes.

We should have some new performance results based on these changes in a few weeks.

[1] http://fastutil.dsi.unimi.it/
[2] http://dsiutils.dsi.unimi.it/
[3] http://www.hypertable.org/doxygen/dir_636d9f0bc773a215f705e2de9f182c4e.html

"Mapped" RDF data loader

2009-08-04T15:19:00.003-04:00

We've just introduced a new "mapped" data loader. The previous data loader assumed that the files were located on specific hosts in a known directory. This was designed map/reduce task in mind, where the RDF files were being dumped into the local directory by the reduce task.

With the mapped data loader you specify a scanner which is executed by the master. The default scanner knows how to identify files to be processed in file system, but it would be easy enough to write a scanner that consumed an HDFS block structured file whose contents were RDF data. This is more like the "map" stage of a map/reduce job, which is why we call it the "mapped" data loader. Regardless of how the scanner identifies the resources to be processed, the master "maps" those resources across client tasks running on the cluster.

Let us know if you are interested in loading data from HDFS or a hadoop map/reduce jobs into a massive distributed semantic web graph and we can help you work through the integration glue.

Who should use bigdata?

2009-07-27T11:40:00.001-04:00

It can be confusing to navigate the various triple store options out there. Which one is best for your application?

Let’s take a step back and look at the history of bigdata. Bigdata was not developed in a vacuum. Bryan and I were building a system for an intelligence community customer that used a triple store as the core of the data layer. This system allowed users to federate and semantically align different structured and unstructured data sets into a single fused view for better analysis. The system had a triple store knowledge base at the data layer, and then user-facing tools that allowed analysts to do things like import structured data, send unstructured documents through a harvest/extract pipeline, search for documents and entities, view graphical link charts, annotate documents, and a host of other things. The system also had an open RESTful service API, which allowed other tools to access the knowledge base to do reads, writes, and queries. The system was multi-user, so it had to handle real-time updates and deletes with concurrent queries. The knowledge base had to be fast enough to keep up with system load and scalable enough to handle lots of data. RDF was a great technology choice for the problem, but we found the RDF database implementations a bit lacking or a bit expensive, or both. And no vendor was tackling scale-out at that time. This was also around the time of Google’s publication on BigTable, and we thought, can we apply these fundamental concepts to RDF data?

Bigdata is not just for applications with multi-billion triple requirements. The single-server version of bigdata is an excellent choice for any system that needs a triple-store, it’s robust, fast, and handles concurrency very well. Bigdata handles real-time updates and deletes with incremental inference and incremental truth maintenance. Concurrent writes are serialized, but in the system for which we designed bigdata, these updates and deletes of about 10-1000 triples were absorbed almost instantly. Meanwhile bigdata’s MVCC concurrency model allows readers to operate totally independently of writers and other readers, so there’s never any waiting for reads or queries to execute. And when they do execute, they go through bigdata’s high-performance join engine for lightning fast query response times.

If you are dissatisfied with the performance, robustness, or feature-completeness of your current triple store (as we were), then look no further. Bigdata was borne of the same dissatisfaction, and designed and implemented specifically for real-world systems like yours.

getting started with bigdata

2009-07-09T17:08:00.002-04:00

A couple of steps forward to help get started with bigdata.

We've published an early draft of a bigdata architecture whitepaper[1]. It's a work in progress as you'll be able to tell by reading it.

Also, we've started sketching out the getting started guide for scale-out on the wiki[2]. We still recommend keeping us involved in the process if you're interested in trying out bigdata on a cluster. There are a lot of do's and dont's when it comes to configuring and writing performant code for a distributed database. What this guide is currently missing is sample code for distributed data load and query. Keep an eye out for this in the next few days.

[1] Whitepaper
[2] Wiki

bigdata, OWL, SWRL

2009-07-07T08:15:00.005-04:00

Now that bigdata is handling billions of triples with ease, we are ready to venture into higher expressivity as well. There is always a tradeoff between the expressiveness of the ontology and the computational complexity of computing the entailments. So far, bigdata has focused on the data scale, now we are ready to look at the reasoner complexity. To do this we are exploring some integration options, including partnering with Clark & Parsia to develop an integration with the Pellet2 OWL reasoner [2].

Speak out and let us know what combination of data scale and ontology complexity you need. Do you want datalog [1], OWL2 profiles (RL, QL, EL)[3], Horn-SHIQ[4]? Do you need SWRL [5], and how you want to use it? Example ontologies, data scale and use caselets would all help.

[1] http://www.iris-reasoner.org/
[2] http://clarkparsia.com/pellet/
[3] http://www.w3.org/TR/owl2-profiles/
[4] http://logic.aifb.uni-karlsruhe.de/wiki/Horn-SHIQ
[5] http://www.w3.org/Submission/SWRL/

come be our fan on Facebook

2009-06-29T18:23:00.001-04:00

The title says it all really. Just search for our bigdata page on FB.

version 0.8 beta

2009-06-26T13:14:00.001-04:00

We are happy to announce the release of version 0.8 beta on Sourceforge. This release packages both the scale-out (distributed) and scale-up (single-server) versions of the bigdata RDF store. This release is capable of loading 1B triples in well under one hour on a 15 node cluster and has been used to load up to 13B triples on the same cluster. It also captures substantial improvements in the scale-out architecture. While query performance has not been optimized recently, it is nevertheless quite reasonable.

We’ve also dramatically improved our Getting Started Guide[1] and have included plenty of sample code in the bigdata-sails module[2]. This guide and sample code is for the scale-up version, we will have a similar guide with sample code for the scale-out version forthcoming. In the meantime, we are happy to assist directly.

We are also currently working on a comprehensive whitepaper describing the bigdata scale-out architecture. We will post a link to that as soon as possible.

[1] http://bigdata.wiki.sourceforge.net/GettingStarted
[2] http://bigdata.svn.sourceforge.net/viewvc/bigdata/trunk/bigdata-sails/src/samples/com/bigdata/samples/

Enterprise ready scale-out?

2009-06-23T10:05:00.008-04:00

People have been inquiring whether the bigdata scale-out architecture is enterprise ready, and, if not, what it would take to get there. Bigdata, as a scale-up platform, has been in operational use for years, but this is still an early adopter phase for the scale-out architecture. What follows is an overview of some enterprise features and their current state of readiness.

If you need some feature that is not finished yet, consider getting involved or helping out by funding feature development. We can tackle any of these issues within a few months.

Datalog support for query time inference. Bigdata has an efficient internal rule execution model. SPARQL queries are translated into the internal rule model and then executed using distributed joins. Some entailments are computed at query time, but generalized query time inference requires a rewrite of the query and the RDF(S)+ entailment rules into a minimum effort program which computes exactly those entailments required to answer the query. This is done using a magic sets integration.

Status: Early development.

Transactional semantics. In order to have a semantic web database which is used as a transactional system at scale you must either use NO inference (just triples) or use query time inference rather than eager materialization of the triples. The issue is that eager materialization of inferences requires the total serialization of all transaction commits, which would be an unacceptable performance bottleneck regardless of the rest of the architecture. Either way, the semantics become those of the underlying database concurrency control algorithm, which in this case is MVCC. Further, in order to avoid race conditions for the lexicon (the mapping from RDF Values, URIs, Literals, and blank nodes, onto internal 64-bit unique identifiers) we use an ACID, but non-transactional, consistent write strategy. This guarantees a consistent mapping of RDF Values onto internal identifiers without limiting concurrency.

Status: Read-only transactions are done and are used to support high-level query. Full distributed read-write transaction support is mostly finished, but we are still working on the distributed commit protocol. MVCC is fully supported in the data services and the indices.

Concurrency Control. Bigdata uses Multi-Version Concurrency Control. We timestamp all tuples in the indices. Transactional commits identify write-write conflicts based on those timestamps. If the timestamp has been changed since the ground state from which the transaction is reading, then there is a write-write conflict. For RDF, we can reconcile write-write conflicts. The typical situation is that two transactions both write the same triple on the database. This is normally not viewed as a conflict since the RDF data is not typically used to establish ad-hoc locking protocols! Further, if we are using either query-time or NO inference, then there is no value associated with the tuples in the statement indices. All of the information is captured by the key. Write-write conflicts do arise, but only when some statements are being retracted.

Status: Done.

Locking. MVCC does not utilize locks for concurrency control. Instead, it does validation during the commit protocol as described above. We do support synchronous distributed locks through an integration with zookeeper, but that is not part of the concurrency control architecture.

Status: Done.

HA architecture. There are two alternatives here. The data service (DS) is the container for the index partitions (key-range shards). There are logical data services and physical data services. Clients always write on the master DS for a given logical DS.

Alternative 1. Clients can read from any physical DS for a given logical DS. Storage can be on either local disk or SAN/NAS. Local disk is acceptable for this alternative because the data are replicated across multiple machines, which provides built in media redundancy. The master (DS) pipelines writes to a failover chain of k secondary DS. That pipeline is flushed during the commit protocol by the master DS. The commit succeeds once the writes are on stable storage on the master and the secondaries or fails and is rolled back. If the master fails, then the 1st secondary is elected as the new master. We handle master election using zookeeper. Zookeeper was developed by Yahoo! as a distributed lock and configuration management service and is now an Apache subproject (part of Hadoop). Among other things, it gets master election protocols right.

Alternative 2. Storage is a shared volume (SAN/NAS). The secondaries DS are registered but inactive until the master fails, at which point the 1st secondary in the failover chain re-opens the same persistence store from the service directory on the SAN/NAS.

Status: Failover has not been implemented. Alternative 2 is the easiest to realize and many organizations perfer to manage storage separately from servers. Alternative 1 probably has the best price/performance for deployments since it can use local disk.

Backup and recovery. Bigdata uses a log structured store known as a "journal" to buffer writes. Periodically, the journal will reach its nominal capacity of ~200 MB. At that point, there is an atomic cutover ("synchronous overflow") to a new journal and an asynchronous overflow process migrates the buffered writes onto read-optimized B+Tree files ("index segments"). Backup therefore entails copying the index segments, the current (live) journal, and the previous journal (to capture the buffered writes). The best time to do a backup is during the atomic cutover to the new journal. At that point, write activity on the journal is suspended and it may be snap-copied. In fact, the copy operation only needs to be protected for the root blocks, which are the first page of the journal on the disk. A backup protocol could be integrated into synchronous overflow processing with very low overhead. Backup must also capture new index segments are they are generated, so there is a second integration point for that (the 1st HA strategy already requires the synchronous propagation of index segments to the failover data services).

Offline recovery is a matter of restoring the persistent state of the services and re-starting the services. Service (re-)start is quite fast, but a total database recovery would not be a fast operation.

Lightweight recovery of data with has been overwritten by transactions that you need to rollback may be achieved using the history retention policy. When you configure a bigdata federation, you can specify the minimum retention age for historical commit points. This can be hours, days, or weeks. Bigdata will not release those commit points until their retention age has expired. This makes it possible to perform correcting actions which bring the database back into a desired state. It would be quite feasible to develop a feature where the database was rolled back to a historical commit point and transactions could then be selectively reapplied from a log.

Status: Not implemented yet.

Incremental data load and query performance

2009-06-19T15:40:00.006-04:00

We have been getting some questions about incremental data load and query performance. Let me tackle both issues here.

With RDFS+ databases, there is always a trade off in when you materialize the statements entailed by the combination of the data and the ontology. This can be done eagerly, when the data are loaded into the database, or on demand, when a query is issued. Eager materialization (also known as eager closure) has advantages for some situations, but implies more latency between when you data are stable on the database and when the database can answer queries based on the new data and also takes up more space on the disk. The alternative is query-time inference, but the downside is that you must do more work to answer the query. In practice, bigdata, like most RDF databases, does a little bit of both -- materializing some statements during data load and others dynamically during query.

One of the next features that we will introduce is a magic sets integration. This will allow us to compute more of the inferences at query time using a minimum cost "program". The magic sets rewrite of the query is basically the original query, plus the entailment rules which were not materialized, plus a bunch of "gates" which prevent those rules from firing unless they are specifically required to answer the query, and even then they are fired with bindings which constrain their scope to exactly the necessary data. This integration will let us offer more flexible inference mechanisms and will give us another alternative for handling truth maintenance (when statements are retracted from the database).

Bigdata is an Multi-Version Concurrency Control (MVCC) architecture. This means that readers never block. You can use a read-only transaction to place a read-lock on a pre-materialized view of the data plus any entailment rules which are eagerly materialized. Clients can then read from that view while you add (or remove) statements to (from) the database. Once the closure of the database has been updated, you can simply update the view from which the clients are reading and they will immediately begin to read against the new state. These updates can be small and fast, or they can be massive. Bigdata simply does not care. However, if you are using eager closure then small incremental data loads will be quite fast while larger loads will take more time to update the materialized statements, but are generally more efficient. In practice, if you have a lot of small updates, it may be better to multiplex them together for more throughput -- but this depends on your application. However, clients are still reading from your last consistent view so the update is atomic from their perspective.

"Historical states" are views which have been superseded by subsequent commits. Historical state is retained automatically for open transactions. In fact, bigdata can be configured as an immortal database, where all history is preserved, if you have the disk and the requirement for that capability. More commonly, you will configure the minimum age that historical state must be retained, say 1 day, and older data is gradually purged to reclaim disk space. This all happens transparently during what we call "overflow" processing -- you never have to "vacuum" the database. When we cite high throughput like 300k triples per second or 1B triples in an hour, that is with ongoing overflow processing and dynamic index partitions partitions.

Query performance is quite good -- and we are looking forward to giving everyone some hard numbers real soon now. Query performance on a single machine is comparable to the best commercial triples stores. Query performance on a cluster varies between 2x faster and roughly equal to the best commercial triple stores running on a single machine. So, we are not loosing any performance by running a distributed database, not even for small queries. We are going to wrap up soon with the work we have been doing on data load throughput and then turn back to query performance optimization. While query performance on a cluster right now is good, we made some changes in how data is distributed across a cluster in order to achieve higher write rates and now we need re-optimize distributed query performance. In fact, I expect query performance will improve substantially when we do this, which is why we are not quoting numbers at this time.

Bigdata has a lot of advantages for query processing. For example, readers are non-blocking (MVCC) and can run concurrently, bloom filters are available at each index partition (aka key-range shard), so they can be applied to very large data sets efficiently, and we incrementally optimize the data sets by migrating buffered writes from a log-structured store onto read-optimized B+Tree segments. Overall, the scale-out architecture allows us to apply vastly more resources to query processing when compared with any single host solution.

We are working on a whitepaper in which we will publish on the scale-out architecture, data load throughput, and query performance on a cluster. We are using synthetic data sets for this, but if you have a lot of data and queries, contact us -- we'd be happy to run the numbers on your data!

Scale-up or scale-out?

2009-06-16T17:15:00.008-04:00

Scale-up or scale-out? You can scale-up to larger and larger problems by buying a machine with more RAM, or by buying an even fancier machine with a dozens of CPUs and a huge amount of RAM. However, you swifly reach the limits of what is practical (the former) or cost-effective (the latter). Commodity hardware is cheap and scale-out approaches let you make the most of it.

Bigdata loads 300,000 triples per second on a commodity cluster. 1,000,000,000 triples loaded in under an hour. You can not touch performance like that on a single machine for anything near the same cost. And it is not just the CPU and the increased RAM, but the parallel DISK IO that comes along with the architecture. And when one machine fails, you just failover. If you have sunk all those resources into a single (very) high end server and it goes down, well, you are just out of luck. Bigdata is a fully persistent architecture. Service restart time is well under a second and your data is available immediately, not re-loaded from a log file.

Bigdata is architected to be highly concurrent. Most databases limit you to one writer, or to one writer on an index. In bigdata, we transparently and dynamically break down scale-out indices into key-ranges called index partitions (shards, really) and each writes on each index partition run concurrently. This means that the potential concurrency of the application grows with the data scale. And since we dynamically partition the data, you can always add more hardware to keep pace with your data size or your query demands without having to reload all your data.

We are still working on query performance tuning, but we have already seen query response times which are better than the best scale-up triple stores. Bigdata distributes the query processing across the hardware, doing JOINs right at the data and it uses MVCC (Multi-Version Concurrency Control), so we run concurrent read-consistent queries without blocking. It is able to put more CPU, more RAM and more DISK bandwidth on any given problem when compared to any single machine.

Oh, it runs on a single machine too.

Scaling limits? No where in sight.

2009-05-29T12:55:00.004-04:00

Some people have been asking about scaling limits for bigdata. We have run out to 10B triples in a single semantic web database instance, and we are producer bound for most of that. By the time we are nearing 10B, the producers have been at maximum CPU utilization for hours while the database servers are at perhaps 35-40% utilization.

So, what is limiting us right now is the single machine capacity for the producer. As the number of index partitions grows over time, the producer needs to allocate the data to be written onto the index into more and more buffers (one per index partition). As we get into 100s of index partitions, the RDF/XML parsers are running full blast into those buffers without putting an appreciable load onto the database.

To get around the single machine limit for the producers, we are going to refactor the clients so that they have an aggregation stage, similar to, but somewhat different from, a reduce phase. That will allow us to run enough RDF/XML parser clients to feed the system and sustain high throughput well past 10B triples.

Since we can scale by adding hardware, even after a bigdata federation has been deployed, the practical scaling limit for bigdata is going to be at least another order of magnitude (100B).

Update: We have since resolved the bottleneck mentioned in the original post without the introduction of an aggregator phase. The problem was traced to some POS index queues in the clients which were being filled with small chunks due to a systematic presentation of specific predicates once per document. Those chunks are now automatically combined on insertion into the queue, which solved the problem -- at least at this scale!

Semantic Technologies Conference 2009

2009-05-21T13:30:00.004-04:00

Please come see my presentation at the Semantic Technologies Conference in San Jose on June 18th. I'll be discussing the use of bigdata and RDF for large-scale mashups.

more results from the cluster - 5 billion triples

2009-05-21T13:22:00.003-04:00

The gods have smiled upon us and given us a bit more time on this cluster.

It seems that Bryan has the RAM problem on the clients solved - they are no longer swapping. This has let us run out to a different problem - RAM demands on the data services. :-)

Good progress though, last night's run yielded 5 billion triples loaded in just under 10 hours for an average throughput of 135k triples per second. Max throughput was just above 210k triples per second. 1 billion triples was reached in an astonishing 78 minutes.

Configuration was 20 data services (10 blades with 2 data services each), 8 client services (4 blades with 2 each), and one blade for centralized services.

Note that these times are for simple RDF load - no closure. Previous tests have demonstrated that closure takes about as long as simple load - so double the time and halve the throughput to get our numbers with closure. Still quite impressive.

1 billion triples in 97 minutes

2009-05-05T16:25:00.002-04:00

With only two days left on the cluster we are really starting to feel under the gun! This afternoon's trial run is producing some exciting results though. Asynchronous writes on the TERM2ID index has produced a nice boost in throughput - we just crossed 1 billion triples loaded in 97 minutes, that is a hefty 173,000 triples per second. We are making a few more tweaks while this trial is running and will probably start a new run later in the evening in the hopes of crossing the 200,000 triples per second barrier. More to come.

100,000 triples-per-second to 3B triples

2009-05-02T10:08:00.003-04:00

At this point, we have run out to 3B triples on a cluster with a net throughput of more than 100,000 triples per second for the cluster. The per-machine throughput is now ~ 10,000 triples per second. We have also addressed a high memory demand issue in the data services which was leading to premature RAM exhaustion.

We are currently looking into memory demand for the clients, which increases in proportion to the #of index partitions. I think that we will solve this by adding compressing to the RDF Values in the ID2TERM index, leading to fewering splits of that index and hence less RAM demand on the clients.

While the throughput is now reasonable at 10,000 triples-per-second/host, I am hopeful that we can improve on this substantially by introducing asynchronous writes for the TERM2ID index.

stream-based API for scale-out index writes

2009-04-15T16:01:00.001-04:00

I have finished an implementation of a stream-based approach for writing on the scale-out indices. This does all the right things in terms of deferring RMI (index partition writes) until it has a good-sized chunk of data and having only a single outstanding RMI per client per index partition. I need to test this and then integrate it into the RDF bulk data loader in a few places and then I can start collecting performance data on it.

I think that the TERM2ID index will still need to use the synchronous RPC index writes since we need to have the assigned term identifiers before we can write on the rest of the indices. Likewise, if statement identifiers are enabled, then there will be another synchronous RPC (which is also on the TERM2ID index) to obtain the statement identifiers. Once we are done with the synchronous RPC on the TERM2ID index, the terms and statements can just be written onto an asynchronous sink for the ID2TERM, TEXT (full text lookup), and SPO, POS, and OSP indices. When the bulk load finishes, it will just await the Future for those asynchronous sinks.

More when I know more.