Comments for bigdata® http://www.bigdata.com/bigdata/blog bigdata® is a scale-out storage and computing fabric supporting optional transactions, very high concurrency, and very high aggregate IO rates. Mon, 16 Apr 2012 10:59:53 -0700 hourly 1 http://wordpress.org/?v=3.1.4 Comment on Client-Server API by admin http://www.bigdata.com/bigdata/blog/?p=441&cpage=1#comment-5385 admin Mon, 16 Apr 2012 10:59:53 +0000 http://www.bigdata.com/bigdata/blog/?p=441#comment-5385 Have you checked out the "Getting Started" page on the wiki? <a href="https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=GettingStarted" rel="nofollow">https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=GettingStarted</a> Basically, once you have a journal properties file set up and your journal is created, you can either work with it directly (embedded) via the Sesame SAIL/Repository API or you can work with it client-server through the NSS. The NSS is just a web service. TestNanoSparqlClient shows you how to set up the web service through Jetty. There is also an ant task to bundle a war for use in Tomcat. By Sesame repository are you talking about the Sesame Server, or just the SAIL/Repository API? The NSS is a replacement for the Sesame Server, which does not interact well with bigdata's concurrency semantics. The SAIL/Repository API is still the easiest way to interact with bigdata, for loading data or otherwise. The NSS is for when you need to add a remoting layer. You can load data through the NSS, but for bulk loading large amounts of data we recommend the java class DataLoader, which must be run directly against the journal. Have you checked out the “Getting Started” page on the wiki?

https://sourceforge.net/apps/mediawiki/bigdata/index.php?title=GettingStarted

Basically, once you have a journal properties file set up and your journal is created, you can either work with it directly (embedded) via the Sesame SAIL/Repository API or you can work with it client-server through the NSS. The NSS is just a web service. TestNanoSparqlClient shows you how to set up the web service through Jetty. There is also an ant task to bundle a war for use in Tomcat.

By Sesame repository are you talking about the Sesame Server, or just the SAIL/Repository API? The NSS is a replacement for the Sesame Server, which does not interact well with bigdata’s concurrency semantics. The SAIL/Repository API is still the easiest way to interact with bigdata, for loading data or otherwise. The NSS is for when you need to add a remoting layer.

You can load data through the NSS, but for bulk loading large amounts of data we recommend the java class DataLoader, which must be run directly against the journal.

]]> Comment on Client-Server API by obi http://www.bigdata.com/bigdata/blog/?p=441&cpage=1#comment-5384 obi Mon, 16 Apr 2012 10:45:34 +0000 http://www.bigdata.com/bigdata/blog/?p=441#comment-5384 Hi, Am having issues understanding how to install/configure bigdata? The documentation isn't clear enough. For instance, do I need to have a sesame repository to create a bigdata store? What's the difference between the endpoint localhost:8080/bigdata and nanosparql endpoint? Are they the same? I will be loading data to the rdf store remotely, do i do that through the REST service or can I programmatically do it through the API? Hi,
Am having issues understanding how to install/configure bigdata? The documentation isn’t clear enough. For instance, do I need to have a sesame repository to create a bigdata store? What’s the difference between the endpoint localhost:8080/bigdata and nanosparql endpoint? Are they the same? I will be loading data to the rdf store remotely, do i do that through the REST service or can I programmatically do it through the API?

]]> Comment on SPARQL 1.1 Basic Federated Query by meDavid http://www.bigdata.com/bigdata/blog/?p=399&cpage=1#comment-4960 meDavid Tue, 13 Mar 2012 20:38:54 +0000 http://www.bigdata.com/bigdata/blog/?p=399#comment-4960 The option of materializing would indeed be very nice, but I would already be happy when a remote graph could be part of a virtual graph. Looking forward to the next release with sparql update. Making a kind of subscription pattern for synchronisation shoudn't be too hard The option of materializing would indeed be very nice, but I would already be happy when a remote graph could be part of a virtual graph. Looking forward to the next release with sparql update. Making a kind of subscription pattern for synchronisation shoudn’t be too hard

]]> Comment on SPARQL 1.1 Basic Federated Query by Bryan Thompson http://www.bigdata.com/bigdata/blog/?p=399&cpage=1#comment-4952 Bryan Thompson Tue, 13 Mar 2012 11:21:58 +0000 http://www.bigdata.com/bigdata/blog/?p=399#comment-4952 You can combine the use of virtual graphs within a query with federated query. However, federated query is (typically) against a "remote" end point. There is no reason to use a federated query if the data is actually in the local end point. Virtual graphs are only simple sets of local graphs at this point. There is a proposal to have virtual graphs be the transitive closure of local graphs so they could be "nested", but it would be a different matter entirely to allow a virtual graph to be declared using what amounts to a CONSTRUCT query which identifies the relevant data. This would be more like a materialized view. We are working on SPARQL UPDATE right now. Once finished, you could certainly materialize a view (including of a remote service) into a local graph using an INSERT WHERE update statement. It would be one more step beyond that to register update operations such that they were automatically maintained. View maintenance can be pretty sophisticated. However, the simplest approach is just to watch for a change in any of the triple patterns. You can combine the use of virtual graphs within a query with federated query. However, federated query is (typically) against a “remote” end point. There is no reason to use a federated query if the data is actually in the local end point.

Virtual graphs are only simple sets of local graphs at this point. There is a proposal to have virtual graphs be the transitive closure of local graphs so they could be “nested”, but it would be a different matter entirely to allow a virtual graph to be declared using what amounts to a CONSTRUCT query which identifies the relevant data. This would be more like a materialized view.

We are working on SPARQL UPDATE right now. Once finished, you could certainly materialize a view (including of a remote service) into a local graph using an INSERT WHERE update statement. It would be one more step beyond that to register update operations such that they were automatically maintained. View maintenance can be pretty sophisticated. However, the simplest approach is just to watch for a change in any of the triple patterns.

]]> Comment on SPARQL 1.1 Basic Federated Query by meDavid http://www.bigdata.com/bigdata/blog/?p=399&cpage=1#comment-4927 meDavid Sun, 11 Mar 2012 22:23:58 +0000 http://www.bigdata.com/bigdata/blog/?p=399#comment-4927 Is it possible to combine federated query and virtual graphs in such a way that you hide the knowlege of remote graphs from the client? Can a virtual graph be a combination of local and remote graphs? Is it possible to combine federated query and virtual graphs in such a way that you hide the knowlege of remote graphs from the client? Can a virtual graph be a combination of local and remote graphs?

]]> Comment on Bigdata 1.0 Release by Bryan Thompson http://www.bigdata.com/bigdata/blog/?p=273&cpage=1#comment-2393 Bryan Thompson Thu, 07 Jul 2011 23:04:28 +0000 http://www.bigdata.com/bigdata/blog/?p=273#comment-2393 William, The cluster configuration was 15 32G nodes with 8 cores and 512G of local disk, 1G Ethernet, 64 bit OS and 64 bit JVM. We've done things like this on Rackspace as well as private clouds. Performance for the cluster (and for analytic query workloads) should improve sharply after our next milestone release which will feature the memory manager to handle very large data sets in main memory without causing GC problems for the JVM. The main requirements to load 50B triples on a single node are plenty of disk space and time. A cluster is significantly faster at that data scale. I would recommend 32+G of RAM and SSD by preference. The indices will be pretty deep by the time you approach that limit. The 50B limit is based on the addressing scheme of the RWStore. William,

The cluster configuration was 15 32G nodes with 8 cores and 512G of local disk, 1G Ethernet, 64 bit OS and 64 bit JVM. We’ve done things like this on Rackspace as well as private clouds. Performance for the cluster (and for analytic query workloads) should improve sharply after our next milestone release which will feature the memory manager to handle very large data sets in main memory without causing GC problems for the JVM.

The main requirements to load 50B triples on a single node are plenty of disk space and time. A cluster is significantly faster at that data scale. I would recommend 32+G of RAM and SSD by preference. The indices will be pretty deep by the time you approach that limit. The 50B limit is based on the addressing scheme of the RWStore.

]]> Comment on Bigdata 1.0 Release by William Sanchez http://www.bigdata.com/bigdata/blog/?p=273&cpage=1#comment-2391 William Sanchez Thu, 07 Jul 2011 20:34:42 +0000 http://www.bigdata.com/bigdata/blog/?p=273#comment-2391 Bryan, This is great, We will install it and test it soon. I have a couple of questions about journal and federation. What is the requirements for hardware and software to load 50B in one server. The second question, what is the node configuration requirements. You mentioned 15 nodes cluster to load 1Bt, what was the node configuration. Thanks, William Bryan,
This is great, We will install it and test it soon. I have a couple of questions about journal and federation. What is the requirements for hardware and software to load 50B in one server. The second question, what is the node configuration requirements. You mentioned 15 nodes cluster to load 1Bt, what was the node configuration.
Thanks,
William

]]> Comment on SemTech 2010 – Practical RDF by Tweets that mention SemTech 2010 – Practical RDF » bigdata® -- Topsy.com http://www.bigdata.com/bigdata/blog/?p=172&cpage=1#comment-219 Tweets that mention SemTech 2010 – Practical RDF » bigdata® -- Topsy.com Tue, 17 Aug 2010 02:59:26 +0000 http://www.bigdata.com/bigdata/blog/?p=172#comment-219 [...] This post was mentioned on Twitter by Vijay Raj, Cambridge Semantics. Cambridge Semantics said: Systap BigData blog post on the relevance of RDF storage in today's world of XML, etc. http://www.bigdata.com/bigdata/blog/?p=172 [...] [...] This post was mentioned on Twitter by Vijay Raj, Cambridge Semantics. Cambridge Semantics said: Systap BigData blog post on the relevance of RDF storage in today's world of XML, etc. http://www.bigdata.com/bigdata/blog/?p=172 [...]

]]> Comment on Online query workloads: Cache, NIO2, GC, SSD. by Andreas Harth http://www.bigdata.com/bigdata/blog/?p=84&cpage=1#comment-128 Andreas Harth Wed, 09 Jun 2010 14:43:51 +0000 http://www.bigdata.com/bigdata/blog/?p=84#comment-128 Bryan, interesting points! The LRU (or LIRS) caching could also be applied to cache network access in a distributed archiecture with network access that requires caching. Best regards, Andreas. Bryan,

interesting points! The LRU (or LIRS) caching could also be applied to cache network access in a distributed archiecture with network access that requires caching.

Best regards,
Andreas.

]]> Comment on Online query workloads: Cache, NIO2, GC, SSD. by Online query workloads: Cache, NIO2, GC, SSD. » bigdata® « cache http://www.bigdata.com/bigdata/blog/?p=84&cpage=1#comment-121 Online query workloads: Cache, NIO2, GC, SSD. » bigdata® « cache Mon, 07 Jun 2010 14:56:17 +0000 http://www.bigdata.com/bigdata/blog/?p=84#comment-121 [...] Adres URL: Online query workloads: Cache, NIO2, GC, SSD. » bigdata® [...] [...] Adres URL: Online query workloads: Cache, NIO2, GC, SSD. » bigdata® [...]

]]>