Subscribe

Moriarty DataTables: Active Record for RDF

DataTables are a new addition to the Moriarty PHP library. They are an implementation of the ActiveRecord pattern for use with RDF data in Talis Platform stores. It draws inspiration from the active record implementation in CodeIgniter.

The intention is to allow querying of RDF data in a natural way for most PHP coders. For example:

$dt->select('firstname')->from('person')->where('surname','Evans');
$dt->get();

In a relational database that kind of code would select the firstname column for every record in the person table that has a surname column with a value of Evans. With RDF we have two problems:

  1. there are no columns or tables, instead we have properties and classes.
  2. URIs are used to name things and URIs are long, ugly and easy to get wrong.

Moriarty’s DataTable class attempts to solve these two problems. It solves the first by treating properties as columns and classes as tables. The second problem it solves by allowing the user to specify short names for URIs. So we can write:

$dt->map('http://xmlns.com/foaf/0.1/firstName', 'firstname');
$dt->map('http://xmlns.com/foaf/0.1/surname', 'surname');
$dt->map('http://xmlns.com/foaf/0.1/Person', 'person');
$dt->select('firstname')->from('person')->where('surname','Evans');
$dt->get();

We can read that as selecting all values of the foaf:firstName property for resources of type foaf:Person that also have a foaf:surname property with a value of Evans. The DataTable class converts that into a SPARQL select query behind the scenes.

This means you can very simply query and use RDF data from a Talis Platform store. To get the first 10 names and nicknames from a store:

$dt = new DataTable('http://api.talis.com/stores/mystore');
$dt->map('http://xmlns.com/foaf/0.1/name', 'name');
$dt->map('http://xmlns.com/foaf/0.1/nick', 'nick');

$dt->select('name,nick')->limit(10);
$res = $dt->get();

foreach ($res->result() as $row) {
   echo $row->name;
   echo $row->nick;
}

I’ve written up a collection of example queries based on the education data held in the data.gov.uk service.

When I was thinking about how to map the ideas from active record into RDF I was stumped at how to implement table joins. This bothered me because if there is one thing RDF excels at it’s links between resources. Here’s an example of how CodeIgniter implements the join syntax:

$this->db->select('firstname, blog.title');
$this->db->from('person');
$this->db->join('blog', 'person.id = blog.id');

It turned out that the answer was incredibly simple and elegant: you don’t need them! The whole concept of the join method in most active record implementations is to compensate for the fact that relational databases don’t name their relationships (some do but it is very rarely used in practice and not commonly supported in SQL). If you think about the RDF equivalent of that query it becomes clearer: select the name of each resource of type person and for each of their blogs select its title. That join is just the property relating the person resource to the blog resource, probably foaf:weblog.

When you use a DataTable you specify a join simply by including a dotted property path in the select method, e.g. blog.title where blog and title both map to properties. That lets us write our query like this:

$dt->map('http://xmlns.com/foaf/0.1/firstName', 'firstname');
$dt->map('http://xmlns.com/foaf/0.1/weblog', 'blog');
$dt->map('http://purl.org/dc/elements/title', 'title');

$this->db->select('firstname, blog.title');
$this->db->from('person');

Ignoring the mappings this is much simpler than the relational database equivalent! Here’s a good example of using these joinless queries.

DataTables aren’t just for querying. They also support insert and update. To insert a new description of a resource:

$dt = new DataTable('http://api.talis.com/stores/mystore');
$dt->map('http://xmlns.com/foaf/0.1/name', 'name');
$dt->map('http://xmlns.com/foaf/0.1/Person', 'person');
$dt->set('name', 'scooby');
$response = $dt->insert('person');

This translates to submitting a description of a blank node, with a foaf:name property having a value of scooby and an rdf:type of foaf:Person. If you want to submit a description about a resource with a known URI, then you need to set the special _uri field like this:

$dt->set('_uri', 'http://example.com/people/1');
$dt->set('name', 'scooby');
$response = $dt->insert('person');

Behind the scenes the insert method generates the RDF and POSTs it into the store’s metabox. Updates work in a similar way:

$dt->set('_uri', 'http://example.com/people/1');
$dt->set('name', 'scooby');
$dt->where('nick', 'scoob');
$response = $dt->update();

Here the update method queries the store for the current value of the name property for the specified resource and generates a changeset which it then submits to the store’s metabox. This also works for multiple resources, so to update the resource description for anything with a name of shaggy to have a name of scooby:

$dt->set('_uri', 'http://example.com/people/1');
$dt->set('name', 'scooby');
$dt->where('name', 'shaggy');
$response = $dt->update();

Full documentation can be found here: DataTable and DataTableResult

About Moriarty… Moriarty is a simple PHP library for accessing the Talis Platform. It follows the Platform API very closely and wraps up many common tasks into convenient classes while remaining very lightweight. It also provides some simple RDF classes that are based on the excellent ARC2 class library. Moriarty is being developed by small community of developers and is in continual beta, subject to a slow stream of updates. To find out more visit its Google Code project

Using Moriarty for Serving Linked Data

Although Moriarty is a general purpose library for building applications with the Talis Platform (and tried and tested in Talis Prism and Talis Aspire) one of the most common uses is simply to provide a browsable interface for linked data held in a Talis Platform store. Typically these scripts take the URI sent by the web browser and use a SPARQL query or the Talis Platform’s describe service to fetch linked data about that URI. They then style that as HTML or send it directly back as RDF. There are a series of technical details they all need to deal with: 303 redirects, content negotiation, converting RDF to HTML etc.

I’ve worked on more comprehensive libraries (e.g. Paget) to manage this kind of publishing but I thought the simple case of fetching and styling the data would make a good example of how to use Moriarty. I spent a bit of time this afternoon putting an example script together based on several I’ve written in the past. You can find the result in the dataspace subdirectory of the examples folder. That subdirectory contains four files:

  • dataspace.php — this is the example script. It contains the logic to fetch the relevant description, handle content negotiation of the best output format and styling the result appropriately. It’s not designed to be called directly, but to be included from a configuration file…
  • index.php — this is an example configuration file. It is designed to be dropped into a web server directory and then intercept all requests beginning with that URI. It contains the configuration describing which Talis Platform store to use, where cache files can be written and where to find Moriarty and ARC2. The last thing it does is to load dataspace.php which then handles the browser request.
  • sample.htaccess — this is a sample .htaccess file for Apache webservers. It redirects all requests via index.php.
  • plain.tmpl.html — this is the default template used to render the HTML views. This can be overridden in the configuration.

Using the example script is simple: just copy index.php to the root directory of your linked data space. If you’re using Apache then you need to copy sample.htaccess into the same directory and rename it to .htaccess. Edit index.php so it refers to your store and your URIs and that’s it! You can see it in action with the default template on my own linked data space.

About Moriarty… Moriarty is a simple PHP library for accessing the Talis Platform. It follows the Platform API very closely and wraps up many common tasks into convenient classes while remaining very lightweight. It also provides some simple RDF classes that are based on the excellent ARC2 class library. Moriarty is being developed by small community of developers and is in continual beta, subject to a slow stream of updates. You can read more about Moriarty on the n² wiki or visit its Google Code project

Moriarty Progess Report

It’s been a while since I wrote about Moriarty, the PHP library I created for working with the Talis Platform. That’s not to say that there have been no changes: on the contrary, there have been lots of improvements and some major new areas of functionality. I’m going to summarise them in this post and then, time permitting, follow up with more detailed posts on particular areas.

  • Fresnel Selector Language — This is a major new addition to Moriarty. A new class called GraphPath has been added which implements almost all of the Fresnel Selector Language specification. I’ve been interested in RDF path languages for a long time and FSL now appears to be the strongest contender. Currently this is a stand-alone class, but after a few more cycles of testing it would be nice to add a convenience method to SimpleGraph to allow selection of resources using paths.
  • Zend-compatible caching — I did a substantial refactoring a while back to convert Moriarty’s HTTP caching implementation to be compatible with Zend’s cache interfaces. Whereas before you were limited to caching HTTP responses to disk, you can now supply any of Zend’s built-in cache classes to enable caching in databases, memcached and many other systems. You do this by creating an instance of HttpRequestFactory with your cache class and then pass the factory to the Store class.
  • JSON usageRelease 24 of the Talis Platform introduced RDF/JSON serialisation for describe and constuct SPARQL queries. Moriarty now requests this format where it and because the SimpleGraph class uses an RDF/JSON structure as its index often there is no result parsing involved at all.
  • OAI Service Support — OAIService is a new class that represents a store’s OAI-PMH Service. It provides simple access to the OAI service allowing all resources in a store to be listed.
  • Automated Builds — We have added Moriarty to an Hudson server which monitors the subversion repository and runs the unit test suite after every checkin. It’s not ideal because the server is not accessible to users outside of Talis (come on Google Code – we need Hudson support!). However it adds an extra level of confidence to checkins because test failures are emailed out to moriarty-dev@googlegroups.com which is open for anyone to join. A few times in the past we have run the unit tests locally and then forgotten to check in some critical dependency so the subversion trunk contains a broken build. Hudson will alert us to these kinds of errors much more quickly.
  • Extended Describes — The Sparql classes now accept an extra parameter to their describe methods that allows you to specify the type of description you want. By default you get the Platform’s default graph which is a list of triples that have the subject you specify (no bnodes remember!). Moriarty allows you to easily request other types such as symmetric bounded description (triples where the URI being described is subject or object), labelled bounded description (like the default description plus the addition of label properties for URIs in the description set) and symmetric labelled bounded description (a combination of the previous two). See the Bounded Description page on the n2 wiki for more information.
  • Richer Store Interface — Up until now Moriarty has used a object model that closely follows the Platform’s separation of services. That tended to make code using Moriarty quite verbose. We’re now gradually introucing convenience methods onto the Store class so common operations can be accessed with less code.

The moriarty Google Code project now has several committers although Keith and myself are still the most prolific. However, having multiple committers is one more step away from this being a personal project and towards it being community owned. Moriarty is being used in lots of small projects in and around Talis, but significantly it is also in the core of two of our most important products: Talis Prism and Talis Aspire. That’s great validation for Moriarty, although it brings a lot more responsibility in terms of quality of testing. I now consider Moriarty to be out of continual alpha and into continual beta!

About Moriarty… Moriarty is a simple PHP library for accessing the Talis Platform. It follows the Platform API very closely and wraps up many common tasks into convenient classes while remaining very lightweight. It also provides some simple RDF classes that are based on the excellent ARC2 class library. Moriarty is being developed by small community of developers and is in continual beta, subject to a slow stream of updates. You can read more about Moriarty on the n² wiki or visit its Google Code project

Moriarty Release 1.1

After some nudging from the Talis development team I tagged the current trunk of Moriarty as version 1.1:

http://moriarty.googlecode.com/svn/tags/release-1.1/

This is a stable release and should be backwards compatible with 1.0. The trunk continues to be the bleeding edge.

Moriarty Documentation

I started adding some API documentation to Moriarty using the excellent PHPDoctor. The documentation is in subversion but you can also view it online.

Moriarty Development List

I noticed that I was the only one getting notificiations of commits to Moriarty‘s subversion. I thought the best way to fix that was to create a Google group for moriarty and ensure the commit reports get sent there. So if you’re interested in keeping track of changes to Moriarty please sign up: moriarty-dev

Alternative to CURL in Moriarty

I just checked in a small update to moriarty that might solve a problem some people have experienced using curl. It appears that even though curl implemented support for HTTP digest way back in 2003 with version 7.10.6, it took several more releases to iron out the bugs. The version I develop with 7.18.0 (and the version installed on Talis application servers) works without issue, but many webhosts have much older versions. In fact my own webhost is still on 7.10.6 which means that digest authentication doesn’t work as expected. To date there has been no workaround. The latest change to Moriarty adds support for using httpclient written by Manuel Lemos. This is a complete HTTP client written in PHP. To use digest authentication you also need sasl which is also written by Manuel Lemos. Moriarty looks for those two classes and uses them if it finds them otherwise it falls back to using curl as before.

To use httpclient with Moriarty you just need to ensure that http_class and sasl_interact_class are loaded before using any HTTP actions. Adding lines like the following to your index.php (or somewhere similar) should do the trick:

    require_once '/path/to/moriarty/lib/httpclient/http.php';

    require_once '/path/to/moriarty/lib/sasl/sasl.php';

About Moriarty… Moriarty is a simple PHP library for accessing the Talis Platform. It follows the Platform API very closely and wraps up many common tasks into convenient classes while remaining very lightweight. It also provides some simple RDF classes that are based on the excellent ARC2 class library. Moriarty is primarily being developed by Ian Davis and is in continual alpha, subject to occasional rapid bursts of change. You can read more about Moriarty on the n² wiki or visit its Google Code project

Moriarty Now Hosted on Google Code

A couple of weeks ago I moved Moriarty from my playground area of the n² SVN repository to a new project at google code. This brings the advantage of neat issue tracking and code review capabilities as well as better management of contributors and collaborators. The new SVN repository is now http://moriarty.googlecode.com/svn/trunk/ (with an interactive view too). Just email me {at} iandavis.com if you’d like to be added to the project.

Batch Changesets ARC Plugin

Platform Release 12 included a very useful new feature: the ability to send more than one changeset in a single POST to your store.

To generate a batch changeset from 2 versions of an RDF graph, you can use an ARC plugin called Talis_ChangeSetBuilderPlugin.

To use it:


	  $args = array(
			'before' => $before, //can be rdf/xml, turtle, or an ARC simpleIndex array
			'after' => $after,  //can be rdf/xml, turtle, or an ARC simpleIndex array
		);
		$cs = ARC2::getComponent('Talis_ChangeSetBuilderPlugin', $args);
		$cs_response = $store->get_metabox()->apply_versioned_changeset($cs); 

The plugin also relies upon the IndexUtils Plugin. The easiest way to get them all set up is to change to your arc directory and do:


svn co http://n2.talis.com/svn/playground/kwijibo/PHP/arc/plugins/trunk/ plugins

Rollbacks in Moriarty

Editing resources in the metabox of Talis Platform stores is done with Changesets. If you choose to use the versioned changesets API, your changesets will be stored as data in the metabox as well.

The great practical benefit of doing this is you can then reverse previous ChangeSets to return a resource to its previous state. You can read about one way to reverse changesets on the wiki. You can also now create rollback changesets from Moriarty with the new Rollback class.

To use it:


define('MORIARTY_ARC_DIR', 'arc/');
require 'moriarty/store.class.php';
require 'moriarty/rollback.class.php';  

//create a store object
$store = new Store('http://api.talis.com/stores/my_store');  

//Instantiate the Rollback class with a sparql service object:
$sparql = $store->get_sparql_service();
$rollback = new Rollback($sparql);  

//Call the to_changeset method, with a changeset's uri as the argument
$HTTP_Response = $rollback->to_changeset('http://api.talis.com/stores/my_store/items/1200302910905#self');  

// the body of the response is the changeset you need to revert back to the
// state of the resource before the changeset that you have given the URI of  

if($HTTP_Response->is_success()){  

//submit changeset  

	$rollbackResponse =  $store->apply_versioned_changeset($HTTP_Response->body);  

	if($rollbackResponse->is_success()){
		//relax!
	}else{
		// throw an error
	}  

}