Background

I asked for a little project I could get my teeth into, Leigh suggested something very tasty. An analytics app, along the lines of Google Analytics or the (very impressive) open source Piwik. Basically tracking things like page visits, referers, outbound clicks and so on. The difference from the existing apps being taking advantage of semweb goodness, specifically a Talis Platform store as a backend.

What this required was something that would run in the browser when someone visited a given Web page and pass on relevant data to a server which would push that data into the store. A script discretely embedded on the page of interest picks up the activity and posts it to the server-side logging system. There wasn’t really a sensible choice other than to use Javascript client-side, and to keep things reasonably portable server-side I opted for PHP. The server-side processing is relatively straightforward (although I’m not actually capturing much yet), but the browser-server comms part turned out to be a real doozy.

It’s not difficult to call a HTTP server from inside Javascript wrapped in HTML loaded in a browser. The snag is that the security model common to popular browsers blocks access to server domains other than the one that originated the page containing the Javascript. I got some code running from http://hyperdata.org that nicely delivered some basic logging of visits to pages on http://hyperdata.org (including the Wiki I have there – though it took a while to find the right template…). Problems started when I tried the same script in pages hosted under http://danny.ayers.name. Browser no likey, wrapping the server call in a try...catch block and throwing up an alert(error) always revealed Exception… “Access to restricted URI denied” code: “1012″ – this is the same origin policy. What follows are the workarounds for this. Googling the titles here will provide a variety of sample code that implements the solutions. I’ve opted for Hidden Form, it being straightforward for my purposes and standards-friendly.

Cross-domain proxy

Conceptually the easiest, this approach uses a server-side pipeline that lives on the same domain as the delivered pages containing the Javascript. It essentially echos calls from the delivering server to the remote server that does the work. This didn’t seem a good choice for the analytics app as every end-user would require such a proxy on their own server.

• Pros: straightforward; independent of browser vagaries; spec friendly
• Cons: needed for every host delivering pages with embedded scripts (if all the servers involved are yours, this is probably a good choice)

Tag Overload Hacks

When a typical browser hits HTML tags <script> and <img> (any others?) it will quite happily do a HTTP GET on them, irrespective of domain. There’s been a fair bit of finesse applied around the use of <script> – notably the elegant but brain-boiling JSONP (JSON with Padding) which passes around scripts padded to be non-executable and involves callbacks. Somehow. I won’t comment further on this, except to say I understood it for about 5 minutes then lost it again when I went to make a coffee. I’m told jQuery will do something similar automagically if you choose datatype: "json" and method: "get".

The <img> approach has been around seemingly forever – it’s also known as a Web Bug. Usually you have a 1×1 pixel image in the page of interest (probably inserted dynamically through DOM calls), every time the page is loaded that image’s URI gets a GET. The trick for tracking is to append the image URI with a bunch of query parameters and have your server intercept the GET call. Apparently this is how Google Analytics does its stuff.

• Pros: good library support
• Cons: limited to GETs; rather an ugly hack

Flash Proxy

Most people suggested this when I was asking around Twitter and the jQuery mailing list. Turns out there’s a really convenient library that does all the hard work (Google “flXHR”). But I’m afraid I prefer to give Flash a miss when there are open standards available, so I didn’t investigate.

• Pros: easy (apparently) with library support
• Cons: uses proprietary stuff

Hidden Form

When I first saw references to this I overlooked it – it seemed to demand an iFrame and ugly hackery. But then (largely thanks to this discussion of cross-domain Ajax) I realised it was almost certainly the best bet for the analytics app. Essentially you dynamically push a <form> into the HTML DOM with your data as input values, then call a form.submit(). Most references to this I found did involve an iFrame to receive the HTTP response – necessary if you’re doing a mashup or something, but not if you only need to POST data off to the server. In this latter circumstance you need to get the server to return a 204 No Content status code, but that’s trivial in PHP (header('HTTP/1.1 204 No Content');), otherwise the browser will try to load the target URI material.

• Pros: supports and is very simple for POSTing to server; standards-friendly in this context
• Cons: gets uglier if you want a response

I’ve not properly doc’d my app code yet (and the functionality is a very long way from complete, let alone tidied up), but you can find it all via my latest Wiki – there’s an example of the Javascript in test.html (just before the closing </body> tag). I’ve only tested it on Firefox so far, but I reckon there’s a good chance of the LazyWeb giving me solutions to any cross-browser issues.

Many thanks for all the helpful suggestions: from this thread on the jQuery mailing list and Twitterers @rjw @flensed @gridinoc @weblivz @JeniT @jQueryHowto.

I’d love to hear of any other solutions to cross-domain Ajax, please drop in comments, mail me or tweet me.

It’s been a while since I wrote about Moriarty, the PHP library I created for working with the Talis Platform. That’s not to say that there have been no changes: on the contrary, there have been lots of improvements and some major new areas of functionality. I’m going to summarise them in this post and then, time permitting, follow up with more detailed posts on particular areas.

• Fresnel Selector Language — This is a major new addition to Moriarty. A new class called GraphPath has been added which implements almost all of the Fresnel Selector Language specification. I’ve been interested in RDF path languages for a long time and FSL now appears to be the strongest contender. Currently this is a stand-alone class, but after a few more cycles of testing it would be nice to add a convenience method to SimpleGraph to allow selection of resources using paths.
• Zend-compatible caching — I did a substantial refactoring a while back to convert Moriarty’s HTTP caching implementation to be compatible with Zend’s cache interfaces. Whereas before you were limited to caching HTTP responses to disk, you can now supply any of Zend’s built-in cache classes to enable caching in databases, memcached and many other systems. You do this by creating an instance of HttpRequestFactory with your cache class and then pass the factory to the Store class.
• JSON usage — Release 24 of the Talis Platform introduced RDF/JSON serialisation for describe and constuct SPARQL queries. Moriarty now requests this format where it and because the SimpleGraph class uses an RDF/JSON structure as its index often there is no result parsing involved at all.
• OAI Service Support — OAIService is a new class that represents a store’s OAI-PMH Service. It provides simple access to the OAI service allowing all resources in a store to be listed.
• Automated Builds — We have added Moriarty to an Hudson server which monitors the subversion repository and runs the unit test suite after every checkin. It’s not ideal because the server is not accessible to users outside of Talis (come on Google Code – we need Hudson support!). However it adds an extra level of confidence to checkins because test failures are emailed out to moriarty-dev@googlegroups.com which is open for anyone to join. A few times in the past we have run the unit tests locally and then forgotten to check in some critical dependency so the subversion trunk contains a broken build. Hudson will alert us to these kinds of errors much more quickly.
• Extended Describes — The Sparql classes now accept an extra parameter to their describe methods that allows you to specify the type of description you want. By default you get the Platform’s default graph which is a list of triples that have the subject you specify (no bnodes remember!). Moriarty allows you to easily request other types such as symmetric bounded description (triples where the URI being described is subject or object), labelled bounded description (like the default description plus the addition of label properties for URIs in the description set) and symmetric labelled bounded description (a combination of the previous two). See the Bounded Description page on the n2 wiki for more information.
• Richer Store Interface — Up until now Moriarty has used a object model that closely follows the Platform’s separation of services. That tended to make code using Moriarty quite verbose. We’re now gradually introucing convenience methods onto the Store class so common operations can be accessed with less code.

The moriarty Google Code project now has several committers although Keith and myself are still the most prolific. However, having multiple committers is one more step away from this being a personal project and towards it being community owned. Moriarty is being used in lots of small projects in and around Talis, but significantly it is also in the core of two of our most important products: Talis Prism and Talis Aspire. That’s great validation for Moriarty, although it brings a lot more responsibility in terms of quality of testing. I now consider Moriarty to be out of continual alpha and into continual beta!

About Moriarty… Moriarty is a simple PHP library for accessing the Talis Platform. It follows the Platform API very closely and wraps up many common tasks into convenient classes while remaining very lightweight. It also provides some simple RDF classes that are based on the excellent ARC2 class library. Moriarty is being developed by small community of developers and is in continual beta, subject to a slow stream of updates. You can read more about Moriarty on the n² wiki or visit its Google Code project

I’ve published the code behind the Talis Convert Service (production release at stable URL coming soon) as an open source project on Google Code, called Metamorph .

Metamorph is a service aimed at semantic web developers. It is much like triplr, babel, swignition and any23 (please leave a comment pointing to any other similar services).

You give it a(n http) URI, an (optional) input format, and an output format, and it will fetch the document from the web, and convert it into the output format.

Understood input values include:

• Semantic HTML (RDFa, eRDF, microformats, POSH)
• RDF (XML, Turtle, JSON)
• SPARQL-XML
• Facet XML (the response format of the facets service available on all platform stores)

Output for all input formats can be:

• JSON
• JSONP
• HTML

If the input is some form of RDF, you can also ask for:

• RDF (XML, Turtle, JSON, – and the default HTML is rendered as RDFa)
• RSS 1.0
• TriX
• Exhibit (web page, JSON, JSONP)

In addition, if the input is an RDF format, you can specify multiple data URIs, and the results will be merged in the output document. For instance, this conversion merges data from two of my homepages, and a Turtle file.

I’m thinking about removing the TriX output, as I’m not sure it would be used by anyone – the reason I didn’t bother to write a parser for it was because I haven’t seen any data published as TriX in the first place.

I welcome any input on what else would be useful from this web service. I suspect that more output options, while fairly easy to add, would not be very useful. More input options may be useful, but perhaps not significantly so.

I suspect what might be more useful, and more likely to distinguish this from similar RDF converter services, are graph transformation services, which might include:

• Diffs
• Intersects
• Smushing
• Augmenting on property and class type URIs with labels and comments, perhaps retrieved from SchemaCache

Metamorph is coded in PHP, and uses ARC for parsing RDF and HTML, and serialising RDF/XML and Turtle.

Please use the issue tracker for raising any bugs or feature requests.

After some nudging from the Talis development team I tagged the current trunk of Moriarty as version 1.1:

http://moriarty.googlecode.com/svn/tags/release-1.1/

This is a stable release and should be backwards compatible with 1.0. The trunk continues to be the bleeding edge.

I started adding some API documentation to Moriarty using the excellent PHPDoctor. The documentation is in subversion but you can also view it online.

I noticed that I was the only one getting notificiations of commits to Moriarty‘s subversion. I thought the best way to fix that was to create a Google group for moriarty and ensure the commit reports get sent there. So if you’re interested in keeping track of changes to Moriarty please sign up: moriarty-dev

I just checked in a small update to moriarty that might solve a problem some people have experienced using curl. It appears that even though curl implemented support for HTTP digest way back in 2003 with version 7.10.6, it took several more releases to iron out the bugs. The version I develop with 7.18.0 (and the version installed on Talis application servers) works without issue, but many webhosts have much older versions. In fact my own webhost is still on 7.10.6 which means that digest authentication doesn’t work as expected. To date there has been no workaround. The latest change to Moriarty adds support for using httpclient written by Manuel Lemos. This is a complete HTTP client written in PHP. To use digest authentication you also need sasl which is also written by Manuel Lemos. Moriarty looks for those two classes and uses them if it finds them otherwise it falls back to using curl as before.

To use httpclient with Moriarty you just need to ensure that http_class and sasl_interact_class are loaded before using any HTTP actions. Adding lines like the following to your index.php (or somewhere similar) should do the trick:

    require_once '/path/to/moriarty/lib/httpclient/http.php';

    require_once '/path/to/moriarty/lib/sasl/sasl.php';

About Moriarty… Moriarty is a simple PHP library for accessing the Talis Platform. It follows the Platform API very closely and wraps up many common tasks into convenient classes while remaining very lightweight. It also provides some simple RDF classes that are based on the excellent ARC2 class library. Moriarty is primarily being developed by Ian Davis and is in continual alpha, subject to occasional rapid bursts of change. You can read more about Moriarty on the n² wiki or visit its Google Code project

A couple of weeks ago I moved Moriarty from my playground area of the n² SVN repository to a new project at google code. This brings the advantage of neat issue tracking and code review capabilities as well as better management of contributors and collaborators. The new SVN repository is now http://moriarty.googlecode.com/svn/trunk/ (with an interactive view too). Just email me {at} iandavis.com if you’d like to be added to the project.

Platform Release 12 included a very useful new feature: the ability to send more than one changeset in a single POST to your store.

To generate a batch changeset from 2 versions of an RDF graph, you can use an ARC plugin called Talis_ChangeSetBuilderPlugin.

To use it:

	  $args = array(
			'before' => $before, //can be rdf/xml, turtle, or an ARC simpleIndex array
			'after' => $after,  //can be rdf/xml, turtle, or an ARC simpleIndex array
		);
		$cs = ARC2::getComponent('Talis_ChangeSetBuilderPlugin', $args);
		$cs_response = $store->get_metabox()->apply_versioned_changeset($cs); 

The plugin also relies upon the IndexUtils Plugin. The easiest way to get them all set up is to change to your arc directory and do:

svn co http://n2.talis.com/svn/playground/kwijibo/PHP/arc/plugins/trunk/ plugins

Editing resources in the metabox of Talis Platform stores is done with Changesets. If you choose to use the versioned changesets API, your changesets will be stored as data in the metabox as well.

The great practical benefit of doing this is you can then reverse previous ChangeSets to return a resource to its previous state. You can read about one way to reverse changesets on the wiki. You can also now create rollback changesets from Moriarty with the new Rollback class.

To use it:

define('MORIARTY_ARC_DIR', 'arc/');
require 'moriarty/store.class.php';
require 'moriarty/rollback.class.php';  

//create a store object
$store = new Store('http://api.talis.com/stores/my_store');  

//Instantiate the Rollback class with a sparql service object:
$sparql = $store->get_sparql_service();
$rollback = new Rollback($sparql);  

//Call the to_changeset method, with a changeset's uri as the argument
$HTTP_Response = $rollback->to_changeset('http://api.talis.com/stores/my_store/items/1200302910905#self');  

// the body of the response is the changeset you need to revert back to the
// state of the resource before the changeset that you have given the URI of  

if($HTTP_Response->is_success()){  

//submit changeset  

	$rollbackResponse =  $store->apply_versioned_changeset($HTTP_Response->body);  

	if($rollbackResponse->is_success()){
		//relax!
	}else{
		// throw an error
	}  

}