Background

I asked for a little project I could get my teeth into, Leigh suggested something very tasty. An analytics app, along the lines of Google Analytics or the (very impressive) open source Piwik. Basically tracking things like page visits, referers, outbound clicks and so on. The difference from the existing apps being taking advantage of semweb goodness, specifically a Talis Platform store as a backend.

What this required was something that would run in the browser when someone visited a given Web page and pass on relevant data to a server which would push that data into the store. A script discretely embedded on the page of interest picks up the activity and posts it to the server-side logging system. There wasn’t really a sensible choice other than to use Javascript client-side, and to keep things reasonably portable server-side I opted for PHP. The server-side processing is relatively straightforward (although I’m not actually capturing much yet), but the browser-server comms part turned out to be a real doozy.

It’s not difficult to call a HTTP server from inside Javascript wrapped in HTML loaded in a browser. The snag is that the security model common to popular browsers blocks access to server domains other than the one that originated the page containing the Javascript. I got some code running from http://hyperdata.org that nicely delivered some basic logging of visits to pages on http://hyperdata.org (including the Wiki I have there – though it took a while to find the right template…). Problems started when I tried the same script in pages hosted under http://danny.ayers.name. Browser no likey, wrapping the server call in a try...catch block and throwing up an alert(error) always revealed Exception… “Access to restricted URI denied” code: “1012″ – this is the same origin policy. What follows are the workarounds for this. Googling the titles here will provide a variety of sample code that implements the solutions. I’ve opted for Hidden Form, it being straightforward for my purposes and standards-friendly.

Cross-domain proxy

Conceptually the easiest, this approach uses a server-side pipeline that lives on the same domain as the delivered pages containing the Javascript. It essentially echos calls from the delivering server to the remote server that does the work. This didn’t seem a good choice for the analytics app as every end-user would require such a proxy on their own server.

• Pros: straightforward; independent of browser vagaries; spec friendly
• Cons: needed for every host delivering pages with embedded scripts (if all the servers involved are yours, this is probably a good choice)

Tag Overload Hacks

When a typical browser hits HTML tags <script> and <img> (any others?) it will quite happily do a HTTP GET on them, irrespective of domain. There’s been a fair bit of finesse applied around the use of <script> – notably the elegant but brain-boiling JSONP (JSON with Padding) which passes around scripts padded to be non-executable and involves callbacks. Somehow. I won’t comment further on this, except to say I understood it for about 5 minutes then lost it again when I went to make a coffee. I’m told jQuery will do something similar automagically if you choose datatype: "json" and method: "get".

The <img> approach has been around seemingly forever – it’s also known as a Web Bug. Usually you have a 1×1 pixel image in the page of interest (probably inserted dynamically through DOM calls), every time the page is loaded that image’s URI gets a GET. The trick for tracking is to append the image URI with a bunch of query parameters and have your server intercept the GET call. Apparently this is how Google Analytics does its stuff.

• Pros: good library support
• Cons: limited to GETs; rather an ugly hack

Flash Proxy

Most people suggested this when I was asking around Twitter and the jQuery mailing list. Turns out there’s a really convenient library that does all the hard work (Google “flXHR”). But I’m afraid I prefer to give Flash a miss when there are open standards available, so I didn’t investigate.

• Pros: easy (apparently) with library support
• Cons: uses proprietary stuff

Hidden Form

When I first saw references to this I overlooked it – it seemed to demand an iFrame and ugly hackery. But then (largely thanks to this discussion of cross-domain Ajax) I realised it was almost certainly the best bet for the analytics app. Essentially you dynamically push a <form> into the HTML DOM with your data as input values, then call a form.submit(). Most references to this I found did involve an iFrame to receive the HTTP response – necessary if you’re doing a mashup or something, but not if you only need to POST data off to the server. In this latter circumstance you need to get the server to return a 204 No Content status code, but that’s trivial in PHP (header('HTTP/1.1 204 No Content');), otherwise the browser will try to load the target URI material.

• Pros: supports and is very simple for POSTing to server; standards-friendly in this context
• Cons: gets uglier if you want a response

I’ve not properly doc’d my app code yet (and the functionality is a very long way from complete, let alone tidied up), but you can find it all via my latest Wiki – there’s an example of the Javascript in test.html (just before the closing </body> tag). I’ve only tested it on Firefox so far, but I reckon there’s a good chance of the LazyWeb giving me solutions to any cross-browser issues.

Many thanks for all the helpful suggestions: from this thread on the jQuery mailing list and Twitterers @rjw @flensed @gridinoc @weblivz @JeniT @jQueryHowto.

I’d love to hear of any other solutions to cross-domain Ajax, please drop in comments, mail me or tweet me.

I just checked in a small update to moriarty that might solve a problem some people have experienced using curl. It appears that even though curl implemented support for HTTP digest way back in 2003 with version 7.10.6, it took several more releases to iron out the bugs. The version I develop with 7.18.0 (and the version installed on Talis application servers) works without issue, but many webhosts have much older versions. In fact my own webhost is still on 7.10.6 which means that digest authentication doesn’t work as expected. To date there has been no workaround. The latest change to Moriarty adds support for using httpclient written by Manuel Lemos. This is a complete HTTP client written in PHP. To use digest authentication you also need sasl which is also written by Manuel Lemos. Moriarty looks for those two classes and uses them if it finds them otherwise it falls back to using curl as before.

To use httpclient with Moriarty you just need to ensure that http_class and sasl_interact_class are loaded before using any HTTP actions. Adding lines like the following to your index.php (or somewhere similar) should do the trick:

    require_once '/path/to/moriarty/lib/httpclient/http.php';

    require_once '/path/to/moriarty/lib/sasl/sasl.php';

About Moriarty… Moriarty is a simple PHP library for accessing the Talis Platform. It follows the Platform API very closely and wraps up many common tasks into convenient classes while remaining very lightweight. It also provides some simple RDF classes that are based on the excellent ARC2 class library. Moriarty is primarily being developed by Ian Davis and is in continual alpha, subject to occasional rapid bursts of change. You can read more about Moriarty on the n² wiki or visit its Google Code project

Another day, another incremental improvement to Moriarty (svn revision 490)! After my last set of changes I thought I’d better hurry up and add the copy_to function to the FieldPredicateMap too. You can now clone Field/Predicate Maps from one store to another:

  $fp = new FieldPredicateMap("http://api.talis.com/stores/mystore/config/fpmaps/1");
  $response = $fp->get_from_network();
  if ( $response->is_success() ) {
    $new_fp = $fp->copy_to("http://api.talis.com/stores/otherstore/config/fpmaps/1");
    $new_fp->put_to_network();
  }

I then set about thinking through my plan for adding HTTP caching support to Moriarty. I want this to work automatically and transparently, taking advantage of conditional GETs on the Platform. I’ll let it be switched off by defining a constant but I want it to be there by default so the developer gets the benefit without any effort.

I stubbed out some initial ideas for the HttpCache class on the train this morning. Then at lunchtime today, Danny pinged me on IRC wondering why Moriarty didn’t have SPARQL ASK support. “Not by design”, I said, “more by lack of time. But it should be easy to add, give me 15 minutes”. Then I promptly went into a series of meetings that ate the rest of my day. In the end the code did only take 15 minutes, but I finished it 11 hours later than I expected. Hopefully Danny didn’t spend all that time waiting for me to respond on IRC

You can perform an ASK query on a store like this:

  $store = new Store("http://api.talis.com/stores/mystore");
  $sparql = $store->get_sparql_service();
  $response = $sparql->ask( "ASK WHERE {?s a .}" );
  if ($response->is_success()) {
    $result = $sparql->parse_ask_results( $response->body);
  }

Enjoy, Danny!

About Moriarty… Moriarty is a simple PHP library for accessing the Talis Platform. It follows the Platform API very closely and wraps ups many common tasks into convenient classes while remaining very lightweight. It also provides some simple RDF classes that are based on the excellent ARC2 class library. Moriarty is primarily being developed by Ian Davis and is in continual alpha, subject to occasional rapid bursts of change. You can read more about Moriarty on the n² wiki and get its source from the n² subversion repository