Background

I asked for a little project I could get my teeth into, Leigh suggested something very tasty. An analytics app, along the lines of Google Analytics or the (very impressive) open source Piwik. Basically tracking things like page visits, referers, outbound clicks and so on. The difference from the existing apps being taking advantage of semweb goodness, specifically a Talis Platform store as a backend.

What this required was something that would run in the browser when someone visited a given Web page and pass on relevant data to a server which would push that data into the store. A script discretely embedded on the page of interest picks up the activity and posts it to the server-side logging system. There wasn’t really a sensible choice other than to use Javascript client-side, and to keep things reasonably portable server-side I opted for PHP. The server-side processing is relatively straightforward (although I’m not actually capturing much yet), but the browser-server comms part turned out to be a real doozy.

It’s not difficult to call a HTTP server from inside Javascript wrapped in HTML loaded in a browser. The snag is that the security model common to popular browsers blocks access to server domains other than the one that originated the page containing the Javascript. I got some code running from http://hyperdata.org that nicely delivered some basic logging of visits to pages on http://hyperdata.org (including the Wiki I have there – though it took a while to find the right template…). Problems started when I tried the same script in pages hosted under http://danny.ayers.name. Browser no likey, wrapping the server call in a try...catch block and throwing up an alert(error) always revealed Exception… “Access to restricted URI denied” code: “1012″ – this is the same origin policy. What follows are the workarounds for this. Googling the titles here will provide a variety of sample code that implements the solutions. I’ve opted for Hidden Form, it being straightforward for my purposes and standards-friendly.

Cross-domain proxy

Conceptually the easiest, this approach uses a server-side pipeline that lives on the same domain as the delivered pages containing the Javascript. It essentially echos calls from the delivering server to the remote server that does the work. This didn’t seem a good choice for the analytics app as every end-user would require such a proxy on their own server.

• Pros: straightforward; independent of browser vagaries; spec friendly
• Cons: needed for every host delivering pages with embedded scripts (if all the servers involved are yours, this is probably a good choice)

Tag Overload Hacks

When a typical browser hits HTML tags <script> and <img> (any others?) it will quite happily do a HTTP GET on them, irrespective of domain. There’s been a fair bit of finesse applied around the use of <script> – notably the elegant but brain-boiling JSONP (JSON with Padding) which passes around scripts padded to be non-executable and involves callbacks. Somehow. I won’t comment further on this, except to say I understood it for about 5 minutes then lost it again when I went to make a coffee. I’m told jQuery will do something similar automagically if you choose datatype: "json" and method: "get".

The <img> approach has been around seemingly forever – it’s also known as a Web Bug. Usually you have a 1×1 pixel image in the page of interest (probably inserted dynamically through DOM calls), every time the page is loaded that image’s URI gets a GET. The trick for tracking is to append the image URI with a bunch of query parameters and have your server intercept the GET call. Apparently this is how Google Analytics does its stuff.

• Pros: good library support
• Cons: limited to GETs; rather an ugly hack

Flash Proxy

Most people suggested this when I was asking around Twitter and the jQuery mailing list. Turns out there’s a really convenient library that does all the hard work (Google “flXHR”). But I’m afraid I prefer to give Flash a miss when there are open standards available, so I didn’t investigate.

• Pros: easy (apparently) with library support
• Cons: uses proprietary stuff

Hidden Form

When I first saw references to this I overlooked it – it seemed to demand an iFrame and ugly hackery. But then (largely thanks to this discussion of cross-domain Ajax) I realised it was almost certainly the best bet for the analytics app. Essentially you dynamically push a <form> into the HTML DOM with your data as input values, then call a form.submit(). Most references to this I found did involve an iFrame to receive the HTTP response – necessary if you’re doing a mashup or something, but not if you only need to POST data off to the server. In this latter circumstance you need to get the server to return a 204 No Content status code, but that’s trivial in PHP (header('HTTP/1.1 204 No Content');), otherwise the browser will try to load the target URI material.

• Pros: supports and is very simple for POSTing to server; standards-friendly in this context
• Cons: gets uglier if you want a response

I’ve not properly doc’d my app code yet (and the functionality is a very long way from complete, let alone tidied up), but you can find it all via my latest Wiki – there’s an example of the Javascript in test.html (just before the closing </body> tag). I’ve only tested it on Firefox so far, but I reckon there’s a good chance of the LazyWeb giving me solutions to any cross-browser issues.

Many thanks for all the helpful suggestions: from this thread on the jQuery mailing list and Twitterers @rjw @flensed @gridinoc @weblivz @JeniT @jQueryHowto.

I’d love to hear of any other solutions to cross-domain Ajax, please drop in comments, mail me or tweet me.

Over the past few years I’ve tinkered with a number of different implementations of an AJAX client library for SPARQL. Before a standard format for SPARQL JSON results was created, this involved having to jump through the extra hoops of parsing the XML format. But things are much easier now, especially when the JSON support is extended to include the results of CONSTRUCT and DESCRIBE queries.

My personal favourite SPARQL client library though is the one produced by Lee Feigenbaum, Elias Torres, and Wing Yung as part of their work on the SPARQL Calendar Demo.

While the sparql.js library only supports JSON it does have a few convenience features which I like, including global PREFIX bindings and some functions for automatically processing the JSON results to produce some simpler javascript objects (e.g. arrays and hashes) that simplify some scripting tasks and make code more readable.

Using this on the Platform is quite straight-forward, as you can upload this library, and any other related Javascript files directly into the Contentbox of your store. This not only avoids any cross-domain issues, but also means that you can deploy simple AJAX applications directly from a store.

I’ve put together a super simple demo that uses the NASA spaceflight data. The source code is here, and I’ve uploaded the two files into the n2-examples store contentbox, so you can play with the running application.

The demo simply fetches the name, homepage, description and launch date for every spacecraft launched in a particular year, also retrieving a link to a photo if there’s one available. The results are dropped into an HTML table for viewing.

The code is well commented so rather than repeat that here, you can look through the Javascript file that does the actual interaction. I’ve used JQuery to help with the DOM manipulation, etc. This is delivered through the Google JQuery CDN rather than the Platform. But the rest of the application is served directly from the Platform.

A rather easy and trivial example, but sometimes its useful to reiterate the basics. And if you want to incorporate the NASA spaceflight data in your own mashups, then you can do so easily by simple using the version of sparql.js in the space data store.

In my view, SPARQL + JSON + scripting languages like JS and Ruby hit a nice sweet spot for working with RDF, especially with the ability to bring together data from multiple sources using a single standard API.

Note: Keith Alexander has written up some of his own experiments with playing with JQuery against the platform here and here. His JQuery plugin provides some additional Platform specific functionality.

We will use the jQuery.Talis plugin to create a simple html+js interface to a talis store.

the Talis plugin is a small wrapper that simplifies retrieving json from the platform remotely (via jsonp). It allows you to query the platform, and specify callback functions for dealing with the retrieved data.

We’ll have a text box to type a search string into; this will retrieve results (of matching resource descriptions) from the platform store, and display them in a list of links. Clicking on the links will display the resource description.

1. The HTML:

We are going to need three elements for this:

2 The Javascript

At the command line, switch to the directory you saved your HTML file in, and do:

    svn co http://n2.talis.com/svn/playground/kwijibo/js/Talis.jQuery.plugin/trunk/ js/

Now we link to the javascript files from the bottom of the <body> of our html page:

<script type="text/javascript" charset="utf-8" src="js/jquery.js" mce_src="js/jquery.js"></script>
<script type="text/javascript" charset="utf-8" src="js/Talis.jQuery.js" mce_src="js/Talis.jQuery.js"></script>
<script type="text/javascript" charset="utf-8" src="js/jsRDF.jquery.js" mce_src="js/jsRDF.jquery.js"></script>

(jsRDF.jQuery.js is just a small, nascent library for manipulating RDF/JSON )

Now open another script tag, and we’ll write some javascript to connect our html with the platform:

First, we’ll declare some variables we’ll want to use:

var RSS_ITEM = 'http://purl.org/rss/1.0/item';
var RDF_TYPE = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type';
var MY_STORE = 'schema-cache';

For this tutorial, I’m using the schema-cache store, which contains many RDF and OWL vocabularies.

Now, what we want is to query the platform when we type in the text box, so:

$("#search").keyup(function(){
    var query = $("search").val();
    $.Talis.Store(MY_STORE).items(query, function(json){
        /*  we do something with the json data from the platform in here ... */
    });
});

What’s happening here, is we are taking the text that has been typed in the textbox (#search), and querying the items service of our store with it. The second parameter of the Store.items method is a callback function, in which you can specify what to do with the data when it is retrieved.

The platform items service returns the results in an RSS feed, which the jQuery.Talis plugin fetches for us in rdf/json, however, for this, we only want the items of the feed, not the RSS feed resource itself, so we need to filter in only the resources that have rss:Item as a value of their rdf:type property:

    var RDF = $.jsRDF(json);
    var rss_items = RDF.filter({p:RDF_TYPE, o:{value:RSS_ITEM}});

Here, we are loading the data into a jsRDF object, which has methods for manipulating it. We’re using the filter method to select the resources that have an rdf:type of rss:Item. Now we want to render them in the page inside our #results list:

$.each(rss_items, function(uri, properties){
    $("#results").append('<li><a href="'+uri+'" mce_href="'+uri+'">'+RDF.get_label(uri)+'</a></li>');
});

OK. We want clicking on those links to show the resource description, so we’ll define a function for retrieving that description from the store, and rendering it, then we’ll bind it the onclick event of the links in the results list:

function browse(uri){
    var uri = this.href;
    $.Talis.Store(MY_STORE).lcbd(uri, function(data){   

        var RDF = new $.jsRDF(data);
        $('#description').html( RDF.to_html(uri) );
        $('#description dd a').click(browse);
    });
    return false;
}

We get the URI of the resource from the @href attribute (which we set when we were rendering the search results), then we call the lcbd method on our store (LCBD is short for labelled concise bounded description, and returns the properties of the resource, and labels for all the resources our description references). Again, we use the $.jsRDF object to render the description as html (it uses a definition list for rendering the properties of the resource).

After we’ve rendered the description in the #description div, we also bind the click event on the links to the resource’s properties to the browse function, so that clicking on those links will retrieve and render the resources being linked to.

And that’s pretty much it.

jQuery.Talis is a plugin for the popular javascript library jQuery. It acts as a wrapper around the talis convert service, for retrieving json, through jsonp, from the Platform.

You can read about it on the n2 wiki and download it from the n2 svn.

You can use it like this:

$.Talis.Store('schema-cache').sparql('DESCRIBE ',
      function(data){
        $("#Person h1").html(data['http://xmlns.com/foaf/0.1/Person']['http://www.w3.org/2000/01/rdf-schema#'][0].value);
});

(This would fetch a description of the foaf:Person class from the http://api.talis.com/stores/schema-cache store, and insert the rdfs:label into the DOM.)

I don’t want to declare this stable yet, but it is usable in it’s current form (I use it in the SIOC Comments Widget). The size currently comes in at ~4k without any compression or minification. So far, it’s only been tested in Firefox, Safari and Opera, so reports of cross-browser problems, and any other bugs, would be appreciated.

In my last post, I said, a little tentatively, that it should be possible to build entirely client-side semweb applications using the Convert service. I was tentative because it seemed like it ought to work, but I hadn’t actually tried it.

So I tried it, and yes, it works!

Yesterday, I put together a widget that adds comments functionality to any page, or page section. Just add this markup to the section you want to have comments on:

<div class="sioc-has_reply"></div>
<script type="text/javascript"
charset="utf-8"
src="http://n2.talis.com/svn/playground/kwijibo/javascript/sioc-comments/bundle.js">
</script>

Ok caveat – I only wrote it yesterday afternoon, it worked for me, on my macbook, but haven’t tested it on IE of any flavour, and tuukah, in #sioc (who was kind enough to try it for me) had problems with firefox on linux.

Anyway, what it does is this: You add the markup, and the page gets a comments form, with fields for your URI, and your comment. When you type in your URI and move to the next field, it will try to dereference your URI and retrieve your foaf:name.
When you submit the form, it will save your comment as RDF to a platform store. The comment will be retrieved from the platform, and displayed on the page. Next to your name in the comments list, is a button called ‘Activity’, which will retrieve a list of all the other posts you have commented on using the widget – these could be anywhere on the web.

I put up a test page you can try it out on.

Now admittedly, it doesn’t look very pretty (assuming you get it to work), and some blatently lacking features are some kind of identity authentication process, and a way for page owners to remove spammers and trolls from their comments. These would take a little bit of custom server-side code to implement.

That asides, as a proof of concept, I think it’s pretty cool (if I do say so myself!) – it takes an existing style of application (a js comments widget), and shows that by adding a URI, you get all kinds of other possibilities. You could have another widget that analysed the FOAF profile’s of the commenters through their URIs, and visualised how incestuous your readership is (eg: do they all know each other? do they all comment on the same blogs?).

It also provokes all kinds of other interesting ideas – there are lots of other sorts of widgets that could take this idea of user input distributed across the web, and working locally on their own pages, but using the network to build up a value greater than the sum of its parts. And here, RDFa starts to look like a good way of exposing the retrieved RDF in HTML for the browser to potentially do more stuff with. Perhaps not just exposing the RDF, but also the SPARQL endpoint it pulls it from.

Another feature of the experimental Convert service is the ability to pull RDF (extracted from RDF/XML, turtle, RDF/JSON, and HTML) into an editing interface – either form fields, or editing free text as Turtle, edit the data, transform it (options include describing the edit as a ChangeSet, reifying, or dereifying the data), convert it, and POST or PUT the results to any URI.

This might not seem so exciting if you are already quite happy doing this on the commandline with Vim and cURL, but what is potentially interesting about it is the syntax used in the name attributes of the form widgets to roundtrip the data from the web, through the HTML forms ( you can read about the forms syntax on the n2 wiki ).

Javascript and JSONP

What this means, is that the Convert service can be used as a proxy for purely client side javascript applications. You can retrieve RDF data form anywhere on the web by requesting a JSONP conversion of it from the service.

JSONP

The JSONP service allows you to specify a callback function, and it will return the data wrapped in a call to that function.

So you define a function called ,eg, my_callback which accepts the json data object as the first parameter, and then create a script tag pointing to the JSON data, with a url parameter of callback=my_callback. The browser will then load the remote javascript into your page, which will call your my_callback function, passing it the data you requested.

RDF in HTML forms

You can then load the data into HTML form widgets, using the same forms syntax as the Convert service’s editor page for the @name attribute, and point the @action attribute of the form at the Convert service. Pass in the appropriate form values describing how you want the data transformed and converted, and where you want to POST or PUT it to. Then when the form is submitted, the POST array will be transformed and converted into the format you chose, and forwarded on to the URI you chose.

The upshot of which, I think, is that you can write pure client-side applications that read, write, and edit data across the web.

It’s kind of like that formmail.pl script, for RDF

Lately I’ve been working on an experimental Convert service. The idea is much like dajobe’s triplr or Simile’s babel – accept a variety of semantic formats as input, and make them available in other flavours as output.

RDF -> RDF

The service accepts HTML (preferrably with eRDF, RDF, or microformats), RDF/XML, turtle, or RDF/JSON as input, outputting to a variety of RDF serialisations. For the parsing of most of these RDF formats, the service uses Benjamin Nowack’s excellent ARC library for PHP.

SPARQL/XML and Facet/XML

The service also accepts SPARQL/XML and the XML from the Talis Platform Facet service, transforming to either JSON, JSONP, or HTML.

Doing the conversions is a PHP library, available in the n2 SVN repository