Comments on: Data Migration using SPARQL and Changesets http://blogs.talis.com/n2/archives/659 All about developing with the Talis Platform Wed, 21 Dec 2011 19:51:10 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: kwijibo http://blogs.talis.com/n2/archives/659/comment-page-1#comment-1025 kwijibo Fri, 10 Jul 2009 10:53:30 +0000 http://blogs.talis.com/n2/?p=659#comment-1025 Interesting points. Rewriting the incoming queries could provide some backwards compatibility - and I guess, if the query is a CONSTRUCT or a DESCRIBE, you ought to also rewrite the RDF on the way out again, otherwise the client code could still break. But maybe this should be for special cases, where there are specific breakages you want to avoid while still migrating. I'd side in general with the view that the client is responsible for coping with the data it gets, and the server is responsible for a stable API, though not necessarily stable data. But it's pretty likely there will also be special cases when being backwards-compatible data-wise is also important, and rewriting queries and data is a good solution (and what about when the data is retrieved by dereffing LOD, not SPARQLed? What should happen then?). You could also leave the old triples in the store instead of rewriting, though there are potential drawbacks with that bloating the size of your store and response document sizes (you could possibly filter out the old triples from the RDF you are returning unless they are specified in the SPARQL query). In some cases having both old and new triples might make your data wrong or inconsistent. The backwards-compatible store idea could be OK if the data is fairly static - you could sniff the SPARQL query for old triple patterns and redirect to the appropriate endpoint maybe? There are lots of different ways you might change your data too - it could be a typo or switching vocab terms, or it could be a radically different modeling; in which case, it would be harder for the client to anticipate the change (and might even require more extensive rewriting of the client application). Assuming that it is the client's responsibility to adjust to changes in data pulled down from the wild web, there are a couple of things that I think might help: 1. A way of bundling up equivalent terms / graph patterns for specific situations. Maybe some collections of owl:sameAs and rdfs:subPropertyOf statements and the like, and a bit of reasoning, could work, but maybe a better solution would be something like profiles, where terms/graph patterns are stated to be functionally substitutable for a given task, eg: Here are a bunch of predicate URIs you can look through if you want to display something as a label, or here are a bunch of things you can try if you want to show an image associated with a (non-IR) resource; or (trickier) here's a bunch of patterns you can try if you want to find tags/taggings. These profiles (for want of a better word) could be kept up to date and served up from somewhere, and the client application could retrieve and cache them - so then the client could be vocabulary agnostic, and cope with evolutions in data modeling, provided the profiles are kept up to date. 2. Ways for the server to hint to clients when a data migration has taken place - and what to do about it. I suppose the lightest weight option is to include old and new triples in the response for a while and hope the client picks up on it. The server could include extra metadata in the response document, or in it's own (RDF, of course) service description, or dataset description (voiD provides terms to say what vocabularies a dataset is using, for example). Interesting points. Rewriting the incoming queries could provide some backwards compatibility – and I guess, if the query is a CONSTRUCT or a DESCRIBE, you ought to also rewrite the RDF on the way out again, otherwise the client code could still break. But maybe this should be for special cases, where there are specific breakages you want to avoid while still migrating. I’d side in general with the view that the client is responsible for coping with the data it gets, and the server is responsible for a stable API, though not necessarily stable data. But it’s pretty likely there will also be special cases when being backwards-compatible data-wise is also important, and rewriting queries and data is a good solution (and what about when the data is retrieved by dereffing LOD, not SPARQLed? What should happen then?).

You could also leave the old triples in the store instead of rewriting, though there are potential drawbacks with that bloating the size of your store and response document sizes (you could possibly filter out the old triples from the RDF you are returning unless they are specified in the SPARQL query). In some cases having both old and new triples might make your data wrong or inconsistent.

The backwards-compatible store idea could be OK if the data is fairly static – you could sniff the SPARQL query for old triple patterns and redirect to the appropriate endpoint maybe?

There are lots of different ways you might change your data too – it could be a typo or switching vocab terms, or it could be a radically different modeling; in which case, it would be harder for the client to anticipate the change (and might even require more extensive rewriting of the client application).

Assuming that it is the client’s responsibility to adjust to changes in data pulled down from the wild web, there are a couple of things that I think might help:

1. A way of bundling up equivalent terms / graph patterns for specific situations. Maybe some collections of owl:sameAs and rdfs:subPropertyOf statements and the like, and a bit of reasoning, could work, but maybe a better solution would be something like profiles, where terms/graph patterns are stated to be functionally substitutable for a given task, eg: Here are a bunch of predicate URIs you can look through if you want to display something as a label, or here are a bunch of things you can try if you want to show an image associated with a (non-IR) resource; or (trickier) here’s a bunch of patterns you can try if you want to find tags/taggings. These profiles (for want of a better word) could be kept up to date and served up from somewhere, and the client application could retrieve and cache them – so then the client could be vocabulary agnostic, and cope with evolutions in data modeling, provided the profiles are kept up to date.

2. Ways for the server to hint to clients when a data migration has taken place – and what to do about it. I suppose the lightest weight option is to include old and new triples in the response for a while and hope the client picks up on it. The server could include extra metadata in the response document, or in it’s own (RDF, of course) service description, or dataset description (voiD provides terms to say what vocabularies a dataset is using, for example).

]]> By: Ian Davis http://blogs.talis.com/n2/archives/659/comment-page-1#comment-1022 Ian Davis Fri, 10 Jul 2009 09:34:21 +0000 http://blogs.talis.com/n2/?p=659#comment-1022 Dan, I think you could do rewriting of queries or some OWL might help. But practically speaking I would just leave the original triples there for a while. This does does raise the question of which side should bear the cost of changing data models: the client or the server. There's an argument to say that since clients are dealing with arbitrary data then they should be flexible in what they look for in a dataset. I have some notes on "open world" development that I really need to write up as a blog post.... Dan, I think you could do rewriting of queries or some OWL might help. But practically speaking I would just leave the original triples there for a while. This does does raise the question of which side should bear the cost of changing data models: the client or the server. There’s an argument to say that since clients are dealing with arbitrary data then they should be flexible in what they look for in a dataset. I have some notes on “open world” development that I really need to write up as a blog post….

]]> By: Elliot Smith http://blogs.talis.com/n2/archives/659/comment-page-1#comment-1019 Elliot Smith Fri, 10 Jul 2009 09:28:05 +0000 http://blogs.talis.com/n2/?p=659#comment-1019 Nice one Keith, could come in handy! Nice one Keith, could come in handy!

]]> By: Dan Brickley http://blogs.talis.com/n2/archives/659/comment-page-1#comment-1016 Dan Brickley Fri, 10 Jul 2009 09:15:53 +0000 http://blogs.talis.com/n2/?p=659#comment-1016 Thanks for writing this up. Nice and practical :) Any thoughts on how to deal with code "out there" which might be using the old graph patterns? Can a back-compatible extra store (or named graph) be generated in similar fashion? But I guess SPARQL endpoint details would need to change too. Have you looked into the pros/cons of rewriting incoming queries? Thanks for writing this up. Nice and practical :)

Any thoughts on how to deal with code “out there” which might be using the old graph patterns? Can a back-compatible extra store (or named graph) be generated in similar fashion? But I guess SPARQL endpoint details would need to change too. Have you looked into the pros/cons of rewriting incoming queries?

]]>