RDF specgen revisited - help needed!

[Dan Brickley]

Hi folks

I'm looking for some Python-hacking help, to get a new FOAF spec out.

Some of you might know about specgen. I've long used a version of this
script to re-generate the FOAF spec from sources. Originally in Ruby,
Uldis Bojars and Chris Schmidt made a Python version and disentangled it
from the specifics of the FOAF spec/site. Their specgen4.py, as well as
the earlier specgen.py the last FOAF spec was built from, both require
the compilation of Redland plus the Python bindings. Although Redland is
a solid, powerful and mature library, it can be tough to get the Python
bindings set up. Lately I've failed to manage this either on my Mac
laptop or under Linux at Dreamhost. Redland is overkill for this task.
I'd like to be able to rev the FOAF spec with far fewer dependencies.

So I've begun exploring a rewrite of specgen. My first attempt (now
abandoned as specgen4b.py) was to simply go through and replace calls to
Redland with similar calls to the librdf library. I found this rather
error prone, since the RDF logic is tangled up with page-generation code.

Despite my ignorance of Python and rdflib, I'm attempting a fresh start.
The idea is to have a few core utility classes, Vocab, Term (Class,
Property) and populate these - as before - from on disk RDF and HTML
descriptions of an RDF vocabulary.

Everything I have so far is at http://svn.foaf-project.org/foaftown/specgen/

The RDFS/OWL-loader and utility code is in libvocab.py. This does
nothing fancy; certainly no RDFS or OWL reasoning.

# Test FOAF spec
spec = Vocab( 'examples/foaf/index.rdf' )
spec.uri = 'http://xmlns.com/foaf.0.1/'
spec.index() # slurp info from sources

# spec.raw()
# print spec.report().encode('UTF-8')

for p in spec.properties:
print "Got a property: " + p
print p.simple_report().encode('UTF-8')
for c in spec.classes:
print "Got a class: " + c
print c.simple_report().encode('UTF-8')

...sample outout:

Got a class: Group
default: Group
id: Group
uri: http://xmlns.com/foaf/0.1/Group
xmlns: http://xmlns.com/foaf/0.1/
label: Group
comment: A class of Agents.

Is anyone interested in helping take this further? The existing
Redland-based specgen code is useful, but the dependency on configuring
Redland plus Python bindings is pretty heavy. Librdf (eg. see
http://www.science.uva.nl/research/air/wiki/RDFlib) seems to contain a
reasonable SPARQL implementation, and there's an RDFa parser around too
somewhere I'm sure. The current code at
http://svn.foaf-project.org/foaftown/specgen/ doesn't quite load up
enough information for the HTML-generation side of specgen to be
implemented. I'd love to find a more Python-aware collaborator or two to
help get this moving. Failing that, I need to get Redland+Python running
again so I can make a new FOAF spec :)

Thanks for any thoughts or help,

cheers

Dan

ps. if it isn't clear from the code, I've never written anything in
Python before! Am taking a 'pretend it is Ruby then google when stuff
breaks' approach...

--
http://danbri.org/

[Uldis Bojars]

On Thu, Dec 11, 2008 at 2:49 PM, Dan Brickley <dan...@danbri.org> wrote:

> Some of you might know about specgen. I've long used a version of this
> script to re-generate the FOAF spec from sources. Originally in Ruby,
> Uldis Bojars and Chris Schmidt made a Python version and disentangled it

> ...

> So I've begun exploring a rewrite of specgen. My first attempt (now
> abandoned as specgen4b.py) was to simply go through and replace calls to
> Redland with similar calls to the librdf library. I found this rather
> error prone, since the RDF logic is tangled up with page-generation code.

Just a quick note that I am rewriting specgen4 at the moment, prompted
by your questions re. using it with rdflib. Will post a link to the
code (work in progress) tomorrow. Perhaps we can combine our work.

> Is anyone interested in helping take this further? The existing
> Redland-based specgen code is useful, but the dependency on configuring
> Redland plus Python bindings is pretty heavy. Librdf (eg. see
> http://www.science.uva.nl/research/air/wiki/RDFlib) seems to contain a
> reasonable SPARQL implementation, and there's an RDFa parser around too

Count me in for getting SpecGen to a good shape. But can not spend too
much time on it right now. :(

> implemented. I'd love to find a more Python-aware collaborator or two to
> help get this moving.

+++

Uldis

[ http://captsolo.net/info/ ]

[Story Henry]

On 11 Dec 2008, at 15:49, Dan Brickley wrote:

> . Their specgen4.py, as well as
> the earlier specgen.py the last FOAF spec was built from, both require
> the compilation of Redland plus the Python bindings. Although
> Redland is
> a solid, powerful and mature library, it can be tough to get the
> Python
> bindings set up. Lately I've failed to manage this either on my Mac
> laptop or under Linux at Dreamhost. Redland is overkill for this task.
> I'd like to be able to rev the FOAF spec with far fewer dependencies.

Yes, I also had this trouble. It may be worth considering jython and
running this
on Sesame, Jena, Pellet or EulerSharp . Euler is very small, and reads
N3 rules, so this could
be done lightly.

(Mind you I think one may also be able to read cwm in Jython)

One could then create a very simple distribution for this in java.

Henry

[Dan Brickley]

On 11/12/08 16:01, Uldis Bojars wrote:
> On Thu, Dec 11, 2008 at 2:49 PM, Dan Brickley<dan...@danbri.org> wrote:
>
>> Some of you might know about specgen. I've long used a version of this
>> script to re-generate the FOAF spec from sources. Originally in Ruby,
>> Uldis Bojars and Chris Schmidt made a Python version and disentangled it
>> ...
>> So I've begun exploring a rewrite of specgen. My first attempt (now
>> abandoned as specgen4b.py) was to simply go through and replace calls to
>> Redland with similar calls to the librdf library. I found this rather
>> error prone, since the RDF logic is tangled up with page-generation code.
>
> Just a quick note that I am rewriting specgen4 at the moment, prompted
> by your questions re. using it with rdflib. Will post a link to the
> code (work in progress) tomorrow. Perhaps we can combine our work.

Ah cool, that would be great! Glad I sent out a mail before doing much
more work. I didn't write a lot, but beginning from a (crude) OO model
is feeling much easier than patching up the original code. I'd guess
much of the original specgen code could be reworked to consult a more OO
representation of Vocab, Class/Property etc.

>> Is anyone interested in helping take this further? The existing
>> Redland-based specgen code is useful, but the dependency on configuring
>> Redland plus Python bindings is pretty heavy. Librdf (eg. see
>> http://www.science.uva.nl/research/air/wiki/RDFlib) seems to contain a
>> reasonable SPARQL implementation, and there's an RDFa parser around too
>
> Count me in for getting SpecGen to a good shape. But can not spend too
> much time on it right now. :(

Great. Hope we can find time between us. All I want for now is enough to
regen the FOAF spec. Then to start making better use of the schema
translations people have done. But first step is simply to get back to
having a functional workflow!

Let's stay in touch on this...

cheers,

Dan

[Dan Brickley]

Cute idea but I'm trying to get away from exotic dependencies :) The
rdflib library seems to offer more than enough for poking around the
schema data. I just got a reply from Uldis on the SIOC list, he is
working on a rewrite, so hope we can fuse those efforts. Mine is pretty
basic so far but I think heading in right direction...

Dan

[Sergio Fernández]

On Thu, 2008-12-11 at 15:49 +0100, Dan Brickley wrote:

Hi Dan,

> I'm looking for some Python-hacking help, to get a new FOAF spec out.
>
> Some of you might know about specgen. I've long used a version of this
> script to re-generate the FOAF spec from sources. Originally in Ruby,
> Uldis Bojars and Chris Schmidt made a Python version and disentangled it
> from the specifics of the FOAF spec/site. Their specgen4.py, as well as
> the earlier specgen.py the last FOAF spec was built from, both require
> the compilation of Redland plus the Python bindings. Although Redland is
> a solid, powerful and mature library, it can be tough to get the Python
> bindings set up. Lately I've failed to manage this either on my Mac
> laptop or under Linux at Dreamhost. Redland is overkill for this task.
> I'd like to be able to rev the FOAF spec with far fewer dependencies.
>
> So I've begun exploring a rewrite of specgen. My first attempt (now
> abandoned as specgen4b.py) was to simply go through and replace calls to
> Redland with similar calls to the librdf library. I found this rather
> error prone, since the RDF logic is tangled up with page-generation code.

Some time ago we took specgen4 and we made many improvements [1], and
now it's full independent of the ontology (some people is using it).

I also planned rewrite it from scratch using RDFLib, becuase I've more
experience with it than with ReadLand and becuase IMHO is easier to use
and distribute with Python.

Probably we can combine all our work (you, Uldis and me).

Cheers,

[1] http://forge.morfeo-project.org/wiki_en/index.php/SpecGen

--
Sergio Fernández - sergio.f...@fundacionctic.org
R&D Deparment
CTIC Foundation - www.fundacionctic.org
Phone: +34 984 29 12 12
Fax: +34 984 39 06 12
Edificio Centros Tecnológicos
Parque Científico Tecnológico
33203 Cabueñes - Gijón - Asturias - Spain

[Uldis Bojars]

On Thu, Dec 11, 2008 at 2:49 PM, Dan Brickley <dan...@danbri.org> wrote:

> help get this moving. Failing that, I need to get Redland+Python running
> again so I can make a new FOAF spec :)

Did you have problems getting Redland+Python running on OS X?
If yes, how did you try to install it?

I have Redland+Python running OK but hear people asking questions re.
how to do that. I am wondering what are the common problems that one
can run into, if any.

Uldis

[ http://captsolo.net/info/ ]

[Dan Brickley]

On 11/12/08 16:12, Uldis Bojars wrote:
> On Thu, Dec 11, 2008 at 2:49 PM, Dan Brickley<dan...@danbri.org> wrote:
>
>> help get this moving. Failing that, I need to get Redland+Python running
>> again so I can make a new FOAF spec :)
>
> Did you have problems getting Redland+Python running on OS X?
> If yes, how did you try to install it?

Various flailing attempts. Last time I asked Dave Beckett he seemed to
think I was hitting a conflict between system-wide Python install and
something from Fink or Ports. All that ./configure stuff is great when
it works, ... but terribly obscure when it fails.

Here's a bug report from one recent attempt,
http://bugs.librdf.org/mantis/view.php?id=280

cheers,

Dan

[Uldis Bojars]

On Thu, Dec 11, 2008 at 3:09 PM, Sergio Fernández
<sergio.f...@fundacionctic.org> wrote:
>> So I've begun exploring a rewrite of specgen. My first attempt (now
>> abandoned as specgen4b.py) was to simply go through and replace calls to
>> Redland with similar calls to the librdf library. I found this rather
>> error prone, since the RDF logic is tangled up with page-generation code.
>
> Some time ago we took specgen4 and we made many improvements [1], and
> now it's full independent of the ontology (some people is using it).

Just to be fair, specgen4 is independent of the ontology. That is what
separates it from earlier versions which were hardcoded for FOAF. [1]
does a good work by adding more OWL-related information, among other
things (if I understand it correctly).

Perhaps we can convert specgen4 to librdf (and make it more OO) and
then reimplement changes done in [1].
What revision of specgen4 is [1] based on?

> Probably we can combine all our work (you, Uldis and me).

Ditto.

> [1] http://forge.morfeo-project.org/wiki_en/index.php/SpecGen

Uldis

[ http://captsolo.net/info/ ]

[Sergio Fernández]

On Thu, 2008-12-11 at 15:25 +0000, Uldis Bojars wrote:
> Just to be fair, specgen4 is independent of the ontology. That is what
> separates it from earlier versions which were hardcoded for FOAF. [1]
> does a good work by adding more OWL-related information, among other
> things (if I understand it correctly).

Exactly.

> Perhaps we can convert specgen4 to librdf (and make it more OO) and
> then reimplement changes done in [1].

That could be faster. But... RDFLib or librdf (aka redland)?

> What revision of specgen4 is [1] based on?

Based on your svn at 2007-12-12 (tomorrow one year!)

Cheers,

--
Sergio Fernández - sergio.f...@fundacionctic.org

Departamento I+D+i
Fundación CTIC - www.fundacionctic.org
Tlfn: +34 984 29 12 12

[Uldis Bojars]

On Thu, Dec 11, 2008 at 3:17 PM, Dan Brickley <dan...@danbri.org> wrote:
>> Did you have problems getting Redland+Python running on OS X?
>> If yes, how did you try to install it?
>
> Various flailing attempts. Last time I asked Dave Beckett he seemed to
> think I was hitting a conflict between system-wide Python install and
> something from Fink or Ports. All that ./configure stuff is great when
> it works, ... but terribly obscure when it fails.
>
> Here's a bug report from one recent attempt,
> http://bugs.librdf.org/mantis/view.php?id=280

Another approach to try is installing via MacPorts. It will build its
own version of Python though (that's their policy for making
dependencies / compiling more predictable).

> port install python25 (or 24 if you prefer it)
> python_select python25
> port [-d] install redland-bindings +python25

Uldis

[ http://captsolo.net/info/ ]

[Dan Brickley]

On 11/12/08 16:25, Uldis Bojars wrote:
> On Thu, Dec 11, 2008 at 3:09 PM, Sergio Fernández
> <sergio.f...@fundacionctic.org> wrote:
>>> So I've begun exploring a rewrite of specgen. My first attempt (now
>>> abandoned as specgen4b.py) was to simply go through and replace calls to
>>> Redland with similar calls to the librdf library. I found this rather
>>> error prone, since the RDF logic is tangled up with page-generation code.
>> Some time ago we took specgen4 and we made many improvements [1], and
>> now it's full independent of the ontology (some people is using it).
>
> Just to be fair, specgen4 is independent of the ontology. That is what
> separates it from earlier versions which were hardcoded for FOAF. [1]
> does a good work by adding more OWL-related information, among other
> things (if I understand it correctly).

Yes, specgen4 was an improvement. If it wasn't for the pain of the
redland dependency I might have managed to use it directly for the FOAF
spec.

All uglyness in this code family is probably my fault - the original
Ruby version
http://svn.foaf-project.org/foaf/trunk/xmlns.com/htdocs/foaf/0.1/specgen.rb
had a lot of bad habits.

> Perhaps we can convert specgen4 to librdf (and make it more OO) and

I tried that (without the OO bit) first:

http://svn.foaf-project.org/foaftown/specgen/old/specgen4b.py (abandoned
in favour of libvocab.py approach now)

I ended up with code like this, through trying to avoid major changes:

for classStatement in wrap_find_statements(m, None, rdf.type,
rdfs.Class):
speclog("OK we got an rdfs class statement. sub
is:"+classStatement[0])
if classStatement[0].startswith(spec_url) :
for range in wrap_find_statements(m, None, rdfs.range,
classStatement[0]):
if not wrap_matches(m, range[0], rdf.type,
owl.DeprecatedProperty ):
classranges.setdefault(classStatement[0],
[]).append(str(range[0]))
for domain in wrap_find_statements(m, None, rdfs.domain,
classStatement[0]):
if not wrap_matches(m, domain[0], rdf.type,
owl.DeprecatedProperty):
classdomains.setdefault(str(classStatement[0]),
[]).append(str(domain[0]))
classlist.append(return_name(m, classStatement[0]))



My sense is that a lot of this would be much cleaner to test (especially
as the family of specs using it grows) if we base it on SPARQL queries.
For example, I made this query:

q= 'SELECT distinct ?x ?l ?c WHERE { ?x rdfs:label ?l . ?x rdfs:comment
?c . ?x a ?type . FILTER (?type =
<http://www.w3.org/2002/07/owl#ObjectProperty> || ?type =
<http://www.w3.org/2002/07/owl#DatatypeProperty> || ?type =
<http://www.w3.org/1999/02/22-rdf-syntax-ns#Property> || ?type =
<http://www.w3.org/2002/07/ow
l#FunctionalProperty> || ?type =
<http://www.w3.org/2002/07/owl#InverseFunctionalProperty>) } '

...after I realised my query against rdf:Property was failing with SIOC
data. Making such improvements over API-based graph access I think will
be quite fragile. Similarly, when I tried to load DOAP's schema, which
has several languages in the core file, I ran into problems that could
be diagnosed via SPARQL.

> then reimplement changes done in [1].
> What revision of specgen4 is [1] based on?
>
>> Probably we can combine all our work (you, Uldis and me).
>
> Ditto.

I'd love to see that :)

Probably a first step might be to have a simple data model that HTML
generation is done against. I don't think this should be the RDF graph
directly, but rather a simple set of OO structures that are populated by
reading the RDFS/OWL file(s).

There are a few issues I had with the FOAF spec, like how to represent
links to offsite schemas, which I ought to writeup asap. For example, in
foaf:based_near I reference geo:SpatialThing. I'd like to have a section
of the spec keep track of these links. What do the other tools do for this?

cheers,

Dan

[Dan Brickley]

Crossposting again, sorry for duplicates.

After discussion on sioc-dev and a bit in Skype with Sergio, I've made a
little more efforts on my new code. I hope and expect these strands can
all integrate, but I don't want to make a giant project of this before
getting a new FOAF rev out.

So what I've done is commit some Python unit tests that externalise some
of my wishlist and TODOs.

http://svn.foaf-project.org/foaftown/specgen/run_tests.py

The script defines a subset suite for tests that ought to pass now, and
includes a couple more that represent work I want to do. It's pretty
basic but a start. If anyone wants write access to this filetree, just
let me know :)

[Danny Ayers]

danbri, any reason for:

?x a ?type .
FILTER (?type = <http://www.w3.org/2002/07/owl#ObjectProperty>)
instead of
?x a owl:ObjectProperty .

?

hmm, although I suspect you might want OPTIONALs to catch
owl:DatatypeProperty etc.


2008/12/11 Dan Brickley <dan...@danbri.org>:

--
http://danny.ayers.name

[Danny Ayers]

2008/12/11 Nicholas J Humfrey <n...@aelius.com>:
> Possible alternative;
>
> Patrick Sinclair wrote ldontospec:
> http://github.com/metade/ldontospec
> "An ontology documentation generator based on the principles of Linked
> Data."
>
> Which is used to build the Programmes Ontology page:
> http://www.bbc.co.uk/ontologies/programmes/

Interesting, but I got a bit stuck on one of its dependencies, after
installing another 4 or 5 other libs, I hit :

./ldontospec

/usr/lib/ruby/1.8/rubygems/custom_require.rb:27:in
`gem_original_require': no such file to load -- activerdf_reddy
(LoadError)
from /usr/lib/ruby/1.8/rubygems/custom_require.rb:27:in `require'
...

I could only find activerdf_redland - somewhat similar name.
Happen to know if that is the one (i.e. I should maybe tweak the naming)?

(not got time to chase any further right now)

Cheers,
Danny.

--
http://danny.ayers.name

[Uldis Bojars]

> Just a quick note that I am rewriting specgen4 at the moment, prompted
> by your questions re. using it with rdflib. Will post a link to the
> code (work in progress) tomorrow. Perhaps we can combine our work.

The code is at http://sw.deri.org/svn/sw/2005/08/sioc/specgen/branches/rdflib/

RDF library specific code is now separate from specgen4.py. It still
uses Redland at the moment but separating template code and RDF-
related code is a step in the right direction. Please note that while
there is some abstraction from a particular RDF library, the code is
limited in functionality to what is needed by SpecGen.

Further discussions re. development of SpecGen will take place at:
https://lists.morfeo-project.org/mailman/listinfo/specgen-devel

Uldis

[ http://captsolo.net/info/ ]