Proposal to start forward momentum
Ross Singer
trying to define what Jangle should be and how we should implement it.
One the problems (in my mind) is that, by trying to build a spec from
scratch, we've got too many possibilities that are causing
distractions.
In the effort to get some of this inertia moving in the right
direction, here's what I'm proposing:
1) We use the DLF ILS Discovery Interface Task Force recommendations
[1] as a basis for the baseline functionality that Jangle must be able
to support (although, not necessarily directly, but modules can be
build atop Jangle to meet the specific functionality). This does not
mean that we're limiting Jangle to this, but using it as a goal to
push Jangle past this hand-wavy place we're currently in and gives us
a reasonable scope.
2) Jangle communicates with clients via the Atom Publishing Protocol.
[2] This will give us a defined framework for broadcasting services,
syndicating results and managing resources RESTfully. It's also got
adoption *outside* libraries and there are client libraries to work
with the protocol in a variety of languages. This effectively makes
one goal of Jangle to be a map between library (and other) services
and the APP.
3) We adopt the Rogue 05 guidelines [3] for development of the Jangle
spec. This probably means breaking down each resource into an
individual components; the entirety of the spec in one month seems a
touch ambitious.
4) Jangle will explicitly define the fields that the connectors should
send back and, in many cases define what kind of value should be
returned. The spec will need to define some strict enumerations where
disparate machines would probably have variation (example: defining
item status as "available", "non-circulating", "not available", "date
due", etc.)
5) The spec will also define the relationship between Jangle core and
the service connectors. To get started, I propose we only focus on
supporting JSON connectors. This way we can sidestep the issue of
finding or drafting XML schemas.
So, what do you think? Every point above is open to debate (as is the
entire proposal, of course), so please weigh in on ideas. Do you
think it's too simplistic? If so, why? Do you have use cases that
cannot be addressed by the above? If so, are they worthy of a
different, more specific solution to that problem or is this proposal
too generic to be of much use?
I think at this point we can do well to work inside some well defined
parameters and see if there's any traction to be had.
Thanks,
-Ross.
[1] https://project.library.upenn.edu/confluence/display/ilsapi/Home
[2] http://bitworking.org/projects/atom/rfc5023.html
[3] http://wiki.code4lib.org/index.php/Rogue
Andrew Nagy
to this list to see what activity is going on. I would like to
eventually replace the driver model in VuFind with the jangle api.
I would be happy to contribute what i have been able to accomplish
thus far with VuFind. And I would also like to help in the
development efforts of jangle. Im fairly new to ruby, but I have been
looking for a project to work on to start diving into ruby code - this
project seems perfect.
I do think the initial plan is a bit too simplistic - the DLF ILS
framework is mainly geared for discovery platforms and I think jangle
should be more than that. To make an ILS truly modular - there should
be a framework to work with to allow any module to work with an ILS.
For example, Liblime's biblios application should be able to work with
any ILS with a framework in place like jangle. But - the DLF ILS model
is a good place to start - so i agree with the plan to start there.
If you are interested you can see the abstract class that I have been
working from for the drivers in VuFind:
http://vufind.org/wiki/building_an_ils_driver
If you want to divide and conquer - just let me know what I can help work on.
Rock on
Andrew
townxelliot
This is a great set of pointers for a way forward.
On Fri, 2008-03-14 at 16:15 -0400, Ross Singer wrote:
> So, I think we've been spinning our wheels a bit while we've been
> trying to define what Jangle should be and how we should implement it.
> One the problems (in my mind) is that, by trying to build a spec from
> scratch, we've got too many possibilities that are causing
> distractions.
>
> In the effort to get some of this inertia moving in the right
> direction, here's what I'm proposing:
>
> 1) We use the DLF ILS Discovery Interface Task Force recommendations
> [1] as a basis for the baseline functionality that Jangle must be able
> to support (although, not necessarily directly, but modules can be
> build atop Jangle to meet the specific functionality). This does not
> mean that we're limiting Jangle to this, but using it as a goal to
> push Jangle past this hand-wavy place we're currently in and gives us
> a reasonable scope.
This sounds like a good place to start. My comment here is that we need
to bear in mind that we'll want to go beyond querying and read-only
operations, and keep that in mind during design. But APP provides the
frame for that, so we can't go too far wrong.
>
> 2) Jangle communicates with clients via the Atom Publishing Protocol.
> [2] This will give us a defined framework for broadcasting services,
> syndicating results and managing resources RESTfully. It's also got
> adoption *outside* libraries and there are client libraries to work
> with the protocol in a variety of languages. This effectively makes
> one goal of Jangle to be a map between library (and other) services
> and the APP.
I like the idea of using APP. I think what would convince me (and
perhaps others) is an example of how this might be applied to both read
and write scenarios. The documentation I've read on APP uses mainly blog
and RSS examples, so I'd be interested to see an example in a different
(library) context. Perhaps borrower info. (which seems pretty important
to DLF) would be the place to start, as we've already been considering
that in Jangle prototypes.
>
> 3) We adopt the Rogue 05 guidelines [3] for development of the Jangle
> spec. This probably means breaking down each resource into an
> individual components; the entirety of the spec in one month seems a
> touch ambitious.
Hadn't come across these before. They look pragmatic and useful. Also
liked the reference to Dogme :)
>
> 4) Jangle will explicitly define the fields that the connectors should
> send back and, in many cases define what kind of value should be
> returned. The spec will need to define some strict enumerations where
> disparate machines would probably have variation (example: defining
> item status as "available", "non-circulating", "not available", "date
> due", etc.)
This is one area which the NCIP group seems to be addressing (the
unspecified nature of enumerable fields in NCIP), and which caused me
most discomfort when I was trying to implement parts of the spec.. I
definitely agree that we should at least attempt to define enumerations.
>
> 5) The spec will also define the relationship between Jangle core and
> the service connectors. To get started, I propose we only focus on
> supporting JSON connectors. This way we can sidestep the issue of
> finding or drafting XML schemas.
Agreed. Good point about sidestepping XML schema. We still need to
define types etc. on response objects and their properties, but the
limited set available in JSON makes this tractable.
>
> So, what do you think? Every point above is open to debate (as is the
> entire proposal, of course), so please weigh in on ideas. Do you
> think it's too simplistic? If so, why? Do you have use cases that
> cannot be addressed by the above? If so, are they worthy of a
> different, more specific solution to that problem or is this proposal
> too generic to be of much use?
I don't think this is too simplistic, just constrained, which is good.
I'd rather have a limited core which can't cope with everything but
which can be circumnavigated, rather than a broad, vague series of
suggestions and hints.
Elliot
Ross Singer
Jangle; VuFind is obviously a really good use case to keep Jangle
rooted in the real world. If it can't meet VuFind's needs, something
needs to change :)
More comments inline:
On Fri, Mar 14, 2008 at 6:02 PM, Andrew Nagy <asn...@gmail.com> wrote:
>
> Ross - this sounds great. I was just thinking about sending an email
> to this list to see what activity is going on. I would like to
> eventually replace the driver model in VuFind with the jangle api.
Hopefully this will be an obvious choice (eventually) for all projects
like VuFind. If the interfaces to all library systems are roughly the
same, I think we will be able to see some amazingly innovative work.
We, unfortunately, are all spending far too much time trying to get
our data out right now.
>
> I would be happy to contribute what i have been able to accomplish
> thus far with VuFind. And I would also like to help in the
> development efforts of jangle. Im fairly new to ruby, but I have been
> looking for a project to work on to start diving into ruby code - this
> project seems perfect.
Well, that's the nice part -- there's no commitment to Ruby at the
moment. This new suggestion of using APP pretty much puts as back to
square one on the Jangle core layer (although I can hear Mike saying
"OpenSRF can handle that!" on a back road in the northern suburbs of
Atlanta somewhere). The connectors communicate with the Jangle core
via JSON, so they can be anything -- PHP, Perl, Java, Lua, COBOL. I
think this flexibility on the backend is critical - I see too many
projects get overlooked because there isn't language expertise in
house to support it. Ok, maybe COBOL was exaggerating...
>
> I do think the initial plan is a bit too simplistic - the DLF ILS
> framework is mainly geared for discovery platforms and I think jangle
> should be more than that. To make an ILS truly modular - there should
> be a framework to work with to allow any module to work with an ILS.
> For example, Liblime's biblios application should be able to work with
> any ILS with a framework in place like jangle. But - the DLF ILS model
> is a good place to start - so i agree with the plan to start there.
I totally agree with this. It doesn't begin to address systems like
interlibrary loan, for example. Biblios is a good example, an
acquisitions system would be another. My only reason for targeting
the DLF ILS spec is that it's an obvious need that has been pretty
well thought out. Jangle wouldn't *directly* meet their needs (in my
mind, a DLF ILS API would live /on top/ of a Jangle API), but as we
define resources (Resource, Item, Actor, etc.) we should use the DLF
recommendation to make sure we could at least meet their needs
(although I'm sure other requirements will come up as well). It's
trying to find some balance between focusing too much on one solution
and falling into the void of NCIP.
>
> If you are interested you can see the abstract class that I have been
> working from for the drivers in VuFind:
> http://vufind.org/wiki/building_an_ils_driver
>
This is a great starting place to begin defining an API. How are you
currently populating VuFind with records? Native ILS bulk MARC
export? Since you're a Voyager site, it seems pretty trivial to set
up an OAI-PMH type solution on the Oracle DB.
> If you want to divide and conquer - just let me know what I can help work on.
>
I'm not sure we need to divide just yet :) I do think leveraging your
experience with what you've already learned about getting data out of
systems for VuFInd will be immensely useful in specing out the
interface between Jangle core and the connectors.
> Rock on
\m/
-Ross.
Ross Singer
>
> This is a great set of pointers for a way forward.
Well, it's *something*. I've been getting a little overwhelmed lately
about how to proceed, mainly because I wasn't sure what we were really
trying to make. This doesn't reduce the amount of work we need to do,
but it least gives us some idea of what we're working towards.
>
> This sounds like a good place to start. My comment here is that we need
> to bear in mind that we'll want to go beyond querying and read-only
> operations, and keep that in mind during design. But APP provides the
> frame for that, so we can't go too far wrong.
Absolutely. And, thankfully, we have a few systems where we can
actually implement that right away (EG, Koha, Horizon, I hear -- I,
uh, suppose we probably have a copy of Alto somewhere we can break).
This gives us the possibility to keep ourselves in check from the
get-go, rather than realizing after the fact that we've painted
ourselves into some corner.
We might want to set up a generic ILS somewhere to test against.
>
>
> I like the idea of using APP. I think what would convince me (and
> perhaps others) is an example of how this might be applied to both read
> and write scenarios. The documentation I've read on APP uses mainly blog
> and RSS examples, so I'd be interested to see an example in a different
> (library) context. Perhaps borrower info. (which seems pretty important
> to DLF) would be the place to start, as we've already been considering
> that in Jangle prototypes.
>
This is good point. How about if I start sketching something out of
the wiki and then others can contribute once a strawman is there?
>
>
> Hadn't come across these before. They look pragmatic and useful. Also
> liked the reference to Dogme :)
We came up with this when we were developing COinS. It doesn't make
the process of building a spec any less painful, but it makes it a
hell of a lot shorter.
>
>
>
> This is one area which the NCIP group seems to be addressing (the
> unspecified nature of enumerable fields in NCIP), and which caused me
> most discomfort when I was trying to implement parts of the spec.. I
> definitely agree that we should at least attempt to define enumerations.
>
And I think we need to err on the side of brevity with this, too.
NCIP (and countless other library specs -- ONIX SOH, for example) bog
themselves down completely with corner cases. In fact, I'll throw out
the argument that this "we're all completely different!" argument is
one of the reasons we've gotten ourselves into the current mess. I
think some standardization and streamlining of all this would do us
all a lot of good (granted, I may be heading for a rude awakening
here).
>
>
> Agreed. Good point about sidestepping XML schema. We still need to
> define types etc. on response objects and their properties, but the
> limited set available in JSON makes this tractable.
Absolutely. JSON doesn't get us around having to model the data.
Having to model it in XML, however, can come later (although,
realistically, might be worthwhile later).
-Ross.
Mike Rylander
> trying to define what Jangle should be and how we should implement it.
> One the problems (in my mind) is that, by trying to build a spec from
> scratch, we've got too many possibilities that are causing
> distractions.
Ross and I had some off-list discussion about this in IRC, during
which I raised some concerns about the applicability of APP to
business processes that don't, in my mind, map to simple
PUT/POST/GET/DELETE operations. Basically, it boils down to this: in
my head, there are operations which require more than one step, and
for completion are based on intermediate data that the client end
would provide. However, after more reflection, I've been convinced
that APP covers all the cases in the DLF spec and then some, and I'm
finding it hard to come up with a reasonable blocker for APP. So ....
>
> In the effort to get some of this inertia moving in the right
> direction, here's what I'm proposing:
>
> 1) We use the DLF ILS Discovery Interface Task Force recommendations
> [1] as a basis for the baseline functionality that Jangle must be able
> to support (although, not necessarily directly, but modules can be
> build atop Jangle to meet the specific functionality). This does not
> mean that we're limiting Jangle to this, but using it as a goal to
> push Jangle past this hand-wavy place we're currently in and gives us
> a reasonable scope.
>
Roadmap++
> 2) Jangle communicates with clients via the Atom Publishing Protocol.
> [2] This will give us a defined framework for broadcasting services,
> syndicating results and managing resources RESTfully. It's also got
> adoption *outside* libraries and there are client libraries to work
> with the protocol in a variety of languages. This effectively makes
> one goal of Jangle to be a map between library (and other) services
> and the APP.
>
APP++
> 3) We adopt the Rogue 05 guidelines [3] for development of the Jangle
> spec. This probably means breaking down each resource into an
> individual components; the entirety of the spec in one month seems a
> touch ambitious.
>
It's a process to follow that we don't have to invent, and it worked
for unAPI ... so rogue++
> 4) Jangle will explicitly define the fields that the connectors should
> send back and, in many cases define what kind of value should be
> returned. The spec will need to define some strict enumerations where
> disparate machines would probably have variation (example: defining
> item status as "available", "non-circulating", "not available", "date
> due", etc.)
>
enum++
> 5) The spec will also define the relationship between Jangle core and
> the service connectors. To get started, I propose we only focus on
> supporting JSON connectors. This way we can sidestep the issue of
> finding or drafting XML schemas.
>
I'm not sure this really stops the pain, but it makes testing and RAD
easier in my mind.
> So, what do you think? Every point above is open to debate (as is the
> entire proposal, of course), so please weigh in on ideas. Do you
> think it's too simplistic? If so, why? Do you have use cases that
> cannot be addressed by the above? If so, are they worthy of a
> different, more specific solution to that problem or is this proposal
> too generic to be of much use?
>
> I think at this point we can do well to work inside some well defined
> parameters and see if there's any traction to be had.
>
Thanks Ross. Way to Champion(tm)!
--miker
> Thanks,
> -Ross.
> [1] https://project.library.upenn.edu/confluence/display/ilsapi/Home
> [2] http://bitworking.org/projects/atom/rfc5023.html
> [3] http://wiki.code4lib.org/index.php/Rogue
>
> >
>
--
Mike Rylander
| VP, Research and Design
| Equinox Software, Inc. / The Evergreen Experts
| phone: 1-877-OPEN-ILS (673-6457)
| email: mi...@esilibrary.com
| web: http://www.esilibrary.com
Richard Wallis
Technology Evangelist, Talis
Tel: +44 (0)870 400 5422 (Direct)
Tel: +44 (0)870 400 5000 (Switchboard)
Tel: +44 (0)7767 886 005 (Mobile)
Fax: +44 (0)870 400 5001
Skype: richard.wallis1
IM: rjw...@hotmail.com
I-Name: =Richard.Wallis
Galen Charlton
On Fri, Mar 14, 2008 at 3:15 PM, Ross Singer <rossf...@gmail.com> wrote:
> 1) We use the DLF ILS Discovery Interface Task Force recommendations
> [1] as a basis for the baseline functionality that Jangle must be able
> to support (although, not necessarily directly, but modules can be
> build atop Jangle to meet the specific functionality). This does not
> mean that we're limiting Jangle to this, but using it as a goal to
> push Jangle past this hand-wavy place we're currently in and gives us
> a reasonable scope.
Sounds like a good, constrained starting point. I would be also
interested in working out a suitable API for the Biblios integration
that Andrew mentioned.
> 2) Jangle communicates with clients via the Atom Publishing Protocol.
> [2] This will give us a defined framework for broadcasting services,
> syndicating results and managing resources RESTfully. It's also got
> adoption *outside* libraries and there are client libraries to work
> with the protocol in a variety of languages. This effectively makes
> one goal of Jangle to be a map between library (and other) services
> and the APP.
+1
> 3) We adopt the Rogue 05 guidelines [3] for development of the Jangle
> spec. This probably means breaking down each resource into an
> individual components; the entirety of the spec in one month seems a
> touch ambitious.
+1. Certainly we ought to be able to do each service individually on
that sort of timeframe.
> 4) Jangle will explicitly define the fields that the connectors should
> send back and, in many cases define what kind of value should be
> returned. The spec will need to define some strict enumerations where
> disparate machines would probably have variation (example: defining
> item status as "available", "non-circulating", "not available", "date
> due", etc.)
I agree that having specific enumerations is the way to go.
> 5) The spec will also define the relationship between Jangle core and
> the service connectors. To get started, I propose we only focus on
> supporting JSON connectors. This way we can sidestep the issue of
> finding or drafting XML schemas.
Also sounds fine to me.
Regards,
Galen
Andrew Nagy
>
> Andrew, welcome aboard! It's really great that you're interested in
> Jangle; VuFind is obviously a really good use case to keep Jangle
> rooted in the real world. If it can't meet VuFind's needs, something
> needs to change :)
Excellent!
> Well, that's the nice part -- there's no commitment to Ruby at the
> moment. This new suggestion of using APP pretty much puts as back to
> square one on the Jangle core layer (although I can hear Mike saying
> "OpenSRF can handle that!" on a back road in the northern suburbs of
> Atlanta somewhere). The connectors communicate with the Jangle core
> via JSON, so they can be anything -- PHP, Perl, Java, Lua, COBOL. I
> think this flexibility on the backend is critical - I see too many
> projects get overlooked because there isn't language expertise in
> house to support it. Ok, maybe COBOL was exaggerating...
The language doesn't matter to me - lets pick one that has the best
support for the components and standards that we want to use and go
gung-ho. I picked PHP for vufind since at the time it seems that PHP
is the best "known" language in libraries and the open-source world.
Just make sure it is easy to install :)
> >
> > If you are interested you can see the abstract class that I have been
> > working from for the drivers in VuFind:
> > http://vufind.org/wiki/building_an_ils_driver
> >
>
> This is a great starting place to begin defining an API. How are you
> currently populating VuFind with records? Native ILS bulk MARC
> export? Since you're a Voyager site, it seems pretty trivial to set
> up an OAI-PMH type solution on the Oracle DB.
We have a OAI-PMH server that sits on top of Solr which houses all of
the bib records. This allows others to access the records - not from
the ILS but from the discovery layer. This probably doesn't help us
any with this effort. We have a java import tool for bulk importing
marc records and I am working on an OAI harvester to allow other
formats other than marc. Voyager has a bulk export tool which is
really nice to get any thing out of it. Our life is much easier than
those with other ILSs.
>
>
> > If you want to divide and conquer - just let me know what I can help work on.
> >
> I'm not sure we need to divide just yet :) I do think leveraging your
> experience with what you've already learned about getting data out of
> systems for VuFInd will be immensely useful in specing out the
> interface between Jangle core and the connectors.
Sure - I'd love to help out. I'd be more than willing to put out a
draft spec in "ROGUE" style and send it out - unless this has already
begun?
Andrew
Andrew Nagy
> syndicating results and managing resources RESTfully. It's also got
> adoption *outside* libraries and there are client libraries to work
> with the protocol in a variety of languages. This effectively makes
> one goal of Jangle to be a map between library (and other) services
> and the APP.
Im not totally familiar with APP and just started poking around at its
spec - I only remember the stuff from Ed's presentation at last years
code4lib.
I am curious if the "paging" in APP's listcollection method will work
in the same method as a resumptiontoken in OAI. The idea behind a
resumptiontoken is that it is basically a session - whereas the
"paging" in the listcollection method seems to be more of a "retrieve
100 records at a time" type of transaction. This could work - but
would OAI be more powerful?
I am also concerned about the search functionality - how would that be
addressed? Would we use SRU for that - or could you still use APP for
that?
Ross Singer
I've been thinking about this (not ignoring it!), so I'll see if I can
clear any of this up.
On Mon, Mar 17, 2008 at 12:49 PM, Andrew Nagy <asn...@gmail.com> wrote:
> Im not totally familiar with APP and just started poking around at its
> spec - I only remember the stuff from Ed's presentation at last years
> code4lib.
>
Uh, well, honestly, I'm not sure *any* of us are totally familiar with
it. I figure there will be a lot of cribbing of ideas from GData.
> I am curious if the "paging" in APP's listcollection method will work
> in the same method as a resumptiontoken in OAI. The idea behind a
> resumptiontoken is that it is basically a session - whereas the
> "paging" in the listcollection method seems to be more of a "retrieve
> 100 records at a time" type of transaction. This could work - but
> would OAI be more powerful?
The way I look at it is that it should be fairly trivial to place an
OAI provider on top of an Atom collection. The collection document
could tell you alternative formats (ListMetadataPrefixes), subsets,
etc. Functionally it would also expose the data to a broader range of
clients. So the OAI-PMH provider (and, note, I think it would have to
be a priority to make a Jangle-compatible OAI service) would be able
to serve to the traditional OAI clients (such as IRs and archives),
but other, less library-centric, services could use the Atom
syndication feed. I don't feel confident that OAI-PMH has that much
uptake outside of libraries (although, undoubtedly there's some).
>
> I am also concerned about the search functionality - how would that be
> addressed? Would we use SRU for that - or could you still use APP for
> that?
Honestly, I'd prefer to use OpenSearch by default here. Even the
OASIS group is moving towards a more OpenSearch-like SRU. I'm pretty
flexible on this, and, obviously, it's not a zero sum game. We can
implement both :)
-Ross.