Data from PIMS
Simone Spencer
Hello everyone,
I’m just after a bit of feedback from those of you to whom I have provided downloads of data from PIMS. I’m about to do one for the Resource Discovery iKit project http://cdlr.strath.ac.uk/rdinfokit/service/index.cfm and wondered if you had experienced any problems with the receipt or loading of the data which I might be able to improve for this next set.
Also, is anyone ready for a new set of data? Can everyone agree on a monthly extraction for example so I can just do one extraction and make it available for you all to access somewhere?
Best wishes,
Simone
Assistant Programme Manager
JISC Executive
Brettenham House South
5 Lancaster Place
London
WC2E 7EN
Email. s.sp...@jisc.ac.uk
Tel. 020 3006 6031
Mob. 07929 378139
Fax. 020 7240 5377
Skype ID. spencer.simone
Anything in this message which does not clearly relate to the official
work of the sender's organisation shall be understood as neither given
nor endorsed by that organisation.
Ross Gardler
> Hello everyone,
>
> I'm just after a bit of feedback from those of you to whom I have provided
> downloads of data from PIMS. I'm about to do one for the Resource Discovery
> iKit project http://cdlr.strath.ac.uk/rdinfokit/service/index.cfm and
> wondered if you had experienced any problems with the receipt or loading of
> the data which I might be able to improve for this next set.
Without a data dictionary it is very hard to make sense of such data
dumps. We've had someone reverse engineer a data dictionary, but
documentation would have saved us a great deal of time. Once our own
data dictionary has been checked we'd be happy to provide this for
your documentation.
We've not got around to importing into Simal yet, we expect that to
happen soon, but we don't currently have a deeloper on staff. Not sure
how long it will be, weeks not months I would think.
> Also, is anyone ready for a new set of data? Can everyone agree on a monthly
> extraction for example so I can just do one extraction and make it available
> for you all to access somewhere?
Fine be me why don't you just set a date of 1st of the month. Once we
have automated importing of the data from your spreadsheets we will be
able to expose that data and thus make it available on demand, in a
more usable form, to anyone who needs it.
Here at dev8d I've got a hacker playing with the data out of Simal.
Hope to have something "cool" with respect to extracting collective
intelligence data from project descriptions (bu don't hold your
breath).
--
--
Ross Gardler
OSS Watch - awareness and understanding of open source software
development and use in education
http://www.oss-watch.ac.uk
Simone Spencer
> dumps. We've had someone reverse engineer a data dictionary, but
> documentation would have saved us a great deal of time. Once our own
> data dictionary has been checked we'd be happy to provide this for
> your documentation.
great
> Here at dev8d I've got a hacker playing with the data out of Simal.
> Hope to have something "cool" with respect to extracting collective
> intelligence data from project descriptions (bu don't hold your
> breath).
working on. Is this the same thing?
http://dev8d.jiscinvolve.org/2009/02/11/cool-tool-jisc-projects-visualis
ation/
Simone
Assistant Programme Manager
JISC Executive, London
Skype. spencer.simone
work of the sender's organisation shall be understood as neither given
nor endorsed by that organisation.
Richard King
From our point of view the data exports in Excel format that you've sent
us to date are fine. Our code currently matches on column headings so as
long as they don't change we're OK (and even if they do, we have a
mapping function that I'm told isn't difficult to update - but that's
less than ideal).
On regular exports, it would be good if we could build some kind of
automatic system for publishing the data. Being at Dev8D has put me in a
mashup frame of mind, so I'm currently thinking along the lines of a
GoogleDocs spreadsheet upload. We could then use the doc's RSS feed to
discover when new data is available, and automatically download it in
whatever format is most useful.
Of course the data confidentiality and security implications of hosting
on Google would need to be considered!
Failing that, we would definitely be interested in a manual monthly
export. First of the month works for us.
Regards
Richard.
--
Psydev Ltd
Specialist Consultants in Software and Standards
tel | +44 (0)114 2664875
mobile | +44 (0)7966 846252
skype | psydev_richard_king
twitter | psydev_rpk
web | http://www.psydev.com/
Legislation requires us to include the following information:
Registration Number: 4819126 in England and Wales
Reg'd Address: 5 Westbrook Court, Sharrowvale Road, Sheffield, S11 8YZ
Ross Gardler
>
> 2009/2/11 Ross Gardler
...
> Andy McGregor has sent me a link to something Tony Hirst from the OU is
> working on. Is this the same thing?
> http://dev8d.jiscinvolve.org/2009/02/11/cool-tool-jisc-projects-visualis
> ation/
Yep, it's Simal he's pulling the data from. Expect more
experimentation (just the 500 in the test system right now)
Ross
Simone Spencer
Simone
Assistant Programme Manager
JISC Executive, London
Email. s.sp...@jisc.ac.uk
Tel. 020 3006 6031
Mob. 07929 378139
Skype. spencer.simone
-----Original Message-----
From: simal...@googlegroups.com [mailto:simal...@googlegroups.com]
On Behalf Of Richard King
Sent: 11 February 2009 16:40
To: simal...@googlegroups.com
Cc: apf
Subject: [simal-users] Re: Data from PIMS
Richard King
> Are you all at Dev8D? why am I stuck in the office?!!
I could only spare the time to attend Day 0 - I've since returned to
snowy Sheffield. It was enough to get my brain thinking in new
directions though :)
Ross Gardler
...
> On regular exports, it would be good if we could build some kind of
> automatic system for publishing the data. Being at Dev8D has put me in a
> mashup frame of mind, so I'm currently thinking along the lines of a
> GoogleDocs spreadsheet upload. We could then use the doc's RSS feed to
> discover when new data is available, and automatically download it in
> whatever format is most useful.
This is what Simal is for. It doesn't currently give you RSS feeds of
updates, although I think its a sensible feature, please add a feature
request to http://code.google.com/p/simal/issues/list
Or data is much easier to use for mashups as it pulls all the data
into a single usable file via a REST interface and does various
semantic linking of data between records. Currently we only export RDF
and JSON, but other formats made avilable on request (or patches
welcome if you want it quick)
>
> Of course the data confidentiality and security implications of hosting
> on Google would need to be considered!
We only publish the public data so no worries there and it's not on
Google servers,
Ross
Richard King
> 2009/2/11 Richard King <r...@psydev.com>:
>
> ...
>
>> On regular exports, it would be good if we could build some kind of
>> automatic system for publishing the data. Being at Dev8D has put me in a
>> mashup frame of mind, so I'm currently thinking along the lines of a
>> GoogleDocs spreadsheet upload. We could then use the doc's RSS feed to
>> discover when new data is available, and automatically download it in
>> whatever format is most useful.
>
> This is what Simal is for. It doesn't currently give you RSS feeds of
> updates, although I think its a sensible feature, please add a feature
> request to http://code.google.com/p/simal/issues/list
Done: http://is.gd/jbDo
> Or data is much easier to use for mashups as it pulls all the data
> into a single usable file via a REST interface and does various
> semantic linking of data between records. Currently we only export RDF
> and JSON, but other formats made avilable on request (or patches
> welcome if you want it quick)
At the moment we're still busy gathering requirements and hashing
together prototypes for discussion. A simple file-based export of the
PIMS data is working well for that purpose, not least because we have
working code to handle it! In the fullness of time I expect we'll want
to look at moving over to using Simal's web-service interfaces. Nothing
is off the table at the moment and we're still in the very early stages
of figuring out what we're doing, so please don't take my rambling here
as being indicative of intent.
I'm sure we'll need to have further conversations with this group to
help guide our development and avoid reinventing any wheels!
ajdin....@gmail.com
The data is fine. Monthly updates are fine too. I am currently using
'Project Short Name' as the key for each project.
So far this is OK but I wander if in the future it might became
unreliable especially as the number of projects grows.
I have not used Simal yet but will try it soon.
Ajdin
On 11 Feb, 17:47, Richard King <r...@psydev.com> wrote:
> Ross Gardler wrote:
> > 2009/2/11 Richard King <r...@psydev.com>:
>
> > ...
>
> >> On regular exports, it would be good if we could build some kind of
> >> automatic system for publishing the data. Being at Dev8D has put me in a
> >> mashup frame of mind, so I'm currently thinking along the lines of a
> >> GoogleDocs spreadsheet upload. We could then use the doc's RSS feed to
> >> discover when new data is available, and automatically download it in
> >> whatever format is most useful.
>
> > This is what Simal is for. It doesn't currently give you RSS feeds of
> > updates, although I think its a sensible feature, please add a feature
ajdin....@gmail.com
Ajdin
Ross Gardler
>
> Hi all
>
> The data is fine. Monthly updates are fine too. I am currently using
> 'Project Short Name' as the key for each project.
> So far this is OK but I wander if in the future it might became
> unreliable especially as the number of projects grows.
Yes, that would be unusable. Especially if you start collecting data
from other sources, such as PROD or even other funding bodies such as
the EPSRC. SIMAL aims to handle this by using a URI for each resource,
this URI is unique to the running instance of Simal (i.e. if you run a
local version you have a URI unique to your own instance). Each Simal
record then tracks sources of data against that object (e.g. PIMS and
PROD).
When you request information from SImal you get all the information
from each resource, with the source of each data item clearly
identified. This gives you:
a) an infinitely large namespace for the entities
b) a customisable namespace (if you choose to run your own instance of Simal)
c) a way of defining a priority source of data for individual elements
(e.g. the description in PIMS is not very useful for OSS Watch who are
interested in community not academic value, thus we can provide our
own description and link through to the PIMS description for those
wanting more detail, similarly, in the case of eLearning data PROD
tends to have more complete details on RSS feeds etc.).
NOTE: the above functionality is available in a branch of 0.2, it is
yet to be ported to trunk, but it does work
> I have not used Simal yet but will try it soon.
Gavin has just put up a new demo server which is more stable than the
old at http://simal.16degrees.com.au
Keep an eye on it, in the next few days some new features on the REST
API will be added and demonstrated from the demo site.
Ross
Ross Gardler
>
> Just another question, will the data export be in .xls or .csv format?
What do you want it in?
(NB I can't influence what format the JISC release their data as, but
I can give you the SImal version of it in pretty much any format you
want, all you have to do is request and wait, or provide a patch)
Ross
ajdin....@gmail.com
>> Just another question, will the data export be in .xls or .csv format?
>What do you want it in?
I started writing a simple java tool which will update a VUE concept
map (http://vue.tufts.edu/features/index.cfm) with information from
JISC export. Since the export is in .xls format I tried saving
as .csv but the result was a bit of a mess. I'll try to use Apache
POI to read xls but I think it would be more efficient to have a
simple csv or xml.
Just out of interest I had a look at the site you suggested and
searched for AWESOME project which was found but there is very little
data on it. As I mentioned in my earlier posts some of the important
data I require (using Pims labelling):
Project Description
Project Progress State
Project Start Date
Project End Date
Output Name
Output Description
Output Progress State
Output Delivery Date
Input Name
Input Description
+
Keywords, Tags
So far this information is incomplete in Pims export as well, however
Pims export currently does contain more info.
Will your database store such information? In the end if you going to
store all Pims information and it is going to be accessible via REST
interface then it makes more sense to use your service. Semantic
linking between projects is very important for my project.
Ajdin
On 12 Feb, 18:04, Ross Gardler <rgard...@apache.org> wrote:
> 2009/2/12 <ajdin.bran...@gmail.com>:
Ross Gardler
> I started writing a simple java tool which will update a VUE concept
> map (http://vue.tufts.edu/features/index.cfm) with information from
> JISC export. Since the export is in .xls format I tried saving
> as .csv but the result was a bit of a mess. I'll try to use Apache
> POI to read xls but I think it would be more efficient to have a
> simple csv or xml.
This sounds really cool (both the concept map work and the exporting work).
> Just out of interest I had a look at the site you suggested and
> searched for AWESOME project which was found but there is very little
> data on it.
Yes, the data in the demo site, for the majority of entries, is screen
scraped from the What We Do pages of the JISC website. Therefore the
data is very incomplete. There are about 50 records that are complete,
these have been manually created.
...
> So far this information is incomplete in Pims export as well, however
> Pims export currently does contain more info.
We have not yet written the importers for the PIMS data. We have
reverse engineered a data dictionary for the data format, but we have
not started writing the code.
Lets get the data we can out of the PIMs exports and keep the pressure
up on the PIMS team to expose more data. As we start to show
increasingly useful presentations of this data it will become easier
to convince the PIMS team to put the effort into exposing more data.
However, we have to understand that the PIMS team have certain
restrictions on what data they can and can't expose. Much of their
database is privileged information and must be protected.
> Will your database store such information?
Where the data is available, yes. The goal of Simal is to collate
information from multiple locations, PIMS being just one of those
locations. For example, a software project hosted on Sourceforge or
GitHub will expose additional information from those sites, Simal will
collect this data and link it together.
In other words, if we can get the data from a reliable source then
Simal will collect it and collate it. The more people help us with
things like PIMS data importers the more effort the core team can put
into grabbing additional sources of data and exposing this data via
REST APIs.
> In the end if you going to
> store all Pims information and it is going to be accessible via REST
> interface then it makes more sense to use your service. Semantic
> linking between projects is very important for my project.
Great.
One thing for us to remember is that the JISC are very concerned that
they remain the authoritative source of data about JISC projects.
Therefore, Simal will not start collecting data from other sources
until we have an automated update of PIMS data so that we can respect
the JISCs wishes with respect to being the official data source.
With this in mind I would suggest the first port of call is for us to
work together on the PIMS data importer. Would you consider sharing
your work on a POI based data importer with the Simal project? (all
code is Apache Licenced and all copyrights remain with the original
owner, I'll explain what this means if detail if necessary)
Of course, we would be happy to share our data dictionary docs with
you (in fact we should put that in the Simal SVN) and will provide an
export in whatever format you need (we currently support JSON and
RDF).
Ross
ajdin....@gmail.com
> With this in mind I would suggest the first port of call is for us to
> work together on the PIMS data importer. Would you consider sharing
> your work on a POI based data importer with the Simal project? (all
> code is Apache Licenced and all copyrights remain with the original
> owner, I'll explain what this means if detail if necessary)
code i'll post.
I'll need to set up a download area for source, api etc.
I'll will be a simple Swing app.
>
> Of course, we would be happy to share our data dictionary docs with
> you (in fact we should put that in the Simal SVN) and will provide an
> export in whatever format you need (we currently support JSON and
> RDF).
Ajdin
On 16 Feb, 01:13, Ross Gardler <rgard...@apache.org> wrote:
> 2009/2/12 <ajdin.bran...@gmail.com>:
>
Simone Spencer
> Just another question, will the data export be in .xls or .csv format?
Simone
Assistant Programme Manager
JISC Executive, London
Email. s.sp...@jisc.ac.uk
Tel. 020 3006 6031
Mob. 07929 378139
Skype. spencer.simone
-----Original Message-----
From: simal...@googlegroups.com [mailto:simal...@googlegroups.com] On Behalf Of ajdin....@gmail.com
Sent: 17:34
To: simal-users
Subject: [simal-users] Re: Data from PIMS
>
> The data is fine. Monthly updates are fine too. I am currently using
> 'Project Short Name' as the key for each project.
> So far this is OK but I wander if in the future it might became
> unreliable especially as the number of projects grows.
>
>
Simone Spencer
>data I require (using Pims labelling):
>Project Description
>Project Progress State
>Project Start Date
>Project End Date
>Output Name
>Output Description
>Output Progress State
>Output Delivery Date
>Input Name
>Input Description
>+
>Keywords, Tags
Simone
Assistant Programme Manager
JISC Executive, London
Email. s.sp...@jisc.ac.uk
Tel. 020 3006 6031
Mob. 07929 378139
Skype. spencer.simone
-----Original Message-----
From: simal...@googlegroups.com [mailto:simal...@googlegroups.com] On Behalf Of ajdin....@gmail.com
Sent: 12 February 2009 18:44
To: simal-users
Subject: [simal-users] Re: Data from PIMS
>> Just another question, will the data export be in .xls or .csv format?
I was enquiring about Jisc data format.
map (http://vue.tufts.edu/features/index.cfm) with information from
JISC export. Since the export is in .xls format I tried saving
as .csv but the result was a bit of a mess. I'll try to use Apache
POI to read xls but I think it would be more efficient to have a
simple csv or xml.
searched for AWESOME project which was found but there is very little
data I require (using Pims labelling):
Project Description
Project Progress State
Project Start Date
Project End Date
Output Name
Output Description
Output Progress State
Output Delivery Date
Input Name
Input Description
+
Keywords, Tags
Pims export currently does contain more info.
interface then it makes more sense to use your service. Semantic
linking between projects is very important for my project.
On 12 Feb, 18:04, Ross Gardler <rgard...@apache.org> wrote:
> 2009/2/12 <ajdin.bran...@gmail.com>:
>
>
>
>
>
> (NB I can't influence what format the JISC release their data as, but
> I can give you the SImal version of it in pretty much any format you
> want, all you have to do is request and wait, or provide a patch)
>
> Ross
>
> --
> --
> Ross Gardler
>
> OSS Watch - awareness and understanding of open source software
> development and use in educationhttp://www.oss-watch.ac.uk
Ross Gardler
>
> Hi Ross
>
>> With this in mind I would suggest the first port of call is for us to
>> work together on the PIMS data importer. Would you consider sharing
>> your work on a POI based data importer with the Simal project? (all
>> code is Apache Licenced and all copyrights remain with the original
>> owner, I'll explain what this means if detail if necessary)
>
> I would be very happy to share the code. As soon as I have a working
> code i'll post.
> I'll need to set up a download area for source, api etc.
> I'll will be a simple Swing app.
Excellent. I'd recommend making it a part of the Simal project since
we are focusing on building a viable community around this project.
This will help ensure your code is maintained and enhanced by others.
For example, I imagine we'd look at extending it to work with the
FOSSMole and FLOSSMetrics databases.
I'd also suggest sharing it even before it is working. The sooner you
get input from others (like me) the sooner it becomes usable for
everyone. We can't contribute to testing and developing until the code
is available. Release early, release often.
However, if you really want to set up a separate project for this
please be sure to licence it correctly so that we can reuse it. If you
are not sure how to do this then contact in...@oss-watch.ac.uk (a JISC
funded open source advisory service for the UK education sector).
Ross
Simone Spencer
>Lets get the data we can out of the PIMs exports and keep the pressure
up >on the PIMS team to expose more data. As we start to show
increasingly >useful presentations of this data it will become easier to
convince the >PIMS team to put the effort into exposing more data.
However, we have to >understand that the PIMS team have certain
restrictions on what data they >can and can't expose. Much of their
database is privileged information and >must be protected.
are just a few fields that we can't share currently and that may change
in the future. I may have also identified some funding to create the
shared API we discussed - I'm unlikely to have a decision or go ahead
until April though. The 'PIMS team' is just me and one other person by
the way so bear with us, we have other responsibilities too!
12 February 2009 Adjin Brandic wrote:
>I am currently using 'Project Short Name' as the key for each project.
would suggest you use that to identify the PIMS records at least. I'm
sorry if I didn't include it in the last export - I will in future.
Simone
Assistant Programme Manager
JISC Executive, London
Email. s.sp...@jisc.ac.uk
Tel. 020 3006 6031
Mob. 07929 378139
Skype. spencer.simone
-----Original Message-----
From: simal...@googlegroups.com [mailto:simal...@googlegroups.com]
On Behalf Of Ross Gardler
Sent: 16 February 2009 01:14
To: simal...@googlegroups.com
Subject: [simal-users] Re: Data from PIMS
> I started writing a simple java tool which will update a VUE concept
> map (http://vue.tufts.edu/features/index.cfm) with information from
> JISC export. Since the export is in .xls format I tried saving
> as .csv but the result was a bit of a mess. I'll try to use Apache
> POI to read xls but I think it would be more efficient to have a
> simple csv or xml.
work).
> searched for AWESOME project which was found but there is very little
> data on it.
scraped from the What We Do pages of the JISC website. Therefore the
data is very incomplete. There are about 50 records that are complete,
these have been manually created.
...
> Pims export currently does contain more info.
reverse engineered a data dictionary for the data format, but we have
not started writing the code.
Lets get the data we can out of the PIMs exports and keep the pressure
up on the PIMS team to expose more data. As we start to show
increasingly useful presentations of this data it will become easier
to convince the PIMS team to put the effort into exposing more data.
However, we have to understand that the PIMS team have certain
restrictions on what data they can and can't expose. Much of their
database is privileged information and must be protected.
information from multiple locations, PIMS being just one of those
locations. For example, a software project hosted on Sourceforge or
GitHub will expose additional information from those sites, Simal will
collect this data and link it together.
In other words, if we can get the data from a reliable source then
Simal will collect it and collate it. The more people help us with
things like PIMS data importers the more effort the core team can put
into grabbing additional sources of data and exposing this data via
REST APIs.
> store all Pims information and it is going to be accessible via REST
> interface then it makes more sense to use your service. Semantic
> linking between projects is very important for my project.
One thing for us to remember is that the JISC are very concerned that
they remain the authoritative source of data about JISC projects.
Therefore, Simal will not start collecting data from other sources
until we have an automated update of PIMS data so that we can respect
the JISCs wishes with respect to being the official data source.
work together on the PIMS data importer. Would you consider sharing
your work on a POI based data importer with the Simal project? (all
code is Apache Licenced and all copyrights remain with the original
owner, I'll explain what this means if detail if necessary)
you (in fact we should put that in the Simal SVN) and will provide an
export in whatever format you need (we currently support JSON and
RDF).
Ross
ajdin....@gmail.com
For some reason I only now got notification of your two posts!!??
>Which would you prefer?
.csv would be preferred because it is quicker/lighter to process. If
it is too much hassle then .xls is fine as I'm already writing code
for xls processing.
>I don't know what you mean by Input Name and Input Description, we don't have any fields with those sorts of names.
fields I guess I got carried away a bit.
Ajdin
On 16 Feb, 10:16, "Simone Spencer" <s.spen...@jisc.ac.uk> wrote:
> 12 February 2009 ajdin.brandic wrote:
>
> >data I require (using Pims labelling):
> >Project Description
> >Project Progress State
> >Project Start Date
> >Project End Date
> >Output Name
> >Output Description
> >Output Progress State
> >Output Delivery Date
> >Input Name
> >Input Description
> >+
> >Keywords, Tags
>
> I don't know what you mean by Input Name and Input Description, we don't have any fields with those sorts of names.
>
> Simone
>
> Assistant Programme Manager
> JISC Executive, London
>
Ross Gardler
>
> 16 February 2009 Ross Gardler wrote:
>
>>Lets get the data we can out of the PIMs exports and keep the pressure
> up >on the PIMS team to expose more data.
...
> I may have also identified some funding to create the
> shared API we discussed - I'm unlikely to have a decision or go ahead
> until April though. The 'PIMS team' is just me and one other person by
> the way so bear with us, we have other responsibilities too!
This is excellent news. Please keep us informed, and if there is
anything we can do to help smooth this work let us know.
Ross
ajdin....@gmail.com
>
>
> > Hi Ross
>
> >> With this in mind I would suggest the first port of call is for us to
> >> work together on the PIMS data importer. Would you consider sharing
> >> your work on a POI based data importer with the Simal project? (all
> >> code is Apache Licenced and all copyrights remain with the original
> >> owner, I'll explain what this means if detail if necessary)
>
> > I would be very happy to share the code. As soon as I have a working
> > code i'll post.
> > I'll need to set up a download area for source, api etc.
> > I'll will be a simple Swing app.
>
> Excellent. I'd recommend making it a part of the Simal project since
> we are focusing on building a viable community around this project.
> This will help ensure your code is maintained and enhanced by others.
> For example, I imagine we'd look at extending it to work with the
> FOSSMole and FLOSSMetrics databases.
joining Simal project.
>
> I'd also suggest sharing it even before it is working. The sooner you
> get input from others (like me) the sooner it becomes usable for
> everyone. We can't contribute to testing and developing until the code
> is available. Release early, release often.
opportunity to work with you and learn off course.
Ajdin
>
> However, if you really want to set up a separate project for this
> please be sure to licence it correctly so that we can reuse it. If you