[Dspace-tech] DSpace REST API

[Hayden Young]

After several months of development, Wijiti is pleased to contribute
this reworked version of the REST API for DSpace 1.8.x.

Details and documentation are available at https://jspace.atlassian.net/
and all code is freely available via Github, the link of which is in the
documentation.

Enjoy.

--
Hayden Young
Managing Director
Wijiti Pty Ltd
p. +61 (0) 8 6398 5010
e. hayde...@wijiti.com
w. www.wijiti.com
vcard. www.wijiti.com/vcard/haydenyoung.vcf

NOTICE
This e-mail and any attachments are intended for the addressee(s) only and may be confidential. They may contain legally privileged or copyright material. You should not read, copy, use or disclose them without authorization. If you are not the intended recipient please contact the sender as soon as possible by return e-mail and then please delete both messages. This notice should not be removed.

[Gary Browne]

Hi Hayden,

I'm keen to try this out, however I am unable to install it - should it run ok on 1.8.2?

On building (mvn package), I get the following error:

[INFO] Unable to find resource 'org.dspace:dspace-rest:war:1.0' in repository central (http://repo1.maven.org/maven2)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to resolve artifact.

Missing:
----------
1) org.dspace:dspace-rest:war:1.0

Try downloading the file manually from the project website.

Then, install it using the command:
mvn install:install-file -DgroupId=org.dspace -DartifactId=dspace-rest -Dversion=1.0 -Dpackaging=war -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there:
mvn deploy:deploy-file -DgroupId=org.dspace -DartifactId=dspace-rest -Dversion=1.0 -Dpackaging=war -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

Path to dependency:
1) org.dspace.modules:rest:war:1.8.1
2) org.dspace:dspace-rest:war:1.0

----------
1 required artifact is missing.

for artifact:
org.dspace.modules:rest:war:1.8.1

from the specified remote repositories:
central (http://repo1.maven.org/maven2),
sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots)

I'm not clear on what I should do next, are you able to help please?

Thanks a lot,
Gary

> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and threat
> landscape has changed and how IT managers can respond. Discussions will
> include endpoint security, mobile security and the latest in malware threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> DSpace-tech mailing list
> DSpac...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech

[Mark Diggory]

Hayden,

Its excellent that your bringing this forward now. Can you itemize the
enhancements you've made over the original REST interface? Or
alternatively, it would be part of the contribution process to
complete the following:

For contribution, it would be ideal if you were to

1.) fork the original repository

https://github.com/DSpace/dspace-rest

2.) layer your changes on top of it in a working branch

and

3.) Provide a Pull Request back to the dspace rest repository master branch

Then we can discuss the new features in context of codebase directly.
Switch over to the dev list or JIRA issues if you want to advisement
during this process.

Best Regards,
Mark

--

Mark Diggory (Schedule a Meeting)
2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010
Esperantolaan 4, Heverlee 3001, Belgium
http://www.atmire.com

[helix84]

Hi Hayden,
I was trying to build it both as a webapp and as a module just to test
it. Here are my notes and observations so far:

1) Here's what I got after I ran mvn package for the first time
building as a webapp:

Downloading: http://source.sakaiproject.org/maven2//org/dspace/dspace-api/1.8.1/dspace-api-1.8.1.pom
[INFO] Unable to find resource 'org.dspace:dspace-api:pom:1.8.1' in
repository sakaiproject repository
(http://source.sakaiproject.org/maven2/)
Downloading: http://repo1.maven.org/maven2/org/dspace/dspace-api/1.8.1/dspace-api-1.8.1.pom
8K downloaded (dspace-api-1.8.1.pom)
Downloading: http://source.sakaiproject.org/maven2//org/sakaiproject/entitybus/entitybus-api/1.0.8/entitybus-api-1.0.8.pom
636b downloaded (entitybus-api-1.0.8.pom)
[WARNING] *** CHECKSUM FAILED - Checksum failed on download: local =
'b66699e72d4c9e4203ec8b9ae7a0a7877a64680a'; remote =
'7d1242d3a967c5291f1af8ea8501248a61ccda6c' - RETRYING
Downloading: http://source.sakaiproject.org/maven2//org/sakaiproject/entitybus/entitybus-api/1.0.8/entitybus-api-1.0.8.pom
636b downloaded (entitybus-api-1.0.8.pom)
[WARNING] *** CHECKSUM FAILED - Checksum failed on download: local =
'b66699e72d4c9e4203ec8b9ae7a0a7877a64680a'; remote =
'7d1242d3a967c5291f1af8ea8501248a61ccda6c' - IGNORING
Downloading: http://source.sakaiproject.org/maven2//org/sakaiproject/entitybus/1.0.8/entitybus-1.0.8.pom
2K downloaded (entitybus-1.0.8.pom)
[WARNING] *** CHECKSUM FAILED - Checksum failed on download: local =
'88beff8c430ca7e9de493ba8da1435c2f0816fb6'; remote =
'5455299acc1383eac8c19391da3ea7624470dc24' - RETRYING
Downloading: http://source.sakaiproject.org/maven2//org/sakaiproject/entitybus/1.0.8/entitybus-1.0.8.pom
2K downloaded (entitybus-1.0.8.pom)
[WARNING] *** CHECKSUM FAILED - Checksum failed on download: local =
'88beff8c430ca7e9de493ba8da1435c2f0816fb6'; remote =
'5455299acc1383eac8c19391da3ea7624470dc24' - IGNORING
Downloading: http://source.sakaiproject.org/maven2//org/sakaiproject/entitybus/entitybus-impl/1.0.8/entitybus-impl-1.0.8.pom
955b downloaded (entitybus-impl-1.0.8.pom)
[WARNING] *** CHECKSUM FAILED - Checksum failed on download: local =
'c9b46df256025d1030549d7038d559733d4041d4'; remote =
'76f6e7c61a137b830fdb2334bd4030df237434ef' - RETRYING
Downloading: http://source.sakaiproject.org/maven2//org/sakaiproject/entitybus/entitybus-impl/1.0.8/entitybus-impl-1.0.8.pom
955b downloaded (entitybus-impl-1.0.8.pom)
[WARNING] *** CHECKSUM FAILED - Checksum failed on download: local =
'c9b46df256025d1030549d7038d559733d4041d4'; remote =
'76f6e7c61a137b830fdb2334bd4030df237434ef' - IGNORING
Downloading: http://source.sakaiproject.org/maven2//org/sakaiproject/entitybus/entitybus-utils/1.0.8/entitybus-utils-1.0.8.pom
746b downloaded (entitybus-utils-1.0.8.pom)

(It built correctly, I'm just pointing out the checksum mismatches)

2) When I tried to build it as a module (with DSpace git master
branch), I got this:

[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Error building POM (may not be this project's POM).


Project ID: org.dspace.modules:rest
POM Location: /home/dspace/svn/DSpace/dspace/modules/rest/pom.xml
Validation Messages:

[0] 'dependencies.dependency.version' is missing for
org.dspace:dspace-rest:war


Reason: Failed to validate POM for project org.dspace.modules:rest at
/home/dspace/svn/DSpace/dspace/modules/rest/pom.xml


[INFO] ------------------------------------------------------------------------
[INFO] Trace
org.apache.maven.reactor.MavenExecutionException: Failed to validate
POM for project org.dspace.modules:rest at
/home/dspace/svn/DSpace/dspace/modules/rest/pom.xml
at org.apache.maven.DefaultMaven.getProjects(DefaultMaven.java:404)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:272)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
Caused by: org.apache.maven.project.InvalidProjectModelException:
Failed to validate POM for project org.dspace.modules:rest at
/home/dspace/svn/DSpace/dspace/modules/rest/pom.xml
at org.apache.maven.project.DefaultMavenProjectBuilder.processProjectLogic(DefaultMavenProjectBuilder.java:1077)
at org.apache.maven.project.DefaultMavenProjectBuilder.buildInternal(DefaultMavenProjectBuilder.java:880)
at org.apache.maven.project.DefaultMavenProjectBuilder.buildFromSourceFileInternal(DefaultMavenProjectBuilder.java:508)
at org.apache.maven.project.DefaultMavenProjectBuilder.build(DefaultMavenProjectBuilder.java:200)
at org.apache.maven.DefaultMaven.getProject(DefaultMaven.java:604)
at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:487)
at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:560)
at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:560)
at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:560)
at org.apache.maven.DefaultMaven.getProjects(DefaultMaven.java:391)
... 12 more

which I fixed by adding <version>1.0</version> to
/home/dspace/svn/DSpace/dspace/modules/rest/pom.xml

3) Then I get a build error like Gary described in dspace-tech, but
[dspace-src]/dspace-rest/target/dspace-rest-1.8.1.war is present, so
when I copy it manually to the webapps directory and set up a context
for it, it works.

4) I see references to DSpace 1.8.1 during the build, but it works
fine with 1.8.2 and HEAD.

5) When deploying as a module, location of dspace.cfg in
src/main/webapp/WEB-INF/web.xml also has to be edited (otherwise it
defaults to /deve/dspace/config.dspace.cfg), but this is not mentioned
in the documentation

6) /rest/users.xml without authentication lists user information which
could be considered sensitive.

My environment:

$ java -version
java version "1.6.0_18"
OpenJDK Runtime Environment (IcedTea6 1.8.13) (6b18-1.8.13-0+squeeze2)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)

$ mvn --version
Apache Maven 2.2.1 (rdebian-4)
Java version: 1.6.0_18
Java home: /usr/lib/jvm/java-6-openjdk/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux" version: "2.6.32-5-amd64" arch: "amd64" Family: "unix"

[Mark Diggory]

I will comment that we determined rather early that we want to
"disentangle" ourselves fromt he SAKAI repositories. It will be best
not to bring this additional repository configuration into maven
central.

> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________

> Dspace-devel mailing list
> Dspace...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-devel

[Steve Swinsburg]

Which part of entitybroker/entitybus are you using that relies on the Sakai maven repo? I'm a Sakai developer and can get things squared away so the artifacts go to central instead.

Cheers
Steve

Sent from my iPhone

[Ben Sheaff]

I got it to work but in pom.xml it looks like you have assumed that the folder dspace-rest has been copied into the source folder of dspace.

You have hard coded:

<parent>
<artifactId>dspace-parent</artifactId>
<groupId>org.dspace</groupId>
<version>1.8.1</version>
<relativePath>..</relativePath>
</parent>

So for me I fixed the relative path and changed the version to 1.7.1, I then got build successful.

Thanks,
Ben

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________

[Hayden Young]

Hi

Okay we'll research these issues further and come up with some solutions.

So the compiled version of the REST API does work despite the mismatches
and some build issues?

Cheers


Hayden

[Hayden Young]

Good news. We'll update the documentation.

Thanks


Hayden

[Hayden Young]

Hi Gary

Please try changing the following line in your pom.xml from:

<version>1.8.1</version>

to:

<version>1.8.2</version>

and see if that solves your issue.

Cheers


Hayden

[helix84]

On Wed, Jul 11, 2012 at 8:20 AM, Hayden Young <hayde...@wijiti.com> wrote:
> Hi
>
> Okay we'll research these issues further and come up with some solutions.
>
> So the compiled version of the REST API does work despite the mismatches and
> some build issues?

Yes, it does. I didn't find any problems so far (only the conceptual
problem with listing users I mentioned).

Regards,
~~helix84

[Peter Dietz]

I've finally gotten around to building this dspace-rest module. So here are some technical notes on my getting started. 

The only surprise was to have to implement a method in dspace-api ItemIterator.skip()

Otherwise the module/pom wiring went pretty smoothly. I've pasted my git diff's to both dspace-source, and to dspace-rest into a GitHub Gist: https://gist.github.com/3109012

Once maven compiled successfully, I edited my IntelliJ Tomcat Deploy configuration to add the dspace-rest:war.

Then, I ran into HTTP 500 errors when trying to view data from the rest api. It was as helix mentioned that by default it wants /devel/dspace/config/dspace.cfg.

To make it use my real dspace.cfg, I had to edit

dspace-rest/src/main/webapp/WEB-INF/web.xml

   <context-param>

     <param-name>dspace-config</param-name>

-    <param-value>${dspace.dir}/config/dspace.cfg</param-value>

     <!-- <param-value>/dspace/config/dspace.cfg</param-value> -->

+    <param-value>/dspace/config/dspace.cfg</param-value>

   </context-param>

Additionally, I found that hard-coding the above dspace-config setting wasn't taking, and found that I also had to comment out some directory overriding...

dspace-rest/src/main/java/org/dspace/rest/servlet/DS16DirectServlet.java

         // for dev testing only COMMENT IN WORKING ENVIRONMENT

-        if (config.contains("dspace.dir")) {

-            config = "/devel/dspace/config/dspace.cfg";

-        }

+        //if (config.contains("dspace.dir")) {

+        //    config = "/devel/dspace/config/dspace.cfg";

+        //}

After that, things started working, and I could use the REST API to get real data.

Once I have some experience with the rest-api and build some psuedo-client app then I'll hopefully have some more feedback.

But, cheers to HedTek, I'm glad that you've been able to take the initiative to make some progress on getting us closer to a usable REST API. I read the README, and there is a fair amount of limitations / known issues. So hopefully we can continue to polish what you've built here, and hopefully that can be called REST v1.0, and then as that gets used, and we have more effort around a refactoring or a rewrite, maybe we can work on a REST v2.0, especially as you mentioned some technical difficulties with working with the Sakai architecture.

I will admit that having not managed the evolution of an API between versions. I'm not sure if you would rename the API from v1 to v2 when you do underneath implementation changes (i.e. compatible), and preserve the existing endpoints and functionalities, or do you change the Rest API version number when you change the endpoints / returned-data format. I'm guessing the latter.

Peter Dietz

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________

[Gary Browne]

Hi Hayden,

Yes, like helix84 said, I can get a .war output that way which unpacks ok under Tomcat. But I'm new to REST and need some pointers on how to test it.

What I started with was issuing URIs to a browser (just manually entering them):

http://$dev-server/rest/

This results in a "describe" page being displayed. Promising, I thought. Let's try one of the entities. When I clicked on communities or communities (xml), once again I get describe pages displaying in the browser. Great.

Now if I try something like:

http://$dev-server/rest/communities

I get a blank page. The catalina log is attached. Would you have any idea what's going on here?

I even tried a simple curl script in PHP and didn't get any errors, but also produced a blank page output.

Thanks a lot,
Gary



> -----Original Message-----
> From: Hayden Young [mailto:hayde...@wijiti.com]

> Sent: Wednesday, 11 July 2012 4:17 PM
> To: dspac...@lists.sourceforge.net
> Subject: Re: [Dspace-tech] DSpace REST API
>

[Peter Dietz]

Hi Gary,

Regarding fixing your 500 error, see my previous response for how to fix it. Either my catalina.log or dspace.log (I forget which) was saying that I had my database config settings wrong instead of my /dspace/config/dspace.cfg, it was looking for a /devel/dspace/config/dspace.cfg.

Also, I think you are supposed to run create_integration_test_db.sh

======

I did spend a little bit of time trying to build some type of client user interface based off of REST. I couldn't decide if there was a "best" language/framework for doing this easily, so I just started playing with Play! Framework to make some HTTP requests to the API. It was fairly easy to get started, and it has pre-included support for parsing JSON.

A few "bugs" perhaps... ?

The documentation says there is a method to get the count of the number of total collections. https://jspace.atlassian.net/wiki/display/DSPACEAPI/GET_collections_count

However, accessing that endpoint didn't seem to do anything, except for error out that it didn't exist.

http://localhost:8080/rest/collections/count.xml

Same with communities/count.xml

When trying to dig deeper, and get the count number of items in a collection, I also ran into issues. 

collections/:id:/items/count.format

just gave me the same data as

collections/:id:/items.format

If anyone has any app they've written to help test the REST API then I'd be interested in it. (I'm thinking more an app the resembles a DSpace user interface, then a Unit test). I started to look at the GSOC 2011 Rest Client, so I suppose that could be something to look at.

Peter Dietz

[Gary Browne]

Hi Peter,

 

Somehow I missed that reply – great, that did the trick!

 

Thanks a lot,

Gary

[Hayden Young]

Hi Gary

Good to hear you got it working.

Hi Peter

Yes, there should be separate count endpoints for endpoints such as /items and /communities (E.g. /items/count and /communities/count). I'm working on a CMS-based UI for DSpace but haven't had time to revisit the Github-based API; I'm hoping to revisit in the next 4 - 6 weeks when things quieten down and will test these endpoints thoroughly. I will also update the documentation (it is currently missing docs for sort parameters from the various list endpoints).

Cheers


Hayden

This body part will be downloaded on demand.

This body part will be downloaded on demand.

[Gary Browne]

Hi Peter, Hayden and anyone else involved with this development.

 

Just wanted to say thanks to you all – it looks very promising and I think has great potential.

 

Regards,
Gary

[Peter Dietz]

I've made some progress on the DSpace-Rest Client I swore I was going to build. 

I've chosen Play! Framework to get started, and I've tossed my code thus far up on GitHub.

https://github.com/peterdietz/dspace-rest-play

The Rest-Client dspace-rest-play is very basic thus far, but I was able to successfully connect, and test out a few endpoints. Thus far, I've tested out the Community List, and a Community-Detail page. Its easy. To get the community list, just ask for that endpoint, then wrap all that data into an html list. I've made each one a link to its community-detail-page. The community page has more info like: shortDescription, introductoryText, handle, copyrightText...

I couldn't get all communities to work from the REST API. I only tested my top-level-communities, but some of them error out with: 

WARN Could not process entity: /communities/940 (404)[null]: Attempted to access an entity URL path (/communities/940.json) for an entity (/communities/940) that does not exist

Which is odd, because it actually exists. I'll need to dig in more to the dspace-rest community controller to see if I can't track down a bug. Its likely that something could be upset that I don't have the asset-store on my laptop.

Peter Dietz

[Hayden Young]

Wijiti has released the REST API for DSpace 1.8 and 3.0.

The latest update includes new features such as the ability to add new
items and delete existing items and bitstreams. Please note that the
item create feature requires a modification to the core DSpace code;
full details will be provided on the JSpace wiki page shortly with
another mailing list post to follow the when it is ready.

Details, downloads and documentation are available at
https://jspace.atlassian.net/wiki/display/DSPACEAPI.