atom publishing module for widget store

[Geert]

Hi all,


Currently I am performing some integration work for publishing widget profiles using ATOM.
Niels van Dijk from SURFnet (NL) asked me to. (Scott talked to him, I heard)

So, I'm in the process of creating a separate module that uses Edukapp's REST api and produces an ATOM feed, to be consumed by a container like Apache Rave.
Soon I will share this effort on Github.
(Perhaps this module could be integrated into Edukapp, but that's a bit further up the road)

While integrating with the Edukapp REST api and getting Edukapp running locally, I made some observations I would like to share.
I hope someone can comment on this.

Although it seems a long list, I did get things running quite easily, so I'm happy after all :-)

1. When running Edukapp for the first time, I get errors when calling the initialise-index-url:
GET http://localhost:8080/api/search?q=[*TO%20*]&resultsize=100000

<openjpa-2.1.1-r422266:1148538 fatal general error> org.apache.openjpa.persistence.PersistenceException: Field 'icon' doesn't have a default value {prepstmnt 21942787 INSERT INTO widgetprofiles (id, created, featured, name, updated, w3c_or_os, wid_id) VALUES (?, ?, ?, ?, ?, ?, ?) [params=?, ?, ?, ?, ?, ?, ?]} [code=1364, state=HY000] FailedObject: uk.ac.edukapp.model.Widgetprofile@3307ed org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4871)

This probably is because one of the Wookie widgets does not have an icon...

Changing the icon field in the database (alter table widgetprofiles modify icon varchar(256)) fixes this, but you probably want it the other way around: make all Wookie widgets have an icon.

2. When trying to register as a user, I also get an error of the same type:
org.apache.openjpa.lib.jdbc.ReportingSQLException: Field 'token' doesn't have a default value {prepstmnt 856916 INSERT INTO useraccount (id, email, password, salt, username) VALUES (?, ?, ?, ?, ?) [params=?, ?, ?, ?, ?]} [code=1364, state=HY000]

Again, the quick fix was to allow NULLs in the token column: alter table useraccount modify token varchar(256);

You probably want to fix this in the entity class instead.

3. Authorization seems not to be enforced in a consequent manner: For example the TagServlet allows you to add new tags unauthorized.

A oneliner that creates a new tag:

curl --data "text=an_unauthorized_tag" http://localhost:8080/api/tag

4. The REST api (using the Widgets.java JAX-RS resource) allows for ordering results:

/api/rest/widgets/search/{query}/{start}/{rows}/{orderby}

However, the code first performs the query, only sorting the results afterwards. This will always yield the same results regardless of the orderby given.

One would expect the ordering to be done first, and then limiting the number of results.

5. Another part of the REST api, the SearchServlet, does not set a Content-Type header on its JSON-response. This confuses Firefox as well as Jersey Client, for example.

Find attached the one line-patch for this.

6. Using the REST api for searching widget profiles that have certain tags, is not possible in one go: I first have to find out what ID a tag name has (using /api/tag/all (not implemented yet)) and then get all widgets having this tag by /api/tag/{id}/widgets.

Could this part of the API be improved by allowing tag names in the URL somewhere? For example: /api/tag/{tagname}/widgets.

7. What part of the API is considered 'the way to go'? I see different strategies, both in the file '/rest api calls examples' and in the implementation (Widgets.java is a JAX-RS resource, other parts are plain HttpServlets)

Should I focus on integrating with (and contributing on) "/api/rest/widgets/search/{query}/{start}/{rows}/{orderby}" or rather with "/api/search?q={search terms}"

8. I'm missing some metadata about widget profiles that we would like to see:

the author, license information, the source (although 'type' does say something). Typical fields present in an ATOM feed.

And while we're at it: 'tags' and 'activities' are not set for any widget, it seems.

Thanks for thinking along with me!

Kind regards,

Geert

[Scott Wilson]

Thanks Geert!

On 17 Jul 2012, at 12:04, Geert wrote:

> Hi all,
>
>
> Currently I am performing some integration work for publishing widget profiles using ATOM.
> Niels van Dijk from SURFnet (NL) asked me to. (Scott talked to him, I heard)

(Indeed I did!)

>
> So, I'm in the process of creating a separate module that uses Edukapp's REST api and produces an ATOM feed, to be consumed by a container like Apache Rave.
> Soon I will share this effort on Github.
> (Perhaps this module could be integrated into Edukapp, but that's a bit further up the road)

That would be very nice :)

>
> While integrating with the Edukapp REST api and getting Edukapp running locally, I made some observations I would like to share.
> I hope someone can comment on this.
>
> Although it seems a long list, I did get things running quite easily, so I'm happy after all :-)
>
> 1. When running Edukapp for the first time, I get errors when calling the initialise-index-url:
> GET http://localhost:8080/api/search?q=[*TO%20*]&resultsize=100000
>
> <openjpa-2.1.1-r422266:1148538 fatal general error> org.apache.openjpa.persistence.PersistenceException: Field 'icon' doesn't have a default value {prepstmnt 21942787 INSERT INTO widgetprofiles (id, created, featured, name, updated, w3c_or_os, wid_id) VALUES (?, ?, ?, ?, ?, ?, ?) [params=?, ?, ?, ?, ?, ?, ?]} [code=1364, state=HY000] FailedObject: uk.ac.edukapp.model.Widgetprofile@3307ed org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4871)
>
> This probably is because one of the Wookie widgets does not have an icon...
> Changing the icon field in the database (alter table widgetprofiles modify icon varchar(256)) fixes this, but you probably want it the other way around: make all Wookie widgets have an icon.

Aha... we can also provide a default icon where none is provided.

>
> 2. When trying to register as a user, I also get an error of the same type:
> org.apache.openjpa.lib.jdbc.ReportingSQLException: Field 'token' doesn't have a default value {prepstmnt 856916 INSERT INTO useraccount (id, email, password, salt, username) VALUES (?, ?, ?, ?, ?) [params=?, ?, ?, ?, ?]} [code=1364, state=HY000]
> Again, the quick fix was to allow NULLs in the token column: alter table useraccount modify token varchar(256);
> You probably want to fix this in the entity class instead.

Good catch - thanks!

>
> 3. Authorization seems not to be enforced in a consequent manner: For example the TagServlet allows you to add new tags unauthorized.
> A oneliner that creates a new tag:
> curl --data "text=an_unauthorized_tag" http://localhost:8080/api/tag

Another good one! Yes, we should use shiro for catching this.

>
> 4. The REST api (using the Widgets.java JAX-RS resource) allows for ordering results:
> /api/rest/widgets/search/{query}/{start}/{rows}/{orderby}
> However, the code first performs the query, only sorting the results afterwards. This will always yield the same results regardless of the orderby given.
> One would expect the ordering to be done first, and then limiting the number of results.
>

I'll leave that one for Kris to answer...

> 5. Another part of the REST api, the SearchServlet, does not set a Content-Type header on its JSON-response. This confuses Firefox as well as Jersey Client, for example.
> Find attached the one line-patch for this.

Thanks - I'll apply it.

>
> 6. Using the REST api for searching widget profiles that have certain tags, is not possible in one go: I first have to find out what ID a tag name has (using /api/tag/all (not implemented yet)) and then get all widgets having this tag by /api/tag/{id}/widgets.
> Could this part of the API be improved by allowing tag names in the URL somewhere? For example: /api/tag/{tagname}/widgets.

Sounds sensible, that would look nice too as a REST URL.

>
> 7. What part of the API is considered 'the way to go'? I see different strategies, both in the file '/rest api calls examples' and in the implementation (Widgets.java is a JAX-RS resource, other parts are plain HttpServlets)
> Should I focus on integrating with (and contributing on) "/api/rest/widgets/search/{query}/{start}/{rows}/{orderby}" or rather with "/api/search?q={search terms}"

For the implementation, eventually we should make the plain HttpServlets use JAX-RS too as that removes quite a bit of code, so I'd suggest focussing on JAX-RS resources. (Kris, Lucas - any view on this..?)

>
> 8. I'm missing some metadata about widget profiles that we would like to see:
> the author, license information, the source (although 'type' does say something). Typical fields present in an ATOM feed.

That sounds reasonable - not all widgets of all types will have this info, of course, but it makes sense to have these available in the widget profile class.

> And while we're at it: 'tags' and 'activities' are not set for any widget, it seems.
>
> Thanks for thinking along with me!

Thank you for contributing!

>
> Kind regards,
>
> Geert
>
> <searchservlet-contenttype.patch>

[Geert]

> So, I'm in the process of creating a separate module that uses Edukapp's REST api and produces an ATOM feed, to be consumed by a container like Apache Rave.
> Soon I will share this effort on Github.
> (Perhaps this module could be integrated into Edukapp, but that's a bit further up the road)

That would be very nice :)

The module is - for now - a Dropwizard-based [1] project, hosted at Github:

https://github.com/gvanderploeg/widget-publisher

Refer to the README for basic build/install/test instructions.

One note though: as the currently running Edukapp-demo at http://widgets.open.ac.uk:8080/ does not yet contain the fix for the Content-Type-header, the default configuration does not work at the moment.

Run the trunk-version of Edukapp locally and then point widget-publisher to it, using the uri defined in /widget-publisher.yml

By the way: I used Dropwizard to get things up and running quick.

It's not my intention to keep it that way if this module gets integrated into Edukapp. It'll run just fine as a plain jax-rs resource within Edukapp.

Next thing on my agenda is federated authentication with Edukapp. I'll keep you posted.

Regards,

Geert

[1] http://dropwizard.codahale.com/

[Scott Wilson]

On 19 Jul 2012, at 09:29, Geert wrote:

> So, I'm in the process of creating a separate module that uses Edukapp's REST api and produces an ATOM feed, to be consumed by a container like Apache Rave.
> Soon I will share this effort on Github.
> (Perhaps this module could be integrated into Edukapp, but that's a bit further up the road)

That would be very nice :)

The module is - for now - a Dropwizard-based [1] project, hosted at Github:

https://github.com/gvanderploeg/widget-publisher

Refer to the README for basic build/install/test instructions.

One note though: as the currently running Edukapp-demo at http://widgets.open.ac.uk:8080/ does not yet contain the fix for the Content-Type-header, the default configuration does not work at the moment.

Run the trunk-version of Edukapp locally and then point widget-publisher to it, using the uri defined in /widget-publisher.yml

I just gave it a go - it works very nicely :)

Btw I added the License and Author fields to Edukapp, and have sent a pull request including these in the publisher (at least, in the model so it processes the input)

By the way: I used Dropwizard to get things up and running quick.

It's not my intention to keep it that way if this module gets integrated into Edukapp. It'll run just fine as a plain jax-rs resource within Edukapp.

That would be good.

(dropwizard does look interesting... I'd used Yammer before but didn't know about dropwizard)