[freebase-gridworks-dev] Custom recon plugins?

[Shawn Simister]

I noticed that there's a class called com.metaweb.gridworks.model.recon.DataExtensionReconConfig with bunch of unimplemented method stubs. Does this mean that there will be a way for developers to contribute their own reconciliation engines to Gridworks? I'd really like to be able to plug in my own recon service. Perhaps they could be loaded from their own jars with something like JPF?

Shawn

[Stefano Mazzocchi]

On 5/16/10 11:44 AM, Shawn Simister wrote:
> I noticed that there's a class called
> com.metaweb.gridworks.model.recon.DataExtensionReconConfig with bunch of
> unimplemented method stubs. Does this mean that there will be a way for
> developers to contribute their own reconciliation engines to Gridworks?

We were afraid of boiling the ocean with extensibility so anything you
see that looks remotely polymorphic might need some real polish. Here be
dragons and all that.

But we are in favor of modularity and extensibility if there is need
coming from the community and wide open to patches along those lines.

> I'd really like to be able to plug in my own recon service. Perhaps they
> could be loaded from their own jars with something like JPF

> <http://jpf.sourceforge.net>?

I've written my share of modular frameworks for java over the years and
I'm really not attached to any. So, whatever works and it's simple
enough to port to all OS configs, it's fine with me.

(there is also Apache Felix which I never used directly but I hear good
things about, although a little on the heavy side due to OSGi which is
not exactly lightweight).

--
Stefano Mazzocchi Application Catalyst
Metaweb Technologies, Inc. ste...@metaweb.com
-------------------------------------------------------------------

[Shawn Simister]

Stefano Mazzocchi wrote:
> On 5/16/10 11:44 AM, Shawn Simister wrote:
>> I noticed that there's a class called
>> com.metaweb.gridworks.model.recon.DataExtensionReconConfig with bunch of
>> unimplemented method stubs. Does this mean that there will be a way for
>> developers to contribute their own reconciliation engines to Gridworks?
>
> We were afraid of boiling the ocean with extensibility so anything you
> see that looks remotely polymorphic might need some real polish. Here
> be dragons and all that.
>
> But we are in favor of modularity and extensibility if there is need
> coming from the community and wide open to patches along those lines.
>
>> I'd really like to be able to plug in my own recon service. Perhaps they
>> could be loaded from their own jars with something like JPF
>> <http://jpf.sourceforge.net>?
>
> I've written my share of modular frameworks for java over the years
> and I'm really not attached to any. So, whatever works and it's simple
> enough to port to all OS configs, it's fine with me.
>
> (there is also Apache Felix which I never used directly but I hear
> good things about, although a little on the heavy side due to OSGi
> which is not exactly lightweight).
>

Hmm. OSGi seems like a good choice for something like Gridworks which
has so many possible extension points. However, I'm no expert on OSGi
design, but it seems like I would have to re-organize all of Gridworks
into OSGi bundles if I wanted to go that route. That's really outside
the scope of what I could accomplish on my own right now. Would it be
acceptable to start with something simple like JPF and then switch over
to OSGi when the resources are available for that sort of migration?

Shawn

[David Huynh]

Just curious but what do you see as extension points? I'm too deep in it
that I might miss some. Perhaps we can even have a wiki page just to
track these possible extension points.

> However, I'm no expert on OSGi design, but it seems like I would have
> to re-organize all of Gridworks into OSGi bundles if I wanted to go
> that route. That's really outside the scope of what I could accomplish
> on my own right now. Would it be acceptable to start with something
> simple like JPF and then switch over to OSGi when the resources are
> available for that sort of migration?

How easy will the switch be? I've used neither framework before.

David

[Shawn Simister]

Well for starters, I think custom facets and recon services would be the
most useful. But just off the top of my head I could see there also
being extensions for input/output data file formats, 3rd party data
sources like RDBMS, additional scripting languages, crowd-sourcing
interfaces like RABJ, data reporting tools, UI enhancements, etc.

>
>> However, I'm no expert on OSGi design, but it seems like I would have
>> to re-organize all of Gridworks into OSGi bundles if I wanted to go
>> that route. That's really outside the scope of what I could
>> accomplish on my own right now. Would it be acceptable to start with
>> something simple like JPF and then switch over to OSGi when the
>> resources are available for that sort of migration?
> How easy will the switch be? I've used neither framework before.
>

I'm pretty new to these frameworks as well. I'm assuming that the
plugins in each system would just be a jar file with classes that
implement some common interfaces. There probably wouldn't be too much
work involved to migrate the plugins to OSGi bundles. However, its
probably also safe to assume that all the plugins would need to be
ported over which is kind of annoying.

From the looks of it, a lot of work went into making Gridworks easy to
run on any platform and from what I understand about OSGi, that process
would have to be completely re-written so that Jetty and Gridworks were
running as OSGi bundles inside Felix with all their dependencies also
managed by the OSGi container. I'm really not that familiar with how
Gridworks is packaged so I wouldn't feel comfortable tearing it apart
and put it all back together.

If you think that the OSGi approach is feasible, I'd be happy to help
out with the reconciliation plugin part of it. =)

Shawn

[Thad Guidry]

OSGi OSGi OSGi.  I can't say enough about the bandwidth that it has now.  It even has touchstones within the SCALA camps and Groovy enterprise software.  Longterm, it finally has legs and under JSR 291.  My developers swear by it now. (Your mileage may vary).

2cents,

Thad

[Stefano Mazzocchi]

Oh boy.

Every extension point that drives different functionality requires a
specific API contract and each API contract causes rigidity because
every back-incompatible change in that contract requires an ecosystem
update (and not just merely a software update).

I would understand going the extension way if Gridworks' code was not
released and its license was not compatible with a diverse ecosystem of
contributions, but since this is not the case (intentionally so), I
would *much* rather welcome new code and functionality than extension
surfaces.

I realize perfectly well that an extension surface yields *drastically*
more peripheral innovation than a centralized submission process (see
firefox extensions).

But I also know that such surface (especially if successful) causes
immediately a vast increase in legacy inertia, so its establishment must
be carefully considered and modeled not out of thin air, but out of
several independent examples that stress that extension surface.

>>> However, I'm no expert on OSGi design, but it seems like I would have
>>> to re-organize all of Gridworks into OSGi bundles if I wanted to go
>>> that route. That's really outside the scope of what I could
>>> accomplish on my own right now. Would it be acceptable to start with
>>> something simple like JPF and then switch over to OSGi when the
>>> resources are available for that sort of migration?
>> How easy will the switch be? I've used neither framework before.
>>
> I'm pretty new to these frameworks as well. I'm assuming that the
> plugins in each system would just be a jar file with classes that
> implement some common interfaces. There probably wouldn't be too much
> work involved to migrate the plugins to OSGi bundles. However, its
> probably also safe to assume that all the plugins would need to be
> ported over which is kind of annoying.
>
> From the looks of it, a lot of work went into making Gridworks easy to
> run on any platform and from what I understand about OSGi, that process
> would have to be completely re-written so that Jetty and Gridworks were
> running as OSGi bundles inside Felix with all their dependencies also
> managed by the OSGi container. I'm really not that familiar with how
> Gridworks is packaged so I wouldn't feel comfortable tearing it apart
> and put it all back together.
>
> If you think that the OSGi approach is feasible, I'd be happy to help
> out with the reconciliation plugin part of it. =)

I'll look into it.

But for now, I'd much rather us work on improve functionality than on
improving modularity, for the reasons above.

Also, keeping a consistent interaction design becomes extremely hard
when extensions delegate that as well.

[Shawn Simister]

I agree that having that many extension points is a liability. I was
merely brainstorming to help give David an idea of where I was coming from.

Maybe I should take a step back to my original question and just look
for the quickest and easiest way to add my own recon service to
Gridworks. I'd like to find a method that lets me share my enhancements
with the community but I might have to fork my own version until a
clearer solution emerges.

Shawn

[David Huynh]

On May/16/10 8:57 pm, Shawn Simister wrote:
> I agree that having that many extension points is a liability. I was
> merely brainstorming to help give David an idea of where I was coming
> from.
>
> Maybe I should take a step back to my original question and just look
> for the quickest and easiest way to add my own recon service to
> Gridworks. I'd like to find a method that lets me share my enhancements
> with the community but I might have to fork my own version until a
> clearer solution emerges.

Thanks, Shawn. Could you talk a bit about your recon service? That
piques my interest. Is it generic or specific to your own use cases? We
could also discuss that over the other mailing list.

David

[Tom Morris]

On May 16, 2:56 pm, Stefano Mazzocchi <stef...@metaweb.com> wrote:
> On 5/16/10 11:44 AM, Shawn Simister wrote:
>
> > I'd really like to be able to plug in my own recon service. Perhaps they
> > could be loaded from their own jars with something like JPF
> > <http://jpf.sourceforge.net>?
>
> I've written my share of modular frameworks for java over the years and
> I'm really not attached to any. So, whatever works and it's simple
> enough to port to all OS configs, it's fine with me.
>
> (there is also Apache Felix which I never used directly but I hear good
> things about, although a little on the heavy side due to OSGi which is
> not exactly lightweight).

If you're going to use a full blown framework, I agree that OSGi is
the way to go. I've used it in the context of writing Eclipse
plugins, but I'm not sure how much is OSGi itself and how much is
extra Eclipse stuff that gets added on top. Basically the extension
points and extensions get declared in XML so that you can defer class
loading until things actually need to be activated.

Going with any type of framework like this doesn't imply the
requirement for an immediate decomposition to use it for the main
Gridworks app. You can have a single OSGi bundle which contains the
entire app except for the plugins/extensions which extend the provided
extension point(s).

Having said all that, I think you could get away with just defining
the extensible API(s) in Java to start with if you don't have a need
for all the dynamic load/unload, deferred loading, etc functionality.
Having a clean API, designed with an eye towards extensibility, and
known commitments to stability (even if it's "there is no commitment")
is probably more important in the short term.

Tom