Plugins and a market place

48 views
Skip to first unread message

Matt Casters

unread,
Nov 29, 2010, 7:45:33 AM11/29/10
to Kettle Developers mailing list
Dear Kettle devs,

For a while now we've been thinking about creating a market place for Pentaho related plugins.  
The constant influx of new steps, job entries and Spoon plugins makes it increasingly hard to do proper dependency management (see previous mail).
One solution could be to push away more and more of these steps and job entries to external plugins.  However, we fear that developers might see those as second class citizens compared to "core" components of Kettle. Developers might think that for the following reasons:
1) Plugins are listed on a wiki page but otherwise not structurally described.
2) Installation of the plugins is manual process and technical of nature, you have to take into account versions, and so on.
3) Communication methods with the plugin authors (e-mail, JIRA, forum) are not always clear. (usually not!)
4) The plugins are not maintained (code fixes, translation/i18n, code migration, ...) the same way as the Kettle core codebase.

I strongly think that having a market in place that would do away with all these problems would in fact promote the creation of new plugins even more.

In short, Pentaho is looking for ways to make it easier to integrate new functionality in Kettle without having to resort to the monolithic structure we have in place now.  Any ideas you might have in this respect are most welcome.

Regards,

Matt
--
Matt Casters <mcas...@pentaho.org>
Chief Data Integration, Kettle founder, Author of Pentaho Kettle Solutions (Wiley)
Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
Pentaho : The Commercial Open Source Alternative for Business Intelligence


Roland Bouman

unread,
Nov 29, 2010, 7:51:54 AM11/29/10
to kettle-d...@googlegroups.com
Hi All!

Overall, a good idea.
I do think that to make it work, there'd have to be clear admission
criteria, and an independent board / comittee to judge submissions
against these criteria.
Feedback to devs should also be in terms of those criteria in case
their submission is currently not admitted.

One of the criteria I would suggest is availability of freely
available documentation (so it can be linked to from the marketplace
site).
I would also suggest criteria for the content and structure of the
documentation.
The value in this is to make it easy for potential users to discover
whether it makes sense for them to use a a particular plugin.

kind regards,

Roland

> --
> You received this message because you are subscribed to the Google Groups
> "kettle-developers" group.
> To post to this group, send email to kettle-d...@googlegroups.com.
> To unsubscribe from this group, send email to
> kettle-develop...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/kettle-developers?hl=en.
>

--
Roland Bouman
blog: http://rpbouman.blogspot.com/
twitter: @rolandbouman

Author of "Pentaho Solutions: Business Intelligence and Data
Warehousing with Pentaho and MySQL",
http://tinyurl.com/lvxa88 (Wiley, ISBN: 978-0-470-48432-6)

Author of "Pentaho Kettle Solutions: Building Open Source ETL
Solutions with Pentaho Data Integration",
http://tinyurl.com/33r7a8m (Wiley, ISBN: 978-0-470-63517-9)

Matt Casters

unread,
Nov 29, 2010, 8:05:39 AM11/29/10
to kettle-d...@googlegroups.com
Hi Roland,

I'm not sure I would want to be restrictive at all.  I want people to have as much fun with the plugins as possible.  I want to have experimental, undocumented stuff in the market. I want that since I strongly believe that's how open source comes to the best results.  However, I also would expect to have a system in place that would allow me to vote plugins up or down from within Spoon, integrated problem reporting &JIRA, mini-forums for feedback, and so on.

Having integrated documentation however is a good point.  Giving a plugin a dedicated wiki page to drop the docs into is a good idea.  Forcing people to document is IMO over the top.  Perhaps we should create a "Vote down: no documentation" option somewhere.  

All in all, if someone contributes a plugin to calculate Pi in arbitrary position to 1M decimals, we can safely say that this is something we would like to keep as a plugin because the use-case is a bit scientific/specific.  However, for a lot of other plugins that are imaginable in the data generation, data quality, data preparation areas, that's not clear at all.  Myself nor the folks at Pentaho, nor any committee or board I can think of would want to make that decision either. 

Personally I kinda like the Android market idea where folks can rate and review apps.  Perhaps we can combine that open "anyone can contribute" idea with a lot of infrastructure to facilitate two-way-communications, development, translation, documentation and source code access.

Just my two Euro-cents,
Matt

2010/11/29 Roland Bouman <roland...@gmail.com>

Roland Bouman

unread,
Nov 29, 2010, 8:23:21 AM11/29/10
to kettle-d...@googlegroups.com
Hi Matt, all,

you raise a number of good points I hadn't thought about.

My reply mostly reflects my own experiences hunting for plugins.

With a few exceptions (web browser plugins), whether its CMS-related
(drupal), web development (jQuery), I find I spend oodles of time
downloading and trying plugins just to find out what they actually do.
In many cases, documentation is so sparse that it's hard to even
comprehend what the function of the plugin is. Just a little bit of
documentation (screenshot + typical example) would remedey say 50% of
those problems.

But indeed maybe documentation criteria is a bit over the top. The
voting system may indeed work as a filter - thanks for pointing that
out.

regards,

Roland

Brandon Jackson

unread,
Nov 29, 2010, 8:34:20 AM11/29/10
to kettle-d...@googlegroups.com
You'll have to catch the plugin at the source where developers feed for information.  That could be a dev list registration like this, or the pentaho forums.   If there was a simple mechanism to 'request' a GUID for use in a plugin, then you could get an idea of how many are being attempted.   The same GUID combined with a plugin.xml could give some catalog of plugins.  The XML file could have a URLs inside that show a preferred link where 'loved' plugins could reside with a pentaho controlled URL, other URLs like 'author home' could point to at least another source.

If each plugin had a GUID, then maybe an option in spoon would be to scan and report which GUIDs are used.   A context menu on a step or job and allow voting up or down on versions of plugins, much like iTunes.

It would probably be cool if there were a utility that could scan all the jobs and transformations just collecting the GUIDs and occurrence counts.  (Popularity).  You would especially want to make this option available on first run of a new version.  (May reveal cracks before first run of a new version.)

Jon Roberts

unread,
Nov 29, 2010, 9:48:07 AM11/29/10
to kettle-d...@googlegroups.com
While a plugin marketplace could help with some of the dependency issues with better documentation,  I suspect that the issue will persist.

A quick search reveals what you might expect.  Dependency problems are not unique to kettle.


Some type of packaging, central repositories, and associated management tools are usually employed to handle this.  There are lots of examples for both OS (rpm, dpk) and programming environments (CPAN, pip - eggs - pypm).  Given the widespread adoption of java and eclipse both, I'm somewhat surprised that a standard has not already emerged for this?  Maybe there is one and I'm just unaware of it?

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-d...@googlegroups.com.
To unsubscribe from this group, send email to kettle-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.



--
Sincerely,

Jon Roberts

Matt Casters

unread,
Nov 29, 2010, 10:07:14 AM11/29/10
to kettle-d...@googlegroups.com
Actually, Kettle plugins are loaded by separate class loaders.  As such, a solid plugin system like a market would indeed lower dependencies since you don't have to ship with libfoo-2.1.3.jar in the main Kettle distribution.  The jar file gets isolated and is loaded for a single specific plugin. 

We don't have to overstate the dependency issues we have now either.  Usually these are minor things like upgrading a commons library from 3.1.2 to 3.1.3 or going from POI 3.6 to 3.7.  The problem is mainly located in the area of knowing which libraries are in conflict with other projects, not the resolution as such.  For example, you want to have an answer to questions like: I want to get data from a Kettle transformation in report designer.  Which libraries do I need, which of those do I already have in PRD and which are in conflict?

Take care,

Matt

2010/11/29 Jon Roberts <bigjon...@gmail.com>

Tony Cook

unread,
Nov 29, 2010, 11:22:52 AM11/29/10
to kettle-d...@googlegroups.com
Now this is a good idea!  The opportunities for PDI as a “platform” are increased by an order of magnitude with the addition of a plugin marketplace.

The PDI community does a great job of extending PDI through new steps and job entries but the core product will only get bigger and bigger over time.  A marketplace would allow the core product to keep maturing, while making it significantly more extensible (like Firefox).

There is also a second benefit – the opportunity for 3rd party developers to build specialist plugins – so users can have new functionality but with the reassurance of commercial support.  Okay – 3rd party developers can already build plugins – but it is difficult to “market” them if you can’t somehow tap into the PDI user base.  A marketplace would solve that problem - and commercial developer marketplaces seem to work well for some major platforms out there.

Just as a bit of background, I’ve set up a small dev company and we're busy building PDI plugins.  We’ve got about 4 or 5 fairly interesting plugins currently in the pipeline and we’ll be releasing these shortly – some open source, some with commercial support.  So I would definitely like to see a plugin marketplace for PDI and we’d be willing to help build it!

Here are some ideas about how I think a marketplace could work...

* A central repository to store the plugins and any dependant jars.  SVN would probably do the job for this.  (like eclipse, I think?).

* Ability for anyone to contribute, but some kind of gatekeeper control, e.g. SVN commit access for “approved” developers.

* An XML file for plugin developers to publish all the related information for their plugin, e.g. documentation link, plugin home page, developer contact details, etc.  Maybe just extend plugin.xml?

* A database of dynamic additional plugin info, e.g. votes, bug reports, etc.  This would help plugin developers connect with the community (this is a "nice to have").

* A GUI within PDI to discover/browse plugins.  Kind of like the Firefox Add-ons GUI, but with Android marketplace extra features, e.g. voting.  Maybe someone could also build a web version like there is for Android & Firefox.

However, it is always easy with great ideas like this to build long lists of requirements for the “perfect system” but IMHO it is always better to get something up and running.  Maybe a bare-bones plugin system would just need a GUI interface within PDI which retrieves plugin listings from a known SVN location...could it be as easy as that to get started?

Oh, did I mention that I thought this was a good idea?

Tony Cook
Cloud2Land.com

Mark Hall

unread,
Nov 29, 2010, 2:48:34 PM11/29/10
to kettle-d...@googlegroups.com
The Weka project went through just this exercise with the last release
of the development version in late July. I introduced a new package
management system, based heavily on what the R statistical software
uses. All contributions were split out of the core of Weka along with
many core algorithms and made into packages. The whole shebang is
hosted on Sourceforge - this includes a central package meta data
repository and the actual packages themselves. Weka allows for
dependencies between not only the base system but other packages too
(something that Kettle doesn't concern itself with given its separate
class loader per plugin approach) and the package management system
takes care of handling dependencies and detecting conflicts. Users can
use either a command line or GUI client to download and install
packages.

We have a concept of "official" and "unofficial" packages for Weka.
Official packages get their meta data hosted at the central repository
(the package itself can reside anywhere that is http accessible) and
are verified by the Weka team (to the extent that the package
functions and does not contain malware etc.). Unofficial packages
don't get their meta data hosted at the central repository and can be
installed by just pointing the package manager at their URL. The
unofficial route allows folks to quickly make (possibly) experimental
stuff available without having to go through the official channel.

The main motivation for the Weka project was to reduce the burden of
supporting external contributions. It makes a clear delineation
between what is provided by the core Weka team and what is a
contribution. A secondary goal was to open up Weka to the use of
third-party libraries (something that has been avoided like the plague
in the past :-)). Third-party library related issues/problems are now
clearly the domain of package contributers who include them in their
packages.

This has all been working just swimmingly since the release. Packages
have been downloaded via the package manager more than 33,000 times
since late July.

The package manager code is independent of Weka and is checked in to
Pentaho's public SVN.

For those who are interested, the central repository is at:
http://weka.sourceforge.net/packageMetaData/

Information on the clients and package structure is at:
http://weka.wikispaces.com/How+do+I+use+the+package+manager%3F
http://weka.wikispaces.com/How+are+packages+structured+for+the+package+management+system%3F

Cheers,
Mark.

codeoncoffee

unread,
Nov 29, 2010, 7:57:26 PM11/29/10
to kettle-developers
Actually there is a standard for this. It's the underpinning of the
Eclipse plugin system, OSGI:
http://en.wikipedia.org/wiki/OSGi
http://www.osgi.org/About/WhyOSGi

The Eclipse project migrated to OSGI in v3 and maintains what's
considered the reference implementation in Equinox ( http://www.eclipse.org/equinox/
). Recently many other companies and projects have migrated to OSGI
themselves or provided support for it including: JBoss, WebShere,
GlassFish, Netbeans, Weblogic, SpringSource, Apache Sling... and the
list goes on.

OSGI provides a component architecture framework for applications
based on plain Jars with runtime dependency information contained as
metadata in the jar manifests. The OSGI runtime manages wiring up the
component's (Bundles) with needed library classes and provides several
ways for services to be shared and tracked between bundles.

There's even built-in support for downloading dependencies over the
internet via a Bundle Repository (OBR) such as SpringSource's (
http://ebr.springsource.com/repository/app/bundle ) or though a Maven
repository. If you've installed a plugin in Eclipse you've seen OSGI
in action. You can install/start/stop/restart a plugin without
bringing down the application.

There's quite a large OSGI community out there with a great number of
bundles to take advantage of and resources for developers. And they're
not light-weights. Last I checked there are around 10 JSRs based on
OSGI. All of the major IDEs support development of Bundles and there
excellent plugins for Ant and Maven to support building. But as I
mentioned, they're just Jars. It doesn't take much to get started.

If you can't tell I'm a big proponent :). Pentaho has large number of
plugin/service/configuration systems, all different and incomplete. As
we further integrate our products we're duplicating a lot of
integration code (plugin wiring, initialization, configuration) and
maintaining and extending the various systems in parallel. All that
could be standardized under a unified component (plugin) system.

For the past couple months I've been leading a group here
investigating just this thing. We decided Kettle would be the natural
place to investigate an OSGI migration. It's plugin system is quite
similar in many ways and it has a very rich component set with many
plugins. Also it touches just about all of our technologies and
projects :) I just recently checked-in the first code:

OSGI-ified version of Kettle with preliminary support for Bundle
provided steps checked-in at:
(svn://source.pentaho.org/svnkettleroot/Kettle/branches/osgi)

This past week I worked on converting over the MaxMind Lookup step.
It's also checked-in:
(svn://source.pentaho.org/svnkettleroot/plugins/MaxMindGeoIPLookup/
branches/osgi)

This effort is nowhere near complete and is provided as a test-bed for
the technology. There is wiki page up outlining the new OSGI-based
plugin system and I am looking for feedback!:
http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle

While this is all preliminary, I'm scheduled to host a brown-bag for
the Orlando developers sharing what I've learning. Doug Moran has also
asked me to present this formally to the Kettle Devs in an upcoming
community WebEx as a potential way forward.

Cheers,
Nick Baker



On Nov 29, 9:48 am, Jon Roberts <bigjonrobe...@gmail.com> wrote:
> While a plugin marketplace could help with some of the dependency issues
> with better documentation,  I suspect that the issue will persist.
>
> A quick search reveals what you might expect.  Dependency problems are not
> unique to kettle.http://www.google.com/search?rlz=1C1GPCK_enUS342US342&sourceid=chrome...
>
> <http://www.google.com/search?rlz=1C1GPCK_enUS342US342&sourceid=chrome...>Some
> > Matt Casters <mcast...@pentaho.org>
> > Chief Data Integration, Kettle founder, Author of Pentaho Kettle Solutions<http://www.amazon.com/Pentaho-Kettle-Solutions-Building-Integration/d...>
> >  (Wiley<http://eu.wiley.com/WileyCDA/WileyTitle/productCd-0470635177.html>
> > )
> > Fonteinstraat 70, 9400 OKEGEM - Belgium - Cell : +32 486 97 29 37
> > Pentaho : The Commercial Open Source Alternative for Business Intelligence
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "kettle-developers" group.
> > To post to this group, send email to kettle-d...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > kettle-develop...@googlegroups.com<kettle-developers%2Bunsubscr i...@googlegroups.com>
> > .

Nicholas Goodman

unread,
Nov 29, 2010, 8:52:13 PM11/29/10
to kettle-d...@googlegroups.com
On Nov 29, 2010, at 4:57 PM, codeoncoffee wrote:

This effort is nowhere near complete and is provided as a test-bed for
the technology. There is wiki page up outlining the new OSGI-based
plugin system and I am looking for feedback!:
http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle

OSGi is a great choice.  Hazaa!  

Nick

dmoran

unread,
Nov 30, 2010, 12:06:12 PM11/30/10
to kettle-developers
For the popularity issue I think the more we can automate the better.
We could add a heartbeat that reports stats back and is available with
the download that shows how many times it has been installed,
uninstalled and run - that would also help with the niche type plugins
where total downloads is small, but people who do use it - use it a
lot. The key to success in something like this is automation and low
touch.

Tony Cook

unread,
Nov 30, 2010, 1:22:12 PM11/30/10
to kettle-d...@googlegroups.com
Reading through the OSGI stuff and looking at the sample code, even though I don't fully understand it, it looks well thought through and I can sense the enthusiasm radiating from Orlando!

I am just wondering if PDI really needs this ability to install modules dynamically without a restart.  Currently to install a plugin you just have to copy the files (and any required jars) into the relevant plugin folder and then restart Spoon.  The benefit of a marketplace is to aid "discovery" by users, but I'm sure a user would be more than happy to restart after installing a plugin (like with Firefox).

Unless there is a use-case of PDI that requires it to be permanently running?  Would this use case need to install/uninstall plugins at run time with no possibility of a restart?

Tony


--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-d...@googlegroups.com.
To unsubscribe from this group, send email to kettle-develop...@googlegroups.com.

Jon Roberts

unread,
Nov 30, 2010, 1:30:44 PM11/30/10
to kettle-d...@googlegroups.com
Yes, 

Installations that are using Carte servers and a repository as an enterprise-level service could benefit from installation without a restart.
--
Sincerely,

Jon Roberts

Peter Hunsberger

unread,
Nov 30, 2010, 1:34:01 PM11/30/10
to kettle-d...@googlegroups.com
On Tue, Nov 30, 2010 at 12:22 PM, Tony Cook <to...@cloud2land.com> wrote:
>
> Reading through the OSGI stuff and looking at the sample code, even though I don't fully understand it, it looks well thought through and I can sense the enthusiasm radiating from Orlando!
>
> I am just wondering if PDI really needs this ability to install modules dynamically without a restart.  Currently to install a plugin you just have to copy the files (and any required jars) into the relevant plugin folder and then restart Spoon.  The benefit of a marketplace is to aid "discovery" by users, but I'm sure a user would be more than happy to restart after installing a plugin (like with Firefox).
>
> Unless there is a use-case of PDI that requires it to be permanently running?  Would this use case need to install/uninstall plugins at run time with no possibility of a restart?
>

Not sure of the answer for the normal use case, but two thoughts:

1) is there any reason _not_ to have the capability to dynamically
install plugins? If an OSGI container works without too high of cost
(and from my experience that should be true) why not use it?

2) I sorta have a vauge idea of using PDI for the configuration /
management of some parts of the back end of an app server that needs
to be able to dynamically add new interfaces to many strange and weird
external data sources. The ability for it to play in the OSGI world
in general may open it up to many new and exciting use cases?

--
Peter Hunsberger

codeoncoffee

unread,
Nov 30, 2010, 1:38:34 PM11/30/10
to kettle-developers
The ability to install without a restart is just a side-effect of the
environment if properly setup. There is extra effort required to track
these comings and goings and I agree that it's not necessary, despite
the coolness factor

-Nick Baker
> > kettle-develop...@googlegroups.com<kettle-developers%2Bunsubscr i...@googlegroups.com>
> > .

dmoran

unread,
Nov 30, 2010, 1:40:53 PM11/30/10
to kettle-developers

Tony, I agree it might not be an absolute must have for spoon but it
does add to the overall user experience. Especially if you are in the
middle of developing a transform and realize you need a new step. It
sure would be nice to be able to grab it and keep going. I'd like to
see it as little more than another pallet to choose from. Another
interesting case would be if you tried to open a transform that you
didn't have the step or steps for, it would be sweet to have a prompt
that says, "This transform requires the CoolAndNewStep, do you want to
install it now? [yes] [no] [more info...]"

We should strive for seamless, especially if it comes with the
framework.

Matt Casters

unread,
Nov 30, 2010, 2:25:03 PM11/30/10
to kettle-d...@googlegroups.com

Kettle is traditionally open to change and open to contributions. This fact has not gone unnoticed even by our own developers at Pentaho.
I like the fact that Kettle is being used to experiment with OSGi since it's a testament to a simple and straightforward architecture and I'm sure we'll all welcome the contribution the once it's done.  Nick's aim is to make this not a replacement to the current plugin architecture (that Nick & I re-wrote in v4) but to be complimentary.

However, Tony's point remains valid. OSGi is currently a technical solution looking for a problem that Kettle might not even have.  If we would currently have a requirement to install a plugin without the need to restart Spoon, I'm sure it would fairly easy to do. The Kettle plugins are already dynamically loaded after all.

In other words, let's retain your user experience requirements and add them to the rest and at the end we can see where we end up. Only then should we determine whether or not OSGi is needed for Kettle and/or for the rest of the Pentaho stack.

In addition to the technical capabilities and the things we want as developers, there might be features that are not always desirable.  One of the "features" I seriously dislike about certain frameworks (Eclipse, Talend and all) is the ability to automatically update.  While technically a great idea, it presents some challenges as far as conformance and testing are concerned.  Likewise, whatever plugin or market system we use, versioning is something we need to do very carefully.  If we support automatic updates, allow for companies to turn it off or manage it centrally. Perhaps we want to support development, test, UAT and production-like environments for certain plugins?  For other plugins a simple major, minor, point-version system would work fine.  The plugins that are downloaded in Spoon, do we automatically want to deploy them in other locations too, signal Carte/DI Servers to download the same package?  What about firewalls?  Can we tell Spoon to download plugins from a shared folder or local server?  

There are many important questions that need to be asked and answered at this stage.  

If we don't ask these questions, and we would adopt Nicks work at this time, we would be putting a lot pressure on Nick to bend to the will of a constantly changing set of requirements and that hardly seems fair.  All I'm saying is that the better the requirements we can gather are, before choices have been made, the easier it will be to phase the development and execute upon it later. 

Finally, it might make sense to have the exact same discussion in the rest of the Pentaho projects.  I'm sure this would be a killer feature for the BI server & report designer as well.  

Thanks for caring!

Matt


2010/11/30 dmoran <dmo...@pentaho.org>
--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-d...@googlegroups.com.
To unsubscribe from this group, send email to kettle-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.




--
Matt Casters <mcas...@pentaho.org>
Chief Data Integration, Kettle founder, Author of Pentaho Kettle Solutions (Wiley)

dmoran

unread,
Nov 30, 2010, 2:39:47 PM11/30/10
to kettle-developers
I agree with Matt 108%

codeoncoffee

unread,
Nov 30, 2010, 3:19:46 PM11/30/10
to kettle-developers
I agree with you Matt. OSGI provides a wealth of options that aren't
all desirable at the same time. Sitting down and deciding how to
handle installation and configuration of modules across all of the
platform is key to determining the direction to take.

Last night I got the MaxMind step installed in Spoon by connecting to
a Bundle Repository running on my box. It automatically installed the
bundle I told it to and it's needed dependencies. Though this happened
over the web it could have installed the files from a shared place on
the filesystem or even a Maven repo. Pretty neat, plus I didn't write
any code.

How to handle system configuration for deployment is absolutely key.
One of the great features of Apache Aries and Apache Karaf (just
bundles you can add) is they allow you to work with "assemblies" of
bundles. These can be locked configurations of the system installed in
bulk. This may be a good approach to production deployment. There's
just too many options to list and I'm discovering more as I continue
this work.

One thing to keep in mind with the OSGI environment is that our users
can customize the entire deployment strategy by using existing OSGI
bundles available in the community. The Karaf extensions allow you to
manage SSHing directly into your running application and stop/start/
install/etc. the running Bundles. I certainly wouldn't ship a system
like this but could imagine wanting to do this as a user.

I've been running Spoon with the Apache Felix Web Console which embeds
a Jetty server running in your application with a web-app front-end to
manage the system. Certainly a lot of overhead for configuring Spoon,
but again, many options.

I would love to schedule some time to discuss the needs of our
individual products and the platform as a whole to try to find a
unified approach. We were hoping to get further along and learn a lot
more before presenting this initiative. I just couldn't wait anymore
when I heard of this Marketplace idea!

My immediate work will be on a walkthrough for creating a OSGI-
provided steps focused on step developers, but if anyone has any
specific questions or would like some pointers to examples, let me
know.

-Nick B
> > kettle-develop...@googlegroups.com<kettle-developers%2Bunsubscr i...@googlegroups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/kettle-developers?hl=en.
>
> --
> Matt Casters <mcast...@pentaho.org>
> Chief Data Integration, Kettle founder, Author of Pentaho Kettle

Nicholas Goodman

unread,
Dec 15, 2010, 1:59:30 PM12/15/10
to kettle-d...@googlegroups.com
On Nov 30, 2010, at 12:19 PM, codeoncoffee wrote:

> I agree with you Matt. OSGI provides a wealth of options that aren't
> all desirable at the same time. Sitting down and deciding how to

With OSGi you get many things "for free" just by using it. The other thing one gets is lots of other smart people sorting out complicated issues like applications with depedencies across class loaders, etc.

For instance, in a service engagement we're building a set of PDI Job plugins. The plugins all have some common functionality (base class) and then we extend them for the specific fcnality. Given the current plugin classloading mechanism, we can't actually install all this as a plugin.

You can only have 1 plugin per plugin.xml, and each plugin has it's own classloader. Currently, you can't have a package (with a base class) either duplicated across all plugins (you'll get strange errors like Object X cannot be cast to Object X) or only in one (class not found from one plugin to another). The only way currently to solve this problem is to include the base classes, which are really part of the plugins, in the kettle libext and then have the individual plugins extended from that classloader.

So... there's one piece of functionality that OSGi solves well. The ability for plugins to build "on top of each other" instead of just being able to build upon core Kettle. Enough to warrant a whole framework? ;) Maybe not. All the same, use cases for the way OSGi handles these types of things are there.

Nick

Matt Casters

unread,
Dec 15, 2010, 2:05:31 PM12/15/10
to kettle-d...@googlegroups.com
Hi Nick,

Your assertions are fortunately not true.
Have a look at the KFF source code (http://is.gd/iNJ0b) for an example on how you can easily deploy a series of plugins.
In that particular case we deploy 8 plugins (job entries and steps) from a single jar file.
Shared dependency jar files can be placed in an adjacent lib/ folder.
And best of all: no XML is involved ;-)

Cheers,

Matt

2010/12/15 Nicholas Goodman <ngoo...@bayontechnologies.com>
--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-d...@googlegroups.com.
To unsubscribe from this group, send email to kettle-develop...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.




--
Matt Casters <mcas...@pentaho.org>
Chief Data Integration, Kettle founder, Author of Pentaho Kettle Solutions (Wiley)

Nicholas Goodman

unread,
Dec 15, 2010, 2:19:59 PM12/15/10
to kettle-d...@googlegroups.com

On Dec 15, 2010, at 11:05 AM, Matt Casters wrote:

> Your assertions are fortunately not true.

Love it when I'm wrong and it means a limitation isn't actually!!!

> And best of all: no XML is involved ;-)

Yeah... I saw some notes in the plugin loader that this was only for testing... Double checking looks like the general classpath search was the one that's marked as testing only. I got the sense that the annotations was the slightly off color method and that the plugin.xml was the primary way to build plugins. Are annotations now the preferred method?

Does the annotation method bring in all annotated plugins into separate classloaders? ie, with XML you get one classloader per plugin. Now, it would seem you get either
1) a single classloader for all annotated plugins
-or-
2) a single classloader per loaded .jar in plugins/ dir

This is great news! Makes my day! Multiple plugins in a single jar! :)

Nick

Matt Casters

unread,
Dec 15, 2010, 2:28:44 PM12/15/10
to kettle-d...@googlegroups.com

Nick, there is no "preferred" method as we decided to support both methods of plugin development.
However, since v4 it's obviously a lot more convenient to use annotations for its ease of use.
There was a slight drawback to developing in Eclipse as documented over here: http://is.gd/iNMn1
However, even that beats restarting Spoon a few hundred times to get a plugin developed ;-)
The ease of use of just being able to unzip a file in plugins/ or drop a jar file in there was worth the effort we put into it I think.

Plugins are added to the collective by whichever way works for you: plugin.xml, annotations, kettle-steps.xml, kettle-jobentries.xml and one day OSGi.  From there on they play their role the same like all steps and job entries.
And yes, they do indeed all use separate class loaders (Except for the SAP Input step I believe)

Cheers,
Matt

2010/12/15 Nicholas Goodman <ngoo...@bayontechnologies.com>


Nick

--
You received this message because you are subscribed to the Google Groups "kettle-developers" group.
To post to this group, send email to kettle-d...@googlegroups.com.
To unsubscribe from this group, send email to kettle-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/kettle-developers?hl=en.

codeoncoffee

unread,
Dec 21, 2010, 8:55:13 AM12/21/10
to kettle-developers
Quick update. I've just committed a revised approach to getting OSGI
in Kettle. This new implementation is completely hidden behind the
current PluginRegistry api. With this commit, OSGI is just another
avenue to getting your plugin registered with the current system.

I will be adding a listener mechanism to the PluginRegistry so Kettle
can become aware as plugins come and go and react appropriately. This
provides the hot-swapping and loading.

The rest of the functionality added by OSGI will be available to the
plugin developers through traditional OSGI APIs. The MaxMind example
step uses the Blueprint beans approach right now. So far I'm only
registering Spoon as a service object but others will surely come.

I'm reasonably satisfied with this current effort. I'd like to see
more types proven-out other than steps, scheduler interface for
instance would be a nice demo. And the current directory watching
system needs to be replaced with Apache Karaf or just removed.

-Nick B.

On Dec 15, 2:28 pm, Matt Casters <mcast...@pentaho.org> wrote:
> Nick, there is no "preferred" method as we decided to support both methods
> of plugin development.
> However, since v4 it's obviously a lot more convenient to use annotations
> for its ease of use.
> There was a slight drawback to developing in Eclipse as documented over
> here:http://is.gd/iNMn1
> However, even that beats restarting Spoon a few hundred times to get a
> plugin developed ;-)
> The ease of use of just being able to unzip a file in plugins/ or drop a jar
> file in there was worth the effort we put into it I think.
>
> Plugins are added to the collective by whichever way works for you:
> plugin.xml, annotations, kettle-steps.xml, kettle-jobentries.xml and one day
> OSGi.  From there on they play their role the same like all steps and job
> entries.
> And yes, they do indeed all use separate class loaders (Except for the SAP
> Input step I believe)
>
> Cheers,
> Matt
>
> 2010/12/15 Nicholas Goodman <ngood...@bayontechnologies.com>
>
>
>
>
>
>
>
> > On Dec 15, 2010, at 11:05 AM, Matt Casters wrote:
>
> > > Your assertions are fortunately not true.
> > Love it when I'm wrong and it means a limitation isn't actually!!!
>
> > > And best of all: no XML is involved ;-)
> > Yeah... I saw some notes in the plugin loader that this was only for
> > testing... Double checking looks like the general classpath search was the
> > one that's marked as testing only.  I got the sense that the annotations was
> > the slightly off color method and that the plugin.xml was the primary way to
> > build plugins.  Are annotations now the preferred method?
>
> > Does the annotation method bring in all annotated plugins into separate
> > classloaders? ie, with XML you get one classloader per plugin.  Now, it
> > would seem you get either
> > 1) a single classloader for all annotated plugins
> > -or-
> > 2) a single classloader per loaded .jar in plugins/ dir
>
> > This is great news!  Makes my day!  Multiple plugins in a single jar!  :)
>
> > Nick
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "kettle-developers" group.
> > To post to this group, send email to kettle-d...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > kettle-develop...@googlegroups.com<kettle-developers%2Bunsubscr­i...@googlegroups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/kettle-developers?hl=en.
>
> --
> Matt Casters <mcast...@pentaho.org>
> Chief Data Integration, Kettle founder, Author of Pentaho Kettle

codeoncoffee

unread,
Dec 21, 2010, 10:36:26 AM12/21/10
to kettle-developers
The wiki has been updated with samples from the MaxMind Plugin:
http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle

codeoncoffee

unread,
Dec 21, 2010, 1:39:01 PM12/21/10
to kettle-developers
The wiki has been updated with samples from the MaxMind Plugin:
http://wiki.pentaho.com/display/EAI/OSGI-enabled+Kettle

On Dec 21, 8:55 am, codeoncoffee <codeoncof...@gmail.com> wrote:
Reply all
Reply to author
Forward
0 new messages