Creating a module system for Lift

[Peter Robinett]

A common theme at the Scala LiftOff London two weeks ago was that it
wasn't clear how to share bits of general purpose, or at least common,
code among separate Lift projects. I proposed that it's not too hard:
just create a jar containing your code and then make sure it's in the
classpath when building your Lift app. lift-modules already shows some
common patterns, such as providing init and sitemap or menu methods on
a module's singleton object that can be used in Boot.scala.

However, we all agreed that things could and should be so much better.
Not only should modules be easier to create and use but they should be
also easier to distribute and discover. Richard Dallaway, I, and
several others (sorry guys, I forget your names!) spent some time
hashing out a system and I think we've got something interesting.
First we want to use the nascent conventions in lift-modules to define
a standard pattern for modules and only after that develop hosting and
various development tools for modules. Let me also preface this that I
am writing this from memory and my own fairly strong opinions, so
Richard or anyone else, feel free to correct me. ;-)

First, what is the standard pattern for a Lift module? At its most
basic a module is a jar with all its code under a common package (e.g.
com.bubblefoundry.lift.FlotHelpers). Because of Scala's problems with
binary incompatibility, the jar name should follow the sbt naming
conventions, including the Scala version but also the Lift version.
There should be a single object with the same specific name (e.g.
FlotHelpers) and it must have a method called init. It is up to the
module developer to determine and document the parameter(s) and the
return value. This means that one module's init method may take no
parameters and return Unit (e.g. def init: Unit) though it also may
take parameters (e.g. def init(username: String, password: String):
AccessToken). The goal is that all a module user needs to do is add
the module to their sbt or maven project definition, import the
singleton object in Boot.scala (e.g. import
com.bubblefoundry.lift.FlotHelpers.FlotHelpers) and call the init
method within the boot method (e.g. FlotHelpers.init) after checking
the module's documentation on whether there are any parameters
expected. Thus, it is up to the module's init method to register the
module wherever necessary in the Lift app, such as adding its
namespace for snippet searches or its Menus to SiteMap.

The pattern I've just outlined is basically completely do-able today
(with one minor qualification). While us agreeing to this standard
would be a great step forward, there's a lot more we can do to make it
easy to use modules. My inspiration is the RubyOnRails community and
its culture of creating and shared plugins[1]. I would like to have a
server which provides a Maven repository of modules updated via a Lift
app. Module developers will be able to easily submit their modules and
module users can easily search the database for modules that do things
they need. Each module will have its own page with links to its
homepage, code repository, ScalaDocs and other such information.

While there are already some well-known repositories like scala-
tools.org, they are (with good reason) rather selective about the
packages they host. I envision the Lift modules server being much more
of a free-for-all, in a good way. It should be as easy to submit a
module as it is on Rails plugin servers[2]. The goal, of course, is
still good code but also lots of it. Things like ranking and tagging
could be used to show the best modules to users, so there isn't a need
for anyone to manually review and categorize every submission before
publication. Such a process also means that code that previously would
have gotten sucked into the Lift vortex under lift-modules could live
outside of the Lift project. This both makes it easier for more people
to contribute code and also means that modules can be updated and
released to the public on their maintainers' own schedules.

Finally, we also discussed at the LiftOff creating tools to make
developing and using modules easier, such as adding a module command
to Lifty[3] to generate the basics of a module. An sbt command to
publish to the Lift modules server might also be very nice, as it
could generate not only all the binaries but also source and ScalaDocs
jars and then upload them in one shot.

I've discussed this initiative with David and he is quite supportive.
First, he will improve SiteMap to make it easier for modules to
register themselves (ticket #685 [4]), such that module users will
only need to call an init method and not also add the results of a
menu method call to the list of Menus used to create the SiteMap.
Second, he already has the domain names lift-tools.org and lift-
modules.org that we can use. I personally like lift-modules.org a lot,
but I'm open to any name the community suggests.

Please let me know what you think. I look forward to your feedback and
hope that this is a nice project all of us in the Lift community can
get behind.

Cheers,
Peter

[1]: http://wiki.rubyonrails.org/getting-started/rails-plugin
[2]: http://agilewebdevelopment.com/plugins/new
[3]: http://lifty.github.com/Lifty/
[4]: http://www.assembla.com/spaces/liftweb/tickets/685

[TylerWeir]

I think it's a great initiative Peter.

What are you looking from the community and how can we help?

[Peter Robinett]

Thanks, Tyler. I'm looking for the following:
1. Feedback on the module conventions
2. Help setting up the Maven repository
3. Feedback on and help building the module server app
4. Feedback on and help building any module-related tools
5. Modules!

Peter

[Martin]

In general this is a very good initiative! I'm sure that me and the
others in our team will do whatever we can to contribute by doing some
module development.

Martin

[Richard Dallaway]

On Wed, Oct 20, 2010 at 12:57 AM, Peter Robinett <pe...@bubblefoundry.com> wrote:

Let me also preface this that I
am writing this from memory and my own fairly strong opinions, so
Richard or anyone else, feel free to correct me. ;-)

The email is how I remember it.   Thanks for kicking this off.

[Peter Robinett]

Martin, Richard, thanks. To everyone, what do you think of the module
conventions that I outlined above? Would you like any changes (in
naming or initialization or anything else)? Am I missing something
that a module should do that we should standardize? Let's hammer this
out now so it doesn't bite us later.

Cheers,
Peter

[Richard Dallaway]

On Wed, Oct 20, 2010 at 10:21 PM, Peter Robinett <pe...@bubblefoundry.com> wrote:

Martin, Richard, thanks. To everyone, what do you think of the module
conventions that I outlined above? Would you like any changes (in
naming or initialization or anything else)? Am I missing something
that a module should do that we should standardize? Let's hammer this
out now so it doesn't bite us later.

I wondering if it might help to give examples for context. Here are a couple of lumps of functionality I was thinking of which might fit into the scheme some how.

First one: add the async google analytics tracking code to a lift app:

- init this module with a Props value for the tracking id.

- hooking into head process; or adding a snippet call just before </head> (in the default template perhaps); or telling the end user to do that

- fetching the source for the snippet or downloading a pre-built jar

Hmm. Even in that example there are a bunch of options.

The second one is to take Tim's Stax.net filter to set production run.mode and apply that to a lift app:

- get the source or jar containing the filter

- modify web.xml to call the filter

- in talking to stax, may need to write or adjust stax-web.xml too (but that's a story for another day).

Those are trivial examples, not adding very much value compared to instructions on a wiki.  But they are concrete examples that touch on some of the issues in going further than the current conventions that Peter outlined.

It does sound like SBT/lifty might help with some of this, and possibly scala fresh might have some overlap.

Richard

[Derek Chen-Becker]

Nice examples. I think that having a way for modules to register data to insert during head merge would be good, among other things.

Derek

--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.

[george]

I think this sums it up pretty well as I recall. Well done Peter!

I'm happy to lend some time to getting this on the road, if each of us
contributed one module then that would be a good start. :-)

One thing that I think would help improve Lift modules over those of
other frameworks would be if there was a reliable (and user friendly)
way to know if a module works properly with the version of Lift I'm
using, without having to download, compile and try it.

This could be as simple as a voting system on the module server (like
how the Wordpress plugin site does it) where users report their
success (or otherwise) with a particular module / lift combo. Or it
could be some full-on continuous integration automated testing with
the results displayed on the module web page. Maybe there could also
be some standard tests included in a 'skeleton' module that would have
to pass before you can publish it.

Not sure how feasible any of this is, but thought I'd throw it in!


George

[Peter Robinett]

Derek, registering data (ie CSS or JS files) for the head merge sounds
like a great idea. This actually ties in with the Javascript plugin
(or, 'module' in the current terminology) system that I proposed some
time ago. Perhaps we can kill two birds with one stone? I don't really
know how Lift does the header merging but perhaps modules can just
append a NodeSeq to the current head?

Geroge, glad to hear your thoughts. I agree that indicating Lift
version support is very important. I think it could be really cool if
something like sbt's %% syntax could be reused so that both the Scala
and Lift version are taken into account. Are there any sbt gurus here
that know if that could be done?

Peter

[David Pollak]

Peter,

Thanks for spear-heading this effort.  I've added S.putInHead(elem) and S.putAtEndOfBody(elem) to give modules more latitude in inserting script tags, etc.

I've added a mechanism for mutating SiteMaps:

Modules

Lift has supported modules from the first version of the project in 2007. Lift's entire handling of the HTTP request/response cycle is open to hooks. Further, Lift's templating mechanism where resulting HTML pages are composed by transforming page content via snippets (See sec:Snippets) which are simply functions that take HTML and return HTML: NodeSeq => NodeSeq. Because Lift's snippet resolution mechanism is open and any code referenced in Boot (See sec:Boot), any code can be a Lift ``module'' by virtue of registering its snippets and other resources in LiftRules. Many Lift modules already exist including PayPal, OAuth, OpenID, LDAP, and even a module containing many jQuery widgets.

The most difficult issue relating to integration of external modules into Lift is how to properly insert the module's menu items into a SiteMap (See sec:SiteMap) menu hierarchy. Lift 2.2 introduces a more flexible mechanism for mutating the SiteMap: SiteMap mutators. SiteMap mutators are functions that rewrite the SiteMap based on rules for where to insert the module's menus in the menu hierarchy. Each module may publish markers. For example, here are the markers for ProtoUser:

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

/** * Insert this LocParam into your menu if you want the * User's menu items to be inserted at the same level * and after the item */ final case object AddUserMenusAfter extends Loc.LocParam /** * replace the menu that has this LocParam with the User's menu * items */ final case object AddUserMenusHere extends Loc.LocParam[Any] /** * Insert this LocParam into your menu if you want the * User's menu items to be children of that menu */ final case object AddUserMenusUnder extends Loc.LocParam[Any]

The module also makes a SiteMap mutator available, this can either be returned from the module's init method or via some other method on the module. ProtoUser makes the sitemapMutator method available which returns a SiteMap => SiteMap.

The application can add the marker to the appropriate menu item:

?

1	Menu("Home") / "index" » User.AddUserMenusAfter

And when the application registers the SiteMap with LiftRules, it applies the mutator:

?

1	LiftRules.setSiteMapFunc(() => User.sitemapMutator(sitemap()))

Because the mutators are composable:

?

1 2	val allMutators = User.sitemapMutator andThen FruitBat.sitemapMutator LiftRules.setSiteMapFunc(() => allMutators(sitemap()))

For each module, the implementation of the mutators is pretty simple:

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14

private lazy val AfterUnapply = SiteMap.buildMenuMatcher(_ == AddUserMenusAfter)   private lazy val HereUnapply = SiteMap.buildMenuMatcher(_ == AddUserMenusHere)   private lazy val UnderUnapply = SiteMap.buildMenuMatcher(_ == AddUserMenusUnder)   <p> /**    * The SiteMap mutator function    */   def sitemapMutator: SiteMap => SiteMap = SiteMap.sitemapMutator     case AfterUnapply(menu) => menu :: sitemap     case HereUnapply(_) => sitemap     case UnderUnapply(menu) => List(menu.rebuild(_ ::: sitemap))   (SiteMap.addMenusAtEndMutator(sitemap)) </p>

We've defined some extractors that help with pattern matching. SiteMap.buildMenuMatcher is a helper method to make building the extractors super-simple. Then we supply a PartialFunction[Menu, List[Menu]] which looks for the marker LocParam and re-writes the menu based on the marker. If there are no matches, the additional rule is fired, in this case, we append the menus at the end of the SiteMap.

I hope this leads to some excellent new modules.

Thanks,

David

--
You received this message because you are subscribed to the Google Groups "Lift" group.
To post to this group, send email to lif...@googlegroups.com.
To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im
Surf the harmonics

[Francois]

On 20/10/2010 01:57, Peter Robinett wrote:
> A common theme at the Scala LiftOff London two weeks ago was that it
> wasn't clear how to share bits of general purpose, or at least common,
> code among separate Lift projects. I proposed that it's not too hard:
> just create a jar containing your code and then make sure it's in the
> classpath when building your Lift app. lift-modules already shows some
> common patterns, such as providing init and sitemap or menu methods on
> a module's singleton object that can be used in Boot.scala.

A related topic is how to build re-uasable components, like Snippets
that would be context-bounded by some states - or more like Snippets
local to other (parent) snippets, that have no self-being as a page[1].

Again, Lift already provides everything needed to build such components,
and again, it will be great to have general guide lines and convention
to make such component development less developer habits specific, at
least in their general APIs and life cycle.

For now, for my personal use, the convention are:
- these snippets don't go to the "snippet" package, but to a "component"
one ;
- they extends DispatchSnippet ;
- they are initialized and kept in their parent Snippet thanks to a
"LocalSnippet" contener (see code http://gist.github.com/729195 )
- they provide a "initJs()" method that manage all their JS stuff (it
happens quite often that these components are shown as results of AJAX
requests) ;
- they have a constructor with all their state configuration (context,
etc) ;
- they give back information to the calling snippet thanks to getter (I
mean, public def/val, not specifically "getSomeComponentState")

What are your opinions about that ? Is it something you also use ? If
not, how do you build your reusable tools in Lift ?

Cheers,

[1] typical example : a form used in several snippets, a JsTree or
datagrid integration, a pop-up, etc

--
Francois ARMAND
http://fanf42.blogspot.com
http://www.normation.com

[Francois]

On 05/12/2010 17:18, Francois wrote:
> A related topic is how to build re-uasable components, like Snippets
> that would be context-bounded by some states - or more like Snippets
> local to other (parent) snippets, that have no self-being as a page[1].

[...]

> - they provide a "initJs()" method that manage all their JS stuff (it
> happens quite often that these components are shown as results of AJAX
> requests) ;

Ah, just saw the thread about Wiring[1], and it's seems übber cool in
that context... Now, we could have complex components, wired in a parent
snippet, with updates on some leading to update on others...

I can't stop thinking to that use case: an embedable search component,
for example to search for emails:
You have a complex form component, build on two parts:
- the criteria definition part, an ajax form with a list of criterion
lines (type of criteria (from/to, date), comparator (is, contains, is
before), user input). You can add/remove lines, and there is a submit button
- a list of result, say a DataGrid integration (which would be an
other component) ;

Now, you could embed that search component on any pages... When you
update the search, automatically other states are updated. Things like
infos/actions: export to CSV, deletes all, etc

Well, I think I will have to look to that feature quickly...

[1]
http://groups.google.com/group/liftweb/browse_frm/thread/5b02ee88adf1d0ae/6b15fa949d34f4b5?tvc=1#6b15fa949d34f4b5

and example: http://demo.liftweb.net/invoice_wiring

[Todd O'Bryan]

Aren't these something like Widgets? The hard part, as it always is,
is to figure out how to make such components general enough that
they're useful to lots of people, but not so generic that they're just
as hard to use as to write something from scratch yourself. (Hey, I
think I just summarized the fundamental problem of object-oriented
programming.)

Todd

[Francois]

On 05/12/2010 18:01, Todd O'Bryan wrote:
> Aren't these something like Widgets? The hard part, as it always is,
> is to figure out how to make such components general enough that
> they're useful to lots of people, but not so generic that they're just
> as hard to use as to write something from scratch yourself. (Hey, I
> think I just summarized the fundamental problem of object-oriented
> programming.)

Well, that is not exactly as if other web framework didn't try and find
for us. Almost any component oriented frameworks like Tapestry 5, JSF
(yeah..), GWT, Wicket etc will have such library.

Examples:
- Vaadin: http://demo.vaadin.com/sampler
- IceFace:
http://component-showcase.icefaces.org/component-showcase/showcase.iface
- echo2/echo3: http://demo.nextapp.com/echo3csjs/

Etc.

So you often ends with:
- rich form inputs (autocompletion, date with calendar, slider,
two-columns multi-selects, etc)
- forms management (fields validation, state and error management, field
layout, etc)
- tree, grid, cover flow
- windows, pop-up, notification
- charts
- file upload & progress bar
- media (flash, quick time, html 5 video, google map, etc)
- secure transaction (paypal and other bank integratioon stuff)

And that covers the 80% of all element you need :)

Note: I'm not saying that Lift does not have any of that. For example,
LiftScreen are a really good candidate for form management :)