RFC: Restructuring Lift Codebase

[Indrajit Raychaudhuri]

Folks,

As followup to the proposed goal of "Keeping lift-core neat and
small", here is the first iteration of the revised structure of Lift
codebase.


liftweb

- lift-core [10]
- lift-base [02]
- lift-actor
- lift-util
- lift-json [03]
- lift-webkit [04]
- lift-testkit [05]

- lift-persistence [06]
- lift-mapper
- lift-record
- lift-jpa

- lift-modules [07]
- lift-osgi
- lift-wizard [08]
- lift-widgets [09]
- lift-machine
- lift-textile
- lift-facebook
- lift-amqp
- lift-xmpp
- lift-openid
- lift-oauth
- lift-paypal
- lift-jta

- lift-archetypes
- ...

- lift-examples
- ...

- lift-site [10]

- lift-resources [lift-varia, lift-infra ?] [11]
- lift-root-model [12]
- lift-site-skin
- lift-installer
- misc config resources (scaladoc, javadoc etc.)

General notes (including some obvious ones):

[A] lift-* prefix looks superfluous, but it's best to have one for all
artifacts that generate jar (<packaging>jar</packaging>). Also Maven
reactor feels happier when artifactId == directory_name (site
generation, scm extrapolation etc., situation might have improved
now).

[B] The top level project categories (lift-core, lift-persistence,
lift-modules etc.) are simple multi-module models at the moment and
not meant to create anything other than pom. Therefore, lift-* prefix
can go away. But they'll have to come back if we plan to generate 'one
jar' in assembly mode per category (lift-core-all.jar, lift-
persistence-all.jar etc.). This could be useful for 'get me
everything, I don't care about size' people. But is it necessary? The
alternative is to have empty 'meta modules' that pull up the necessary
dependencies and can be included by the users in their project (quite
similar to what lift-core does now).

[C] The members in a project category (lift-mapper, lift-record etc.)
would inherit the category model (lift-persistence in case of lift-
mapper, lift-record). Related modules clubbed together helps similar
configuration pulled up to the parent pom (improves DRY-ness). Added
benefit: modules can be developed even outside Lift codebase with the
given parent pom (available in global repo) and the developer won't
have to worry about most of the infra related boilerplate
configurations (couple of config still would need change though).

[D] Presentations and other materials aren't really source code for
inclusion in source repository. Can this go in wiki? (immediate
problem: github doesn't take attachment). Has this been discussed
earlier? They can go as a separate github project too.

[E] The categorization is mostly based on my interpretation as a late
entrant. If there is a different structure that fits the philosophy
better (quite likely), this would get regrouped. (Tim ?)

[F] The modules in a category can be further sub-grouped, but with
caution. Basically, need the right mix between 'flat'-ness and deep
nesting. Thoughts on this?

[G] Each category can eventually be split up into separate projects
and have their own release schedules (less frequent for core, more
frequent for modules etc.). This might be little overkill at the
moment. Just mentioned to enable us mind the long term perspective :)

[H] Other points on the draft hierarchy (see the # in brackets above):

[01] With these members, if lift-core doesn't sound as the right name,
we need the right name. Provided the grouping is right that is.

[02] Base interfaces for Lift (currently present in dpp_wip_actorized)

[03] Doesn't really have to be part of Lift core in current form, but
this is eventually meant to be part of Lift's JS infrastructure (thus
kept here at the moment)

[04] Candidate for decomposition. But kept intact at the moment. Would
be taken up in next pass once the top level reaches steady state. This
could either become a category of its own or a module with submodules.

[05] Little unsure about it's intent and purpose, I could be
completely mis-interpreting this. Thoughts from somebody with more
understanding please :)

[06] Doesn't have to be part of lift-core. Lift applications not using
persistence doesn't have to need this.

[07] Extra stuff, necessary iff one needs. Right now, I am putting
'everything else' here for lack of better place, but I see a scaling
up issue there. This can turn out to be a place for all the 'nowhere
else to go' modules. But we can take it up in next pass. Suggestions?

[08][09] See #04 above. Can be subpackage of lift-webkit eventually

[10] The website! The intent is to bring liftweb.net codebase into the
streamlines structure. Can be deferred if this is not a burning need.

[11] Recommendation for a good name?

[12] The top level pom for Lift project. All models (the categories)
are expected to inherit this. These categories doesn't necessarily
have to belong to the same git repo.


Let the discussion/debate begin!

Cheers, Indrajit

[marius d.]

Generally I like this structure.Please see my other comments below:

I'm not sure about base interfaces for lift ...

>
> [03] Doesn't really have to be part of Lift core in current form, but
> this is eventually meant to be part of Lift's JS infrastructure (thus
> kept here at the moment)

JSON should not only be part of JS stack as it is used in many
contexts for REST API's. So I'd keep it in the core,

>
> [04] Candidate for decomposition. But kept intact at the moment. Would
> be taken up in next pass once the top level reaches steady state. This
> could either become a category of its own or a module with submodules.

Why splitting it up? .. .the webkit if the very core of lift.

>
> [05] Little unsure about it's intent and purpose, I could be
> completely mis-interpreting this. Thoughts from somebody with more
> understanding please :)


>
> [06] Doesn't have to be part of lift-core. Lift applications not using
> persistence doesn't have to need this.

I definitely agree.

>
> [07] Extra stuff, necessary iff one needs. Right now, I am putting
> 'everything else' here for lack of better place, but I see a scaling
> up issue there. This can turn out to be a place for all the 'nowhere
> else to go' modules. But we can take it up in next pass. Suggestions?
>
> [08][09] See #04 above. Can be subpackage of lift-webkit eventually

I would definitely keep them separated from core/webkit.

[Heiko Seeberger]

Indrajit,

Impressive work!

See my comments below ...

Heiko

2009/9/27 Indrajit Raychaudhuri <indr...@gmail.com>

[A] lift-* prefix looks superfluous, but it's best to have one for all
artifacts that generate jar (<packaging>jar</packaging>). Also Maven
reactor feels happier when artifactId == directory_name (site
generation, scm extrapolation etc., situation might have improved
now).

Let's keep it!

 

[B] The top level project categories (lift-core, lift-persistence,
lift-modules etc.) are simple multi-module models at the moment and
not meant to create anything other than pom. Therefore, lift-* prefix
can go away. But they'll have to come back if we plan to generate 'one
jar' in assembly mode per category (lift-core-all.jar, lift-
persistence-all.jar etc.). This could be useful for 'get me
everything, I don't care about size' people. But is it necessary? The
alternative is to have empty 'meta modules' that pull up the necessary
dependencies and can be included by the users in their project (quite
similar to what lift-core does now).

Again, let's keep the prefix, because for Lift users not using Maven these *-all.jars will be valuable.

 

[D] Presentations and other materials aren't really source code for
inclusion in source repository. Can this go in wiki? (immediate
problem: github doesn't take attachment). Has this been discussed
earlier? They can go as a separate github project too.

GitHub's Wiki or Downloads are not well suited for these materials. I tried myself for ScalaModules and it was frustrating.

Therefore I think keeping them in the repository in a separate folder which of course in not a Maven module is a pragmatic solution.

 

[F] The modules in a category can be further sub-grouped, but with
caution. Basically, need the right mix between 'flat'-ness and deep
nesting. Thoughts on this?

Let's not go deeper right now.

--
Heiko Seeberger

My job: weiglewilczek.com
My blog: heikoseeberger.name
Follow me: twitter.com/hseeberger
OSGi on Scala: scalamodules.org
Lift, the simply functional web framework: liftweb.net

[David Pollak]

Indrajit,

Excellent work!

My thoughts inline.

On Sun, Sep 27, 2009 at 1:44 PM, Indrajit Raychaudhuri <indr...@gmail.com> wrote:

Folks,

As followup to the proposed goal of "Keeping lift-core neat and
small", here is the first iteration of the revised structure of Lift
codebase.


liftweb

- lift-core [10]

lift-core currently means "everything Lift."  I think we need another name, but none is jumping out at me.

From an administrative perspective, having two repositories represents twice the work for me, so I'd rather everything from Lift committers stays in one repository.  With that being said and with our new focus on not merging anything into Master without a review-board cycle, I propose that we have a "presentations" branch and people put their presos on this branch.  Whenever someone merges their review-boarded code into master, they also merge in the presentations branch.

 

[E] The categorization is mostly based on my interpretation as a late
entrant. If there is a different structure that fits the philosophy
better (quite likely), this would get regrouped. (Tim ?)

[F] The modules in a category can be further sub-grouped, but with
caution. Basically, need the right mix between 'flat'-ness and deep
nesting. Thoughts on this?

[G] Each category can eventually be split up into separate projects
and have their own release schedules (less frequent for core, more
frequent for modules etc.). This might be little overkill at the
moment. Just mentioned to enable us mind the long term perspective :)

[H] Other points on the draft hierarchy (see the # in brackets above):

[01] With these members, if lift-core doesn't sound as the right name,
we need the right name. Provided the grouping is right that is.

[02] Base interfaces for Lift (currently present in dpp_wip_actorized)

[03] Doesn't really have to be part of Lift core in current form, but
this is eventually meant to be part of Lift's JS infrastructure (thus
kept here at the moment)

[04] Candidate for decomposition. But kept intact at the moment. Would
be taken up in next pass once the top level reaches steady state. This
could either become a category of its own or a module with submodules.

[05] Little unsure about it's intent and purpose, I could be
completely mis-interpreting this. Thoughts from somebody with more
understanding please :)

The lift-testkit is supposed to be a set of tools to generate integration tests against the application.  I would put it in a separate high-level package (perhaps lift-modules).  It is used during testing, but should not, in my opinion, be part of a deployed WAR.

 

[06] Doesn't have to be part of lift-core. Lift applications not using
persistence doesn't have to need this.

[07] Extra stuff, necessary iff one needs. Right now, I am putting
'everything else' here for lack of better place, but I see a scaling
up issue there. This can turn out to be a place for all the 'nowhere
else to go' modules. But we can take it up in next pass. Suggestions?

[08][09] See #04 above. Can be subpackage of lift-webkit eventually

[10] The website! The intent is to bring liftweb.net codebase into the
streamlines structure. Can be deferred if this is not a burning need.

[11] Recommendation for a good name?

[12] The top level pom for Lift project. All models (the categories)
are expected to inherit this. These categories doesn't necessarily
have to belong to the same git repo.


Let the discussion/debate begin!

Cheers, Indrajit

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics

[Timothy Perrett]

Indrajit,

What is the purpose of lift-resources? We cannot make the lift
installer part of the build process - belive me, i've looked into this
extensively... basically, it boils down to needed install4j licensed
on that machines which would be a stupid requirement to place on any
person building the sources - so we wont be doing that ;-)

What the hell is lift-site-skin?

Cheers, Tim

[Viktor Klang]

Can't the presentations be added as binaries to GitHub, and then link to them from the wiki?

--
Viktor Klang

Blog: klangism.blogspot.com
Twttr: viktorklang

Lift Committer - liftweb.com
AKKA Committer - akkasource.org
Cassidy - github.com/viktorklang/Cassidy.git
SoftPub founder: http://groups.google.com/group/softpub

[Viktor Klang]

On Mon, Sep 28, 2009 at 9:08 PM, Viktor Klang <viktor...@gmail.com> wrote:

Can't the presentations be added as binaries to GitHub, and then link to them from the wiki?

Under "Downloads" I mean.
 

[Indrajit Raychaudhuri]

On 28/09/09 9:45 AM, marius d. wrote:
>
> Generally I like this structure.Please see my other comments below:

Thanks, please see my responses inline.

Cheers, Indrajit

The description says: "Base Interfaces for Lift and perhaps other
frameworks". But looking at the dependencies in dpp_wip_actorized,
lift-actor and/or lift-util would depend on this (Box has moved there).
David is probably working on expanding this (it's wip afterall) and can
can comments.

>
>>
>> [03] Doesn't really have to be part of Lift core in current form, but
>> this is eventually meant to be part of Lift's JS infrastructure (thus
>> kept here at the moment)
>
> JSON should not only be part of JS stack as it is used in many
> contexts for REST API's. So I'd keep it in the core,

Agreed. So core it is :)

>
>>
>> [04] Candidate for decomposition. But kept intact at the moment. Would
>> be taken up in next pass once the top level reaches steady state. This
>> could either become a category of its own or a module with submodules.
>
> Why splitting it up? .. .the webkit if the very core of lift.

Indeed, that's how it is currently. But the way I see this is: A part of
webkit (not whole of it) should be part of core of lift.

The intent is to have (almost) all of webkit that every project would
need as part of core. For example, 'headless' applications would have
nothing to do with the YUI, JQuery libraries that are currently part of
webkit.

One of the ways to achieve this could be to move net.liftweb.http.js
into separate package. And then there is overall discomfort that a
member of the core library has gained lot of weight (the jar is 2.5M) at
the moment! Just a discomfort, not showstopper and thus de-prioritized
at the moment.

>
>>
>> [05] Little unsure about it's intent and purpose, I could be
>> completely mis-interpreting this. Thoughts from somebody with more
>> understanding please :)
>
>
>>
>> [06] Doesn't have to be part of lift-core. Lift applications not using
>> persistence doesn't have to need this.
>
> I definitely agree.
>
>>
>> [07] Extra stuff, necessary iff one needs. Right now, I am putting
>> 'everything else' here for lack of better place, but I see a scaling
>> up issue there. This can turn out to be a place for all the 'nowhere
>> else to go' modules. But we can take it up in next pass. Suggestions?
>>
>> [08][09] See #04 above. Can be subpackage of lift-webkit eventually
>
> I would definitely keep them separated from core/webkit.

I'll keep note of this. Would like to take up together with [04]

[Indrajit Raychaudhuri]

On 28/09/09 11:38 AM, Heiko Seeberger wrote:
> Indrajit,
>
> Impressive work!
> See my comments below ...

Thank Heiko! My response inlined.

/Indrajit

>
> Heiko
>
> 2009/9/27 Indrajit Raychaudhuri <indr...@gmail.com

> <mailto:indr...@gmail.com>>

>
>
> [A] lift-* prefix looks superfluous, but it's best to have one for all
> artifacts that generate jar (<packaging>jar</packaging>). Also Maven
> reactor feels happier when artifactId == directory_name (site
> generation, scm extrapolation etc., situation might have improved
> now).
>
>
> Let's keep it!

So be it! BTW, we already have an anomaly (lift -> lift-webkit) that
would have to be fixed in that case.

>
> [B] The top level project categories (lift-core, lift-persistence,
> lift-modules etc.) are simple multi-module models at the moment and
> not meant to create anything other than pom. Therefore, lift-* prefix
> can go away. But they'll have to come back if we plan to generate 'one
> jar' in assembly mode per category (lift-core-all.jar, lift-
> persistence-all.jar etc.). This could be useful for 'get me
> everything, I don't care about size' people. But is it necessary? The
> alternative is to have empty 'meta modules' that pull up the necessary
> dependencies and can be included by the users in their project (quite
> similar to what lift-core does now).
>
>
> Again, let's keep the prefix, because for Lift users not using Maven
> these *-all.jars will be valuable.

Very good point, and a good practical reason for having *-all.jars. Thanks!

>
> [D] Presentations and other materials aren't really source code for
> inclusion in source repository. Can this go in wiki? (immediate
> problem: github doesn't take attachment). Has this been discussed
> earlier? They can go as a separate github project too.
>
>
> GitHub's Wiki or Downloads are not well suited for these materials. I
> tried myself for ScalaModules and it was frustrating.
> Therefore I think keeping them in the repository in a separate folder
> which of course in not a Maven module is a pragmatic solution.
>
> [F] The modules in a category can be further sub-grouped, but with
> caution. Basically, need the right mix between 'flat'-ness and deep
> nesting. Thoughts on this?
>
>
> Let's not go deeper right now.

Good. So we'll avoid nesting at the module level.

>
> --
> Heiko Seeberger
>
> My job: weiglewilczek.com <http://weiglewilczek.com>
> My blog: heikoseeberger.name <http://heikoseeberger.name>
> Follow me: twitter.com/hseeberger <http://twitter.com/hseeberger>
> OSGi on Scala: scalamodules.org <http://scalamodules.org>
> Lift, the simply functional web framework: liftweb.net <http://liftweb.net>
>
> >

[Heiko Seeberger]

Yes, in public repos this is possible. Take a look at ScalaModules, there I did this.

Only thing is, you got no version control. But for presos that's probably not neccessary.

=> Let's put non-code (presentations) to Downloads. 

Heiko

2009/9/28 Viktor Klang <viktor...@gmail.com>

--
Heiko Seeberger

My job: weiglewilczek.com

My blog: heikoseeberger.name
Follow me: twitter.com/hseeberger
OSGi on Scala: scalamodules.org

[Indrajit Raychaudhuri]

On 28/09/09 11:30 PM, David Pollak wrote:
> Indrajit,
>
> Excellent work!
>
> My thoughts inline.

Thank you :-) My response inlined too.

/Indrajit

>
> On Sun, Sep 27, 2009 at 1:44 PM, Indrajit Raychaudhuri
> <indr...@gmail.com <mailto:indr...@gmail.com>> wrote:
>
>
> Folks,
>
> As followup to the proposed goal of "Keeping lift-core neat and
> small", here is the first iteration of the revised structure of Lift
> codebase.
>
>
> liftweb
>
> - lift-core [10]
>
>
> lift-core currently means "everything Lift." I think we need another
> name, but none is jumping out at me.

Firstly, just realized there is a stupid typo. [10] should read [01].

Indeed, lift-core has a different connotation already. Moreover, with
the proposed member modules, a more apt name would help.

Ok, so as Heiko suggested, we can have a separate location (not part of
maven build reactor) for this.

Taken. So lift-testkit could be under lift-modules. And applications
would typically use it under 'test' scope.

>
>
> [06] Doesn't have to be part of lift-core. Lift applications not using
> persistence doesn't have to need this.
>
> [07] Extra stuff, necessary iff one needs. Right now, I am putting
> 'everything else' here for lack of better place, but I see a scaling
> up issue there. This can turn out to be a place for all the 'nowhere
> else to go' modules. But we can take it up in next pass. Suggestions?
>
> [08][09] See #04 above. Can be subpackage of lift-webkit eventually
>
> [10] The website! The intent is to bring liftweb.net

> <http://liftweb.net> codebase into the

[Indrajit Raychaudhuri]

On 29/09/09 1:12 AM, Heiko Seeberger wrote:
> Yes, in public repos this is possible. Take a look at ScalaModules,
> there I did this.
> Only thing is, you got no version control. But for presos that's
> probably not neccessary.
>
> => Let's put non-code (presentations) to Downloads.
>
> Heiko
>
> 2009/9/28 Viktor Klang <viktor...@gmail.com

> <mailto:viktor...@gmail.com>>

>
>
>
> On Mon, Sep 28, 2009 at 9:08 PM, Viktor Klang
> <viktor...@gmail.com <mailto:viktor...@gmail.com>> wrote:
>
> Can't the presentations be added as binaries to GitHub, and then
> link to them from the wiki?
>
>
> Under "Downloads" I mean.
>

Cool! This is a better option for sure. Tempted to rollback what I just
wrote in reply to David's proposal :-) But then, let's see what other
options/ideas pop up.

Cheers, Indrajit

[David Pollak]

On Mon, Sep 28, 2009 at 12:44 PM, Indrajit Raychaudhuri <indr...@gmail.com> wrote:

On 28/09/09 11:30 PM, David Pollak wrote:
> Indrajit,
>
> Excellent work!
>
> My thoughts inline.

Thank you :-) My response inlined too.

I say "do it on a branch and see what happens".  Do I hear a second?

[Heiko Seeberger]

I say "do it on a branch and see what happens".  Do I hear a second?

Not sure, I like both approaches.

David, why would you prefer to go for a repository branch instead of the Downloads section of GitHub?

Heiko

[Indrajit Raychaudhuri]

Tim,

lift-resources is the place for everything that is not part of regular
Maven build reactor (although some of them could be isolated Maven
module in themselves). This category is different from the other ones in
that sense - sorry for not being unambiguous enough.

Obviously, lift-installer cannot be a Maven module, that out of
question! I can see what triggered the confusion though.

lift-site-skin (can have better name) is basically the substitute of
org.scala-tools.skins:scala-skin-simple. Having own site skin for Lift
would help us have the mvnsite
(http://scala-tools.org/mvnsites/liftweb-1.1-M5/) look similar to
http://liftweb.net which I would much rather ;-)

Cheers, Indrajit

[David Pollak]

I like everything to be centralized in the repository.

 

Heiko

[Timothy Perrett]

+1

[Timothy Perrett]

OK, see what your saying. Perhaps it needs a better name than lift-
resources... conceptually I agree though.

Cheers, Tim

[Derek Chen-Becker]

Agreed. Also, for a while we were putting the Exploring Lift PDF on GitHub and we would get people who couldn't download the file because they didn't have a GitHub account. I don't know if this is something that is fixed now, but it seems like just keeping it in the repo makes things more centralized.

Derek