Versioning namespaces

[Andrew Eddie]

Because I can't help myself, I've been thinking a lot about how I would construct Joomla 4 given the chance (3 words - test-driven-development). One of the big drama's is always backward compatibility.

Historically speaking, our greatest site migration success has come with keeping new work parallel (ala 1.0 to 1.5 legacy layer style) as opposed to a clean break (1.5 to 1.6 style).

That got me thinking. What if you used the package namespace to support different major versions of a package so that the old and the new code could installed side-by-side. Would that allow the CMS to be able to stay within a minor version longer? Perhaps.

Here's an example.

Let's say we convert Joomla 3.x to use the Composer installed Model package and some refactoring of core is done to keep things up to date.

Let's assume it's v2 of Model that's been released as LGPL (/me folds arms and taps foot).

Let's say some of the better extension developers follow suit.

Now we want to do a major upgrade to the Model package. Right now, we are stuck adding that to core until 4.x. But what if v3 of the model package was namespaced as "Model3" where "3" is the major version of the package. Now in Composer we can install Model and Model3 in the same install, and old and new core and old and new extension will work side by side. It could be that Model3 still used most of Model, but just changes interfaces or whatever.

In the code, older elements would use:

`use Joomla\Model\Model as Model`;

and newer elements can say:

`use Joomla\Model3\Model as Model`;

Alternatively you could do something more traditional like (makes Composer SemVer a little more interesting):

`use Joomla\Model\v3\Model as Model`; 

Type hinting is still a bit curly, but as long as there is a rule that only interfaces are allowed for type hinting, I think we can get around it.

There is already a sort of precedent with FOF and F(zero)F where this kind of thing has had to happen (I'm not complaining about that happening, I just think it can happen in a different and better way). Outside of Joomla, best practice for RESTful API's for example is to version them from the outset, usually with a `v#` prefix in the route - so this is not a new idea.

Much as I hate trying to accommodate old code from staying on life support, trying to fight that mindset is too hard so it's easier to just work around it.

The advantage of this type of strategy is that it will provide the CMS community some confidence that they can put off the worry of doing a full blown migration anytime soon. I believe it also means that the Framework Team can take the lead on a single attack front of architectural reform without also having to fight for how much change they are allowed (don't care - y'all just keep using the old code if you want). At the end of the day, as long as the site still works after an upgrade is all the User cares about, and the Developer is only worried about whether you break their extension.

If the Framework Team can essentially work on the new engine for Joomla 4 within Joomla 3, I think that gives this team great purpose and identity. The CMS team can then concentrate on building out user-facing features. To be honest, the strategy could last indefinitely (though there are practical limits on how much code bloat you want to keep in the core).

Thoughts or alternatives?

Regards,

Andrew Eddie

[Andrew Eddie]

Let me be more clear about the problem I am trying to identify:

1. No developer or user likes B/C breaks
2. Historically the project, imo, has not recovered from the severe
break from 1.5 to 1.6, and then the template break from 2.5 to 3.
3. Trying to negotiate or agree on the amount of B/C we can break
going from 3 to 4 is hard (Joomla community sucks at negotiating,
they'd rather burn the farm than compromise).
4. Therefore a fight on two fronts (we need change, and we need to
convince others they need to adopt the change) is harder than one (we
need change, and you can opt into it when you need to).
5. Fighting on two fronts, historically, leads to your best dev's
loosing interest because they spend all their time trying to justify
the work they want to do rather than actually doing it (death by
committee).
6. It's better to get behind some idea that could lead to a modicum of
success than to do nothing and continue to loose momentum.

The disadvantages of this approach are that it leads to code bloat,
and it can lead to confusion about which "code" to use, and it does
make it hard for site builders to customise when you have different
developers using different standards (although my experience is that
the bar is not that high so anything is better than nothing).

The alternative, however, is the continued paralysis that this project
is experiencing in terms of forward innovation and architectural
reform.

Something's got to give.


Regards,
Andrew Eddie

> --
> Framework source code: https://github.com/joomla-framework
> Visit http://developer.joomla.org for more information about developing with
> Joomla!
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Joomla! Framework Development" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/joomla-dev-framework/gOLt65eS9Go/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> joomla-dev-frame...@googlegroups.com.
> Visit this group at http://groups.google.com/group/joomla-dev-framework.

[Andrew Eddie]

And one last thing to reiterate. It is my belief that the Framework
can lead code reform in the CMS, not the other way around. If this is
not possible, then please delete this thread as there is no point in
discussing it further.

Regards,
Andrew Eddie

[Noel Dixon]

1. No developer or user likes B/C breaks 

Agreed, it does actually end up costing real dollars at the end of the day. My experience has been a greater hit in confidence going from 2.5.x to 3.x migrations, from the clientele side.

2. Historically the project, imo, has not recovered from the severe 
break from 1.5 to 1.6, and then the template break from 2.5 to 3.

I have seen this from the commercial side, schools and government in particular that have been long time users of Joomla. I actually still have a school on 1.5.26 because they are skittish on cost and unknowns of change.

3 & 4 & 5 (fight on two fronts). Trying to negotiate or agree on the amount of B/C we can break 

going from 3 to 4 is hard (Joomla community sucks at negotiating, 
they'd rather burn the farm than compromise). 

I hear you, I see it from the observational sidelines and the social media universe. I hope this gets sorted as some point in the future, I am worried it might be catastrophic to the integrity of the application itself as leadership wanes and neglect sets in. It truly is the wild west out there but we don't have to let the wolves eat the lambs.

6. It's better to get behind some idea that could lead to a modicum of 
success than to do nothing and continue to loose momentum. 

Couldn't agree more. While there should be respect for input from all genuine contributors to an open-source project like Joomla, sometimes it pays to put our own idea aside to champion a better outcome for the greater good. Thinking innovation, decoupling, dev friendly and well documented here as well.

Not sure if my own comments here are helpful but a couple of things I would like to see are a move to Composer, decoupling the core extensions, decoupling the bootstrap template from the CMS framework (just because I'd like an easier integration with Foundation or other, 'cause I might want to), a continuation of innovations like the "Install from Web" tab and Google 2-step authentication option(s).

I understand there is a bigger picture here but it's my two cents.

Noel

[Nils Rückmann]

HI Andrew, am i right that you suggest different packages/repositories for each major release ? I'm not against it, it's the same approach which is used by many libraries in the deb-universe. I just want to ask if i understand your proposal. It's a pity that composer can't handle multiple versions of the same package.

[Michael Babker]

Do you have examples of libraries which change their namespaces in some form on each major release?  I know Guzzle did it (they also moved to a new repo) with their last major, and I believe the Amazon SDK did too, but those are the only examples I can think of right away.

I personally don't think versioned namespaces is the right way to go.  I don't see it addressing any issue, in fact I only see it opening the door to more issues.  Sure, in the CMS users would absolutely love it if you could run FW v1, FW v2, CMS v3, and CMS v4 all in the same request cycle (I'm sure they'd love their Mambots to still work today too), but I see code bloat and really bad code structures if you even give developers the option to do this.  It just seems like a majorly bad idea to build an application environment or enable developers to build one where one component is using Registry 1.4 and another component 2.0.  I'll concede the one thing that versioned namespaces enables is easier development of a CMS extension for two major version branches, but is that really worth making this change?

On Tue, May 19, 2015 at 9:51 AM, Nils Rückmann <ma...@nueckman.de> wrote:

HI Andrew, am i right that you suggest different packages/repositories for each major release ? I'm not against it, it's the same approach which is used by many libraries in the deb-universe. I just want to ask if i understand your proposal. It's a pity that composer can't handle multiple versions of the same package.

--

Framework source code: https://github.com/joomla-framework
Visit http://developer.joomla.org for more information about developing with Joomla!
---

You received this message because you are subscribed to the Google Groups "Joomla! Framework Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-frame...@googlegroups.com.

[Nils Rückmann]

Prominent examples would be GTK, Python or Apache.

[Nils Rückmann]

I admit, it's not ideal. But it has benefits with minimum effort. Personally i don't know any other solution to solve the issue with multiple version. Well, there would be the possibility to bind  a package and its dependencies like Snappy (and others) does, but that's no solution for php.

[Don Gilbert]

Another way for the CMS to start using the Framework is for the CMS to define the interfaces it requires, then have the existing code transformed to meet those interfaces. This will allow the CMS start using the Framework to rebuild its underlying structure while remaining compatible with existing code. 

We tried something like this when we changed JController to be an interface, but it caused problems because we used an existing name instead of a new interface name which the existing class name then implemented. It could have been a much simpler transition if we would have changed it to be:

class JController implements JControllerInterface {}

The downside to this approach is we become "stuck" with current method naming and signatures for class functionality (display, save, save2new etc), but we gain BC and flexibility. But it's not really a downside, because a big part of retaining BC is retaining public method names and signatures - meaning it's something we should be doing anyway.

With that complete (when/if ever that happens), it becomes much more straightforward to upgrade the lower level structure without affecting higher level components. Or, if you prefer the words of Martin Fowler: "altering its internal structure without changing its external behavior". (http://refactoring.com)

On Tue, May 19, 2015 at 9:31 AM, Nils Rückmann <ma...@nueckman.de> wrote:

I admit, it's not ideal. But it has benefits with minimum effort. Personally i don't know any other solution to solve the issue with multiple version. Well, there would be the possibility to bind  a package and its dependencies like Snappy (and others) does, but that's no solution for php.

--

[Andrew Eddie]

On 20 May 2015 at 00:17, Michael Babker <michael...@gmail.com> wrote:
> Do you have examples of libraries which change their namespaces in some form
> on each major release?

Java perhaps? I'm really just taking the principles of good API design
that we use for web services and imagining how we could apply them to
solve the same problems in code.

> I personally don't think versioned namespaces is the right way to go.

Initially, it doesn't matter because there is stuff all FW code in the
CMS now we can probably cope for a number of years. But at some point
we will want breaking change introduced into the CMS (assuming, once
again, that the FW leads the CMS, not the other way around), but I'm
happy to defer that conversation and decision for as long as possible.

> I'll concede the one thing that
> versioned namespaces enables is easier development of a CMS extension for
> two major version branches, but is that really worth making this change?

In principle, yes. Don't you remember the stupid tirade over dropping
the DS constant!

Regards,
Andrew Eddie

[Andrew Eddie]

On 20 May 2015 at 01:08, Don Gilbert <> wrote:
> Another way for the CMS to start using the Framework is for the CMS to
> define the interfaces it requires, then have the existing code transformed
> to meet those interfaces. This will allow the CMS start using the Framework
> to rebuild its underlying structure while remaining compatible with existing
> code.

I agree. The very act of adding the FW to the CMS is effectively
"versioning" the namespace, albeit we only get one shot in the
conventional matter. I do agree that moving to a published system of
public interfaces is crucial moving forward. It is best practice and
we need to drag the CMS kicking and screaming yet with as much
backward compatible support as possible.

So ok, if we can tolerate at least a bit of code bloat where we keep a
lot of the legacy code beside the new FW code, but we have a
systematic plan to reform and replace the need for that code, I'm in!

Regards,
Andrew Eddie

[Michael Babker]

I can't say it's any good, but https://github.com/mbabker/joomla-next is my idea bouncing playground for architecture related stuff.  Including maybe the first real attempt to define proper interfaces for the CMS APIs.  When I dropped this on the CMS list there was a fair amount of support behind the concept of rebuilding and breaking B/C in a logical manner.  The feeling I got from that thread is if it makes sense, we'll get support behind it, but a user migration path MUST be provided by the "core team", we can't put all of our money in jUpgrade or SPUgrade covering tasks that should be done by core.  So it isn't blind permission to say "screw B/C", but I get the feeling B/C doesn't necessarily need to be the absolute first priority looking toward a major overhaul of the app.  And in complete honesty, I think we're shooting ourselves in the collective feet if we try to balance B/C and rearchitecting the legacy structure of the CMS; one of the two will have to give in some form.

[Nils Rückmann]

As for migration, we need to separate user and developer migrations. User migrations is data (db + files) only, which should not be a big deal. I've never understood why there wasn't any data-migration tool for 1.5 -> 1.6 written by the core. And the migration for third party developers depends mostly on interface, at least it should. And that's were Andrews propasal comes into play. If we can have 2 versions of the same package in one installation, we could easily deprecate the old one and give developers enough time to migrate their extensions.

The lack of interfaces is something which CMS and framework does have in common. But i think that's a very general issue within test-driven development. So many people are focusing on writing tests without taking enough time to design the structure which is typically represented with interfaces. I miss the time were UML was trendy ;) 

To put it in a nutshell: Interfaces first!

[Andrew Eddie]

On 20 May 2015 at 08:32, Michael Babker <michael...@gmail.com> wrote:
> The feeling I
> got from that thread is if it makes sense, we'll get support behind it, but
> a user migration path MUST be provided by the "core team", we can't put all
> of our money in jUpgrade or SPUgrade covering tasks that should be done by
> core. So it isn't blind permission to say "screw B/C", but I get the
> feeling B/C doesn't necessarily need to be the absolute first priority
> looking toward a major overhaul of the app. And in complete honesty, I
> think we're shooting ourselves in the collective feet if we try to balance
> B/C and rearchitecting the legacy structure of the CMS; one of the two will
> have to give in some form.

As I said, I don't want to fight on two fronts. I don't want to have
to justify sensible changes to people that just don't want to
understand. I'd rather give them the option of not having to worry
about things until the CMS bumps a major version whenever it feels it
can.

Starting to run into a side issue here but I think the key is to
change the purpose of the Framework to deliberately serve the CMS,
whilst still making a case that other non-Joomla people might be
interested in our code.

Here's my take-over-the-world-again plan:

1. We work out the workflow for bringing FW work into the CMS via
Composer. That is, when something is ready here, there is a well
defined and clean path to get that code into the CMS quickly (taking
days, not years).

2. We identify some quick wins in the CMS to do our initial
experiments on. It might be a tiny bit of the code, but we:
* identify the behaviour in the core
* define an explicit Interface for the API
* write the FW code, tests and documentation
* include the new code into the CMS via Composer and adjust the core
code to use it

Rinse and repeat a couple of times just to get into a groove and prove
to people that the sky is not going to fall.

Hint - don't take ORM first - pick a few small things.

3. To facilitate this process we change our collaboration approach
from "do whatever you want whenever your want" to "we decide to focus
on one thing at a time". Joomla has certainly flourished when there
are highly skilled, self-motivated developers that just know what to
do and know what to fix. These days, they are few and far between
because of the way we have made decisions in the past (aka, you have
to please everyone and end up pleasing no one).

So the change would be to borrow from the Agile methodologies where we
deliberately break our time into "sprints" (not to be confused with
what Joomla historically calls a code sprint).

You pick a cycle length - either 2 weeks or 1 month.

You identify one, perhaps two goals to achieve in that sprint and
everyone is asked to focus on that one or two goals.

You break goals into tasks so that people know what to do when they have time.

For now, most of the goals will revolve around improving the CMS with
well designed, lightly or non-coupled packages. This is a deliberate
ploy to make the CMS team(s) see that we are the good guys, in fact we
are the crack developers that allow them to make crazy magic in their
UIs.

This leaves the CMS to concentrate more on the end user features and
eye-candy that we have historically dazzled the world with.

In the process, we'll get goals that do non-CMS specific things. But
solving problems in the CMS the right way are going to benefit
everyone anyway.

Each sprint we use the Volunteer Portal to report back to the
community on what we did and what we plan to do next.

4. By this time we are in a good groove - people know what we are
doing and contributors know what to do. The next step is simply to
handle progressively more challenging areas of the code.

5. We put needless debates to bed.

PSR-2 keeps coming up? Ok, let's just go PSR-2 so we don't have to
waste time arguing all the flippin' time why we aren't using it.

Documentation needs to be done some better way. Ok, whatever, let's
just thrash it out once and for all and decide how it should be done.

If there is no clear consensus, put it to a vote of the Team Members
(which today stands at 3 so it's going to be a quick vote on every
issue - hint: being a Team Member is where the power to shape Joomla
now resides and I think this is a GOOD thing).

Here endeth the manifesto.

Sound exciting, challenging, fun? I think so.

Regards,
Andrew Eddie

[Michael Babker]

Once you get into components and modules, it's possible then you could have an app that supports two major version branches; there is some flexibility there.  How do you deal with application APIs that are infrastructure in the app itself, i.e. JDocument, session, installer, or app classes themselves, without introducing circular dependencies across major versions?  If we refactor the session API, where does the SessionInterface get defined, in v1 or v2?  Is there an interface for each major version or just one interface?  If you go through with the idea some seem to have of making interfaces a separate package, how do you version those?  If the CMS is using the v2 API, how can an extension use the v1 API then?

I'm not trying to build a higher brick wall each time it's climbed, but these are issues I see with an application environment supporting two major versions of two separate APIs (right now, we very realistically are talking about 3 separate versions of APIs in some cases).

--
Framework source code: https://github.com/joomla-framework
Visit http://developer.joomla.org for more information about developing with Joomla!
---
You received this message because you are subscribed to the Google Groups "Joomla! Framework Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-frame...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-framework.

--
- Michael

Please pardon any errors, this message was sent from my iPhone.

[Andrew Eddie]

On 20 May 2015 at 12:02, Michael Babker <> wrote:
> Once you get into components and modules, it's possible then you could have
> an app that supports two major version branches; there is some flexibility
> there. How do you deal with application APIs that are infrastructure in the
> app itself, i.e. JDocument, session, installer, or app classes themselves,
> without introducing circular dependencies across major versions?

That's all part of the exercise. I don't have the answers until we
start looking at the code in detail. Even then, it will be taking
small, steady steps at a time.

It's a "we won't know until we get there" type of problem, but it's
better than nothing.

> If we
> refactor the session API, where does the SessionInterface get defined, in v1
> or v2?

If that's a FW version, then I'm assuming this is all V2 LGPL code
(but it could be V1 if you are ok with changing to the LGPL now which
I think would be perfectly fine, but it's not my call).

> Is there an interface for each major version or just one interface?

I think the question is the wrong way around. You only bump the major
version when you make a backward compatible change to any interface.
I'm also assuming you aren't locking every package in the FW to
exactly the same version number (if you are, yeah, this plan sucks and
is completely unworkable).

> If you go through with the idea some seem to have of making interfaces a
> separate package, how do you version those?

I've already said previously, don't make this a problem until it's a
problem. Keep the interfaces in packages until there is demand for
them to be separate.

> If the CMS is using the v2 API,
> how can an extension use the v1 API then?

I'm getting a bit lost. Are you talking current v1 and v2 of the FW,
or speculating on a future version? My assumption is we are working
with V2 of the FW but only because that is what has to happen to
relicense it.

> I'm not trying to build a higher brick wall each time it's climbed, but
> these are issues I see with an application environment supporting two major
> versions of two separate APIs (right now, we very realistically are talking
> about 3 separate versions of APIs in some cases).

Ok, so let's take a few steps back. Of what's in
https://github.com/joomla/joomla-cms/blob/staging/composer.json,
what's planned to have a breaking change in V2 of any of our packages?

Regards,
Andrew Eddie

[Michael Babker]

So I'm basically using the two major versions of both software platforms right now; v1 and v2 of the Framework packages, and v3 and v4 of the CMS.  Just to try and make that clear.

On Tue, May 19, 2015 at 11:32 PM, Andrew Eddie <mamb...@gmail.com> wrote:

> Is there an interface for each major version or just one interface?

I think the question is the wrong way around. You only bump the major
version when you make a backward compatible change to any interface.
I'm also assuming you aren't locking every package in the FW to
exactly the same version number (if you are, yeah, this plan sucks and
is completely unworkable).

Right, but does the namespace of the interface then change too?  So you could have \Joomla\Session\SessionInterface in the 1.x branch then \Joomla\Session\v2\SessionInterface in the 2.x branch.  How are the two branches remotely compatible with one another then?  Your idea suggests that developers may be able to use both versions of the API in parallel, but if CMS v4 is built using the Session v2 code, how can an extension use the Session v1 API then?

In truth, I think I can live with the hole we've dug ourselves into with 30+ individual components going on 30 different paths as far as versioning goes, but only within a major branch.  I don't think it'd do us a lot of good to have Database v3, Application v2, and Registry v4 all floating around as the "active" versions, that'll get confusing quickly (and yes I know that's really hypothetical but I hope it illustrates one of the things I dislike about a scenario that idea allows to happen).

> If the CMS is using the v2 API,
> how can an extension use the v1 API then?

I'm getting a bit lost. Are you talking current v1 and v2 of the FW,
or speculating on a future version? My assumption is we are working
with V2 of the FW but only because that is what has to happen to
relicense it.

I'm speculating on the v4 CMS codebase.  You're suggesting that the API be versioned via namespaces (which the only way to do this with Packagist is creating new packages for every major release because that and Composer won't let you install two versions of the same code), so in theory you open the door for a CMS extension running on a potential v4 of the CMS which is running the CMS v3 "legacy layer" and potentially uses Framework v1 code.  Reading this, it's really confusing, but it makes sense in my head, I swear!

> I'm not trying to build a higher brick wall each time it's climbed, but
> these are issues I see with an application environment supporting two major
> versions of two separate APIs (right now, we very realistically are talking
> about 3 separate versions of APIs in some cases).

Ok, so let's take a few steps back. Of what's in
https://github.com/joomla/joomla-cms/blob/staging/composer.json,
what's planned to have a breaking change in V2 of any of our packages?

Of what's there today, the only package with a roadmap idea that may have B/C implications is splitting off the Registry package's loader classes to form some sort of abstract layer usable for different purposes (one idea floating around is adding support for more file formats in the Language package, which would serve the PHP community moreso than Joomla most likely).  That also implies that no other Framework packages will get merged into the CMS for the remaining lifetime of its 3.x branch.  If the Language package gets integrated back into the CMS, that one would have B/C implications going between major versions.  And on the MVC side of things, the current v2 View package is a fair bit different than the previous version.

And truthfully, if even half of that roadmap post sees any action, there's a good chance other stuff will break.

[Andrew Eddie]

On 20 May 2015 at 23:55, Michael Babker <> wrote:
> Right, but does the namespace of the interface then change too? So you
> could have \Joomla\Session\SessionInterface in the 1.x branch then
> \Joomla\Session\v2\SessionInterface in the 2.x branch. How are the two
> branches remotely compatible with one another then?

I don't think that's a good example because you physically can't
implement 2 sessions at the same time. But let's think about what we
can do "today" with Session.

Let's say we finalise \Joomla\Session@2.0. We want to add that to the
CMS and there are no dependency problems (which I'm sure there will be
to solve).

Today we can add Session@2.0 to package.json

We can then migrate all the core code over to using it.

We have JSession implement the Session@2.0 interface.

We convert JSession to be a wrapper for the \Joomla\Session@2.0 code
and we mark JSession as deprecated.

One day, we decide we need a breaking change and the FW Team cuts Session@3.0.

The plan is to integrate that into the CMS for whatever reason.
Because of the way Session is used, that will introduce a breaking
change so the CMS must plan to bump to the next major version.

Now, let's say, however, that we upgrade the MVC packages
significantly (like they don't need some further decoupling) to v3.

Let's say the CMS is already using MVC@2.y and we have to assume some
custom sites are relying on the new interfaces.

For the CMS to upgrade to MVC@3.0 it needs to consider bumping its own
major version. But in this case, does it really need to?

If you can put MVC@2.y and MVC@3.0 beside each other, then we can have
core running on @3.0 but custom work is still using the @2.y version
of MVC and running quite happily thank you very much.

When the CMS does get around to bumping to the next major version, it
can drop MVC@2.y completely.

The question is how can we have those running side-by-side. I think
there are a number of options and I put to you that namespace is just
one option, but I also put to you that we can worry about that bridge
when we come to it. Making a deliberate decision to reform the CMS
architecture because we are the developers that should be doing it is
far more important.

> In truth, I think I can live with the hole we've dug ourselves into with 30+
> individual components going on 30 different paths as far as versioning goes,
> but only within a major branch. I don't think it'd do us a lot of good to
> have Database v3, Application v2, and Registry v4 all floating around as the
> "active" versions, that'll get confusing quickly (and yes I know that's
> really hypothetical but I hope it illustrates one of the things I dislike
> about a scenario that idea allows to happen).

No, the idea of having two versions of Application and even Database
is nonsensical. They run the show so you need a strategy like I
outlined for Session above. I would do the same for Registry, but I
think Registry is a good starting point to pick on for major code
reform.

> I'm speculating on the v4 CMS codebase. You're suggesting that the API be
> versioned via namespaces (which the only way to do this with Packagist is
> creating new packages for every major release because that and Composer
> won't let you install two versions of the same code),

At least it is today. We could talk to Jordi about way we could
potentially solve this problem via Composer.

> so in theory you open
> the door for a CMS extension running on a potential v4 of the CMS which is
> running the CMS v3 "legacy layer" and potentially uses Framework v1 code.
> Reading this, it's really confusing, but it makes sense in my head, I swear!

As I outlined, there are is a line in the sand as to what is practical
to version and what is practical to upgrade. Some packages are going
to be very close to that line, but I think there is enough work to be
done that it doesn't have to be hit for some time.

The reality is, if we invent a better Joomla Component paradigm,
developers are going to try and make their own bridging layers anyway,
and they never do it well and your CMS ends up polluted with 50
different kinds of MVC triad wrappers (and don't get me started on
template frameworks - what is happening now it absolutely nuts). So
it's better that the core set the standard for how old and new should
live together, and one day you just have to break things. My premise
is, however, that we can do a lot with 3.x with smoke and mirrors to
avoid the fear and uncertainty that surrounds the number 4.

> Of what's there today, the only package with a roadmap idea that may have
> B/C implications is splitting off the Registry package's loader classes to
> form some sort of abstract layer usable for different purposes (one idea
> floating around is adding support for more file formats in the Language
> package, which would serve the PHP community moreso than Joomla most
> likely). That also implies that no other Framework packages will get merged
> into the CMS for the remaining lifetime of its 3.x branch. If the Language
> package gets integrated back into the CMS, that one would have B/C
> implications going between major versions. And on the MVC side of things,
> the current v2 View package is a fair bit different than the previous
> version.

Bottom line is there needs to be a plan and some of the decisions made
to date may make it difficult to move forward in the best possible
way.However, if the FW Team is just doing it's own thing, there is no
point to be attached to the drama of the Joomla community. It would be
much better to leave. If one of the roles is to serve the CMS, then we
need a plan and the CMS needs to play ball as well. Given that George
is here, I don't see that as a problem.

I think, perhaps, Registry, Language and MVC would be good places to
start. Make sure they are SOLID and work up from there. A bigger
project is looking at how extensions themselves could be installed
with Composer and I think we'd need to invent a lot of the engine to
do that.

As far as the Team goes, however, I think we need to change the "do
your own thing" attitude and having most people working to solve the
same problem most of the time, or having the same short term goals
most of the time.

Does that make sense?

Regards,
Andrew Eddie

[Michael Babker]

Conceptually, it makes sense and has merits.  Conceptually, the idea solves a major problem that we ourselves created due to poor change management and decision making.  It's the execution I'm not confident in, or the idea to redefine PHP standards (building an environment where multiple versions of a software package could be included in a single application).

I see conflicting visions in how the software is developed.  We want to move toward a modular application environment where each API component can take on a destiny of its own.  In that vision, I see a collection of 30+ individual and isolated components doing their own thing and loosely grouped under a single brand.  Not one component has been changed to take advantage of our own dependency injection code; is our aim very loosely coupled packages full of good ideas or to provide a strong application framework to build upon?  We also want to build the next generation of the application our brand is known for using these modular and independent moving components.  In an environment where historically we can't decide whether to meet for lunch tomorrow or not.  So I have concerns that we (as a whole) could execute any plan that calls for several independent moving pieces to come together to form a single application framework, without taking into consideration that those components could potentially be restructured in a way that multiple versions of them could conceivably be run in parallel.

If I take my developer hat off and look at this idea as an end user, I see "that's amazing" written all over it; upgrading to the next major of the version is nowhere near as painful as before and I don't even need to be concerned my extensions aren't natively compatible, they'll just work.  So yes, for the 99%, it's a good idea that should be followed through on.

For the 1%, those people getting their hands dirty producing the product and the extensions that makes Joomla's ecosystem function, it's a hit-or-miss thing.  On one hand, devs won't have to update their extension code anytime soon (and even if the CMS dropped compatibility for something, the extension should be able to include that code on its own anyway by that point, so no big deal for them).  On the other hand, it would destroy any possibility of majorly overhauling and modernizing the application because the CMS would be burdened with ensuring that somehow, a subset of APIs still work as advertised.  As far as the code structure goes, my worst case scenario vision is a platform with so much spaghetti code that it makes WordPress' PHP code look beautiful.

I don't know the right way forward.  But in my heart, a proposal that versions the API via class names or creates a need for separate packages per major branch does not feel right, even if it aims to solve our own problems.

Regards,
Andrew Eddie

[Andrew Eddie]

On 21 May 2015 at 09:24, Michael Babker <michael...@gmail.com> wrote:
> Conceptually, it makes sense and has merits. Conceptually, the idea solves
> a major problem that we ourselves created due to poor change management and
> decision making. It's the execution I'm not confident in, or the idea to
> redefine PHP standards (building an environment where multiple versions of a
> software package could be included in a single application).

PHP does it, just not in the same way (compare functional and OOP
versions of Array handling API).

> I see conflicting visions in how the software is developed. We want to move
> toward a modular application environment where each API component can take
> on a destiny of its own. In that vision, I see a collection of 30+
> individual and isolated components doing their own thing and loosely grouped
> under a single brand.

I think that was a fatal mistake and it's led to the chasm between the
CMS team and the FW team. And as I say, if that's the case, there are
better brands than Joomla to be under.

> Not one component has been changed to take advantage
> of our own dependency injection code;

That's a chicken and egg problem. But on that point, DI is very easy
to abuse so let's chalk that up to a good thing that they haven't
developed any bad habits yet.

> is our aim very loosely coupled
> packages full of good ideas or to provide a strong application framework to
> build upon?

I cannot say this enough. Like it or not, the Joomla brand means CMS.
If this Team is a framework club, ditch the Joomla brand and align
with Phil Sturgeon or someone else. At worst, the FW team needs to
coax the CMS into thinking it's working for them, even if the
motivation is not primarily to make the CMS better (just a means to an
end). It's not that devious, the everyone can win.

> We also want to build the next generation of the application
> our brand is known for using these modular and independent moving
> components. In an environment where historically we can't decide whether to
> meet for lunch tomorrow or not.

Debate and decide - that's the point of the new Team structure isn't
it? This mentality has got to change. People that want to argue and do
nothing (but generate click-love to their personal blogs) just need to
butt out.

> So I have concerns that we (as a whole)
> could execute any plan that calls for several independent moving pieces to
> come together to form a single application framework, without taking into
> consideration that those components could potentially be restructured in a
> way that multiple versions of them could conceivably be run in parallel.

Yes but you have to start somewhere. Let me be brutally honest. Today,
right now, there is not point in the FW Team even being here other
than it's under the Joomla brand and moving to another brand means we
have to start again building our reputations and street cred. It's a
club at best, and a poorly attended one at that. If that's all it is,
it's a waste of resources and the PLT should decide to just nuke it
because it's a distraction to the greater whole.

> If I take my developer hat off and look at this idea as an end user, I see
> "that's amazing" written all over it; upgrading to the next major of the
> version is nowhere near as painful as before and I don't even need to be
> concerned my extensions aren't natively compatible, they'll just work. So
> yes, for the 99%, it's a good idea that should be followed through on.

Spot on - the user doesn't care. Just don't break anything.

> For the 1%, those people getting their hands dirty producing the product and
> the extensions that makes Joomla's ecosystem function, it's a hit-or-miss
> thing.

The goal is to make it far less hit-and-miss than it is today. Those
1% of people, and I'm one of them, are experiencing 99% frustration.
If that can be lowered 10% a year it's a win.

> On one hand, devs won't have to update their extension code anytime
> soon (and even if the CMS dropped compatibility for something, the extension
> should be able to include that code on its own anyway by that point, so no
> big deal for them). On the other hand, it would destroy any possibility of
> majorly overhauling and modernizing the application because the CMS would be
> burdened with ensuring that somehow, a subset of APIs still work as
> advertised.

That's what major version increments are for. The point here is to
either soften that blow, or delay it for as long as possible.

> As far as the code structure goes, my worst case scenario
> vision is a platform with so much spaghetti code that it makes WordPress'
> PHP code look beautiful.

Sounds like a typical Joomla site with 10 totally disappoint
extensions installed. How many times to we out in the real world have
to say things are a complete mess for people to start listening? It
can't get any worse.

> I don't know the right way forward. But in my heart, a proposal that
> versions the API via class names or creates a need for separate packages per
> major branch does not feel right, even if it aims to solve our own problems.

Ok this is going round in circles. I'm just going to have to wait
until you and George are available to chat in real time.

To be honest it's sounding like we can't do anything because it's all
too hard. And the fact that nobody else is chiming in leads me to
think that this is a dead end street to try and at least have a go at
fixing something - anything. If it's going to take 6 months just to
decide whether we have lunch today, it's not worth my time.

Regards,
Andrew Eddie

[Nils Rückmann]

Am Donnerstag, 21. Mai 2015 01:59:29 UTC+2 schrieb Andrew Eddie:

> And the fact that nobody else is chiming in leads me to 

> think that this is a dead end street to try and at least have a go at 

> fixing something - anything. 

Well, maybe others are waiting to see the output of your two-person dialog and don't want to interfere ;)

Honestly i think the discussion has moved far away from its topic, which unfortunately happens all the time on Joomla mailing lists, but also discovers the real problem: There is no real plan for the FW. I remember a year (or even more) ago i asked about the relationship between Framework and CMS, because i thought it might be a new base for the CMS, but everyone shouted the FW would be independend and does not need to worry about the CMS. Now, it seems like every decision for the FW is coupled to the CMS. So what's the reason for the debacle (plattform, framework, legacy, cms, fof or not fof, etc.) in the first place ?

[Andrew Eddie]

On 21 May 2015 at 11:51, Nils Rückmann <> wrote:
> Well, maybe others are waiting to see the output of your two-person dialog
> and don't want to interfere ;)

Sorry - I exaggerated too much. Didn't mean to ignore you. But I think
the handful of people that are interested in the FW is not enough to
justify its existence as team, given the amount of code it is
responsible for.

> Honestly i think the discussion has moved far away from its topic, which
> unfortunately happens all the time on Joomla mailing lists, but also
> discovers the real problem: There is no real plan for the FW.

That's part of it. From the "I'd like to contribute, but will my
contribution be effective" point of view, that's a real problem.
Everyone's time is limited, and we need to know that we aren't wasting
it without need (that's not to say we can't make mistakes, or do work
that we find doesn't end up working the way we expected). Where do the
people that want to build a ladder out of the hole go?

> I remember a
> year (or even more) ago i asked about the relationship between Framework and
> CMS, because i thought it might be a new base for the CMS, but everyone
> shouted the FW would be independend and does not need to worry about the
> CMS. Now, it seems like every decision for the FW is coupled to the CMS. So
> what's the reason for the debacle (plattform, framework, legacy, cms, fof or
> not fof, etc.) in the first place ?

The Framework started out as a way for some of our best developers
(me, Sam, Louis and others), to be able to code without having to
constantly argue with the CMS maintainers and users about "you can't
do that because the CMS does not use that code". The Framework allowed
a good group of developers to work on other interesting stuff without
having to deal with the politicking.

That was the theory, the trouble is the politicking didn't stop and
everyone ended up leaving because they had just had enough in one
fashion or another.

So in one respect, the FW has been doomed from the very beginning, not
because the idea is wrong, but because the community allows and
tolerates too much rubbish to be thrown at its best contributors. To
fix that, however, requires strong and gutsy leadership and, nothing
personal, but we have simply not seen that in Joomla leadership for
many years. The reason, I think - and I understand and empathise, is
out of fear of causing yet another drama.

Anyway - all I'm trying to do is work out how we can fix the rubbish
code in Joomla and in the process we get some really good packages
that someone else might be interested in along the way. I don't see
that this team has any relevance unless it is working to improve the
CMS.

OR, the PLT needs to restructure the teams so that people that want to
talk about code and talk about code without users interjecting because
they simply don't understand what's being discussed.

But no matter how you slice is, code reform is needed and the scope is
simply massive (so you are going to need all the experience you can
get). How on earth do people help make that happen?

Regards,
Andrew Eddie

[Nils Rückmann]

Am Donnerstag, 21. Mai 2015 04:13:46 UTC+2 schrieb Andrew Eddie:

So in one respect, the FW has been doomed from the very beginning, not
because the idea is wrong, but because the community allows and
tolerates too much rubbish to be thrown at its best contributors. To
fix that, however, requires strong and gutsy leadership and, nothing
personal, but we have simply not seen that in Joomla leadership for
many years. The reason, I think - and I understand and empathise, is
out of fear of causing yet another drama.

I'm not sure if we need a "strong" leadership, but like every social group which is following a goal, there has to be a responsible mediator who can lead a group without forcing them into his direction only.

  

OR, the PLT needs to restructure the teams so that people that want to
talk about code and talk about code without users interjecting because
they simply don't understand what's being discussed.

A first step might be project leaders for single packages. If i want to improve a package, why i need to go through a full FW leading tribune ?

 

But no matter how you slice is, code reform is needed and the scope is
simply massive (so you are going to need all the experience you can
get). How on earth do people help make that happen?

I proposed a solution a while ago: Making a straight cut and be able to produce without unnecessary dependencies and politics like "Hey, we want awesome code, but it should fit in the current mess". You can't build a plane by using the spare parts from your old car. And it has nothing to do with moving away from Joomla, it's just about building the future now. Typo3 is a great example with Neos. I thought "Icarus" (still the worst title ever) might be the right step, but unfortunately it fell like it's eponym.

This is only my personal opinion as a person who works which Joomla since years, always trying to stay in touch with the core without being successfull, but seeing how many people are leaving Joomla is really frustrating.

[Andrew Eddie]

> This is only my personal opinion as a person who works which Joomla since
> years, always trying to stay in touch with the core without being
> successfull, but seeing how many people are leaving Joomla is really
> frustrating.

I'd love to start from scratch, but my concern is the community would
not forgive us "again" (after 1.6). I think the only practical way to
upgrade the CMS, and justify this team, is to do it by stealth.

Regards,
Andrew Eddie

[Rob Clayburn]

Hi This has been a really good read for me.

I don't think re-building the CMS from scratch is a good idea. As a coder its always tempting, but ultimately its always a tricky path to navigate and one which is hard to sell to stakeholders be they 3rd party devs, site maintainers etc. That's why I think Andrew's idea of slowly implementing small sections of the framework in "stealth" mode is a really good idea. If I understand correctly that has already started to happen with some of the vendor/ code that is currently in the cms.

What we need to do is get developers enthused about writing for the Joomla project. 

The is a massive base of people with varying degrees of competency in writing PHP for Joomla, the project needs to engage those developers and show them that they have a place within the project and produce strong guidelines on what to write when. With that in mind I like Andrew's concept of sprints on certain areas of code. You can have a couple of architects draw up an iterative process that advances the CMS/FW and then launch the code monkeys along that path. 

This is massively more productive that the 'write it and perhaps we'll use it' attitude that is currently used, and which is a disaster. As a dev I want to know that what I'm writing is relevant and that I'm writing code in a manner that will be accepted into the code base. 

If you mentor new devs supplying them with the best practices to ensure their code is correct (writing tests, code style etc). The iterative process will produce a positive feedback loop where by devs see the fruits of their labour in a short time, are encourage and enthused by the results and then want to repeat the process, the complete opposite of the current negative feedback loop when you bust a gut for a month on something you personally deem important only to have someone block it based on some arbitrary requirement you weren't even aware existed before you started.

This is also why I'm against say "lets start again from scratch". It reduces confidence in the merit of incrementally contributing to the existing code, and leaves those dev's not capable or willing to work out an entire new architecture watching from the sidelines. 

95% of those developers will be coming from the CMS, and that is where their interest lies. When I do a custom project realistically I am not going to use the Joomla framework, I'd sooner work on Express or Laraval, or pick individual composer projects few of which would be those available in the framework. So for me the Joomla Framework is only relevant IF its directly feeding into the CMS. I'd love to see that change and for the framework to become my defacto choice, but it will only ever get there if there are a lot more people contributing to it, and to get those contributors we need to set up a short positive feedback loop in the contribution process. 

With regards to Andrew's idea of version'd classes. To my mind there are definitely certain classes which you would not want multiple versions of. JSession, JApplication etc. Basically any class which is required to be initiated before the call stack arrives at a component/plugin entry point should be a known quantity for a 3rd party developer. However MVC etc could have different versions, indeed we already have that in the fact that JModel and JModelLegacy are already in the CMS. I'm currently refactoring Fabrik to use namespaces and the new MCV classes. What I hadn't really appreciated before starting was that it really does allow you to easily swap out class definitions. So you can start with :

use \JRegistry as JRegistry

and at a later stage I imagine I could replace that with 

use \vendor\Joomla\Registry\Registry2 as JRegistry, and as long as the two classes implement the same interface all is going to work as before.

So that could enable rapid iteration within the framework and deployment to the CMS (at least for those modules which are not umbrella requirements for the CMS)  with little backwards compatibilty issues.

perhaps some of the short term sprints we could look at would be related to namespacing the current core components? I've always learnt how to code in Joomla by taking those components as a reference for how things should be done. If 3rd party dev's start seeing that type of change then they are naturally going to start to mimic that within their own code. This would have the double impact of helping them update their PHP practices and also increase their potential usefulness to the project

 

[Nils Rückmann]

Am Donnerstag, 21. Mai 2015 07:37:59 UTC+2 schrieb Andrew Eddie:

I'd love to start from scratch, but my concern is the community would
not forgive us "again" (after 1.6). I think the only practical way to
upgrade the CMS, and justify this team, is to do it by stealth.

We have to separate between users and developers. Developers should be thrilled about a modern API unless they are that kind of developer which want to make money with the least effort.  And the reason why users blame Joomla for 1.6 is not 1.6 per se, it's the fact that there weren't a solution to migrate without a bunch of investments. 

But i don't think that migration would be problem as it would be a new product. There are concerns about dividing the community and manpower, but that's just math.

You can't just "we will loose x% manpower, because they will work on the future release only", you have to see how many developers you are loosing if nothing changes.

Even with a ability to migrate the data to a new Joomla i think that building a new application + migration tool is gonna be faster than thousand little steps where each step has to be discussed for weeks.

[Nils Rückmann]

Am Donnerstag, 21. Mai 2015 09:29:45 UTC+2 schrieb Rob Clayburn:

If you mentor new devs supplying them with the best practices to ensure their code is correct (writing tests, code style etc). The iterative process will produce a positive feedback loop where by devs see the fruits of their labour in a short time, are encourage and enthused by the results and then want to repeat the process, the complete opposite of the current negative feedback loop when you bust a gut for a month on something you personally deem important only to have someone block it based on some arbitrary requirement you weren't even aware existed before you started.

This is also why I'm against say "lets start again from scratch". It reduces confidence in the merit of incrementally contributing to the existing code, and leaves those dev's not capable or willing to work out an entire new architecture watching from the sidelines. 

Interfaces first ;) The architecture is mostly defined by it's interfaces. So after the interfaces are settled, there are plenty construction areas where everyone can contribute.   

 

95% of those developers will be coming from the CMS, and that is where their interest lies. When I do a custom project realistically I am not going to use the Joomla framework, I'd sooner work on Express or Laraval, or pick individual composer projects few of which would be those available in the framework. So for me the Joomla Framework is only relevant IF its directly feeding into the CMS. I'd love to see that change and for the framework to become my defacto choice, but it will only ever get there if there are a lot more people contributing to it, and to get those contributors we need to set up a short positive feedback loop in the contribution process. 

Another point for "building from scratch". The current CMS libraries were never been written to be used in another application, that's history conditioned. But if we would have the freedom to build a FW and a CMS as an application of that FW, the FW can still evolve while the CMS picks it needs. Of course, there will always be the CMS in the mind. But there's a difference in building a package for the CMS and building a standalone package which also could fit in the CMS. 

 

With regards to Andrew's idea of version'd classes. To my mind there are definitely certain classes which you would not want multiple versions of. JSession, JApplication etc. Basically any class which is required to be initiated before the call stack arrives at a component/plugin entry point should be a known quantity for a 3rd party developer. However MVC etc could have different versions, indeed we already have that in the fact that JModel and JModelLegacy are already in the CMS. I'm currently refactoring Fabrik to use namespaces and the new MCV classes. What I hadn't really appreciated before starting was that it really does allow you to easily swap out class definitions. So you can start with :

 

Well, if we are particular about that idea, than the only challange would be the application package as it's the crucial point. Of course, i could use two different session packages at the same time, they just have to be separated. Think about a plugin which starts a Session2 on initialisation. The real question is which session is bind to the application. But since total API changes for a session package are very unlike, i think we don't have to worry about it. Just throw in a legacy wrapper before the next major release.