A Half-Baked Experiment

[Michael Babker]

To save the time for those who might not be interested, if talking code and application architecture aren't your cup of tea, you can skip this e-mail.

With that out of the way...

The CMS application structure is largely unchanged since the release of 1.5 in 2008.  This means that we've carried forward a lot of application designs and practices for 7 years that are starting to show their age or prove difficult to work with.

As a concept, I've started playing with some of the top APIs of the CMS application and started looking into a modernized architecture without doing too much damage with regards to how things are pieced together (the filesystem skeleton mostly resembles what we've all come to know, mostly because a lot of it isn't there at the moment).  The current status is a half-baked experimental code repo, found at https://github.com/mbabker/joomla-next

As this is a concept only right now, it's in essence a clean start.  Bear this in mind as someone will probably ask "well what about B/C?", to which I will simply respond that when playing with concepts and experimenting with different structures, B/C doesn't exist.

So what's different?

The "base" API is grouped, for better or worse, into two folders: libraries/core and libraries/vendor.  The core folder is the CMS' own code while the vendor folder is a Composer installation of packages.  As this is all experimental, Framework v2 packages are used where able for no other particular reason than that's what I decided on.

PHP 5.4 is the minimum here.  5.3 is dead, and 5.4 is even pushing it.  But, I wanted to be able to use PHP traits at a minimum, so there's that.

No JFactory.  Yep.  Trying to focus in a DI/Services oriented structure.  I have Factory objects for some packages, but the only thing singleton about most of them is a reference to the DI Container.

Layout rendering.  Framework v2 introduces a refactored View package for the MVC triad and the HTML class looks for a RendererInterface.  The implementation in this concept is in essence a beefed up JLayoutFile object with some tweaking to fit the interface.  There isn't support for a component/view/tmpl folder here; the entire thing is designed to enforce a separation of the layouts from the MVC triad (even the method rendering the layout file is scope isolated so as to not allow access to any properties of the LayoutRenderer class; the layout only has what we have given it).

Redesigning JDocument.  The LayoutRenderer is left to fetching and rendering layout files while the Document package is left to handle the rest of the output document.  The base object doesn't have all the HTML stuff crammed into it anymore.  It still has support for the <jdoc> tags and is responsible for actually echoing out the fully assembled template.  The goal here is to put the document object (and inherently the actual template file) in charge of all things output; the template file isn't even scope isolated and still has access to the full document API (same as the existing structure).

Application Interface.  The whole application is heavily geared toward web requests with an HTML response.  What's in place now tries to reshift this focus by defining a proper interface and base application API so that things should be able to run whether through the web or on the command line and serving HTML or RSS content without hacks.

What's there now?

It's a really really thin skeleton.  There's a site application echoing hello world and supporting the Framework's Profiler package.  Nothing else here yet.

The installation application has gone a little further.  It has the bases of its MVC structure set up and can somewhat render the first page of the installer.  There aren't any JHtml type objects in place right now, nor is the HTML document API set up, so it's just a raw HTML output.  It does somewhat demonstrate using the factory objects here and starts enhancing the Framework code as needed to suit the CMS needs (for example the form factory enables dependencies to be injected into form fields now, see the language field code).

Why share this?

I really don't know.  I'd really like to see the application structure cleaned up some and shed some of the legacy baggage that does make things a little difficult at times.  Maybe I just need someone to slap me around and say "we're happy with what we have, don't go changing it on us".  Maybe this will serve as an invitation to let folks collaborate around code and share ideas (something that's failed miserably a few times in the last couple of years, one can have hope though).

Either way, it's out there now for all to see.  Enjoy it, tear it apart, whatever floats your boat.

[Dmitry Rekun aka b2z]

Well that's interesting Michael. Thanks for sharing.
I see that you left administrator folder. Do you think that we should separate the application? Or it is just there because you did not think further about it? ;)

Dmitry

[Fedir]

People, please do not slap Michael, it is good initiative to throw very old legacy, and make things good ;)

thanks man for sharing

понеділок, 23 березня 2015 р. 09:39:12 UTC+2 користувач Dmitry Rekun aka b2z написав:

[Nicholas Dionysopoulos]

I fully agree with you, Michael. The current architecture is seriously outdated. What you're doing is really good. I have one question and one observation.

Question: Are we still allowed to circumvent the renderer and provide our very own data representation, be it HTML or anything else (CSV, JSON, Office XML, bespoke XML, video, image, audio, something none of us can think of)? I am asking both for objective –what about all the data representations we can't think of– and selfish –I have a framework which outputs pre-rendered HTML, JSON and CSV– reasons.

Observation: While I agree 100% that JFactory is evil you could provide a bridge class in Joomla! 4, removed in Joomla! 5, to ease the transition pain. The main index.php will be creating a DI container, so why not set it by reference to a JFactory class which exposes its services? Calls to these old methods would produce a deprecated log entry. This would help convert legacy code. I'm speaking from experience. I understand it's a bit architecturally problematic, but it's the easiest way to tell developers to, let's say, go through the application object and get its container to grab the user object instead of going through JFactory. Mark the class as deprecated, put the new method in the docblocks, add the deprecated log message and there you go.

FYI, playing around with other frameworks and rolling my own –for Joomla!, standalone and WordPress– I did come up with the same conclusions on what is necessary for a modern architecture: PHP 5.4 or later, namespaces, traits, DI container, break b/c hard. I understand that some people will grab their torches and pitchforks at the mere sight of the words "b/c break". I'm all against b/c breaks for no reason. But now there is a damn good reason and I'm all for it! The good reason being the need for Joomla! to stay relevant in this decade instead of withering and dieing. So yeah, break b/c hard since this is what it takes for J! to be relevant for yet another decade. Rewriting a complex component when the underlying framework breaks b/c hard takes all of 20 days: been there, done that, got the public repo to show for it, so what's your excuse? :)

Michael, thank you for your initiative. I love what you're doing! Keep it up!

[Bakual]

I like the idea this is going. Glad to see constructive discussion starting. Thanks for that :)

As for B/C. With a major version we can obviously break things. I don't mind much if it's a complete rewrite breaking everything or more a few little things with proper deprecations first.
However what I think is very important is how the user will experience it. In the end we will need a simple, reliable working migration path. However I agree with Michael we can leave that topic for the moment. It's more important now to see where we want to go. If we know the goal, we can choose the best way to go there.

For example if we have a completely new Joomla, a way to migrate the content would be to migrate all data into a new instance instead of updating the existing one. If webservervices go into 3.6, it could be as easy as pressing a button in the Joomla next which will then fetch the data from the old site. I'm sure we find a better way than what we had from 1.5 -> 2.5 :)

[Soren Beck Jensen]

Great to see some initiative with this. 

One of the main complaints we hear from developers is breaking B/C but we also hear them complain about old fashioned code structure. I personally would love to see Joomla developed from ground up being more modern, something that we as developers can enjoy working with. Maybe then it will not be such a pain to upgrade components either.

Great work Michael, looking forward to following the progress and discussion.

Soren

[Nikolaos K. Dionysopoulos]

Thomas,

The way I see it migration consists of two distinct things: code and data.

The major ailment of Joomla! 3 is code. The data structure is quite alright, even though I have my reservations about UCM. But since we’re now using UCM for a valid purpose (versioning) I’m willing to accept its necessity. To cut a long story short, the data structure doesn’t need to change. This means that a migration would merely require copying over the database tables to the new site which is feasible.

From the code’s perspective, the migration won’t be as simple as a single-click upgrade. There’s almost no way to provide an extension package which will work with both Joomla! 3.x and 4.0. This will put off some people but our alternative to that is not fixing the architectural problems which is a bigger problem (I full-heartedly agree with Soren – build the thing from scratch to make something developers can enjoy working with).

So, how would a migration work? First create a new J! 4 site. Then import the database from your J! 3 site. As luck would have it, I already have the technology to create a reliable solution for that and you know I’ll volunteer for it. I can imagine a database-only backup component which exports a package to be used by an import feature in com_joomlaupdate. If some minor schema changes are required we can have the SQL files to run against the site’s newly imported database. There’s already provision for that in com_joomlaupdate's restore.php. These features wouldn’t take me more than 2 weeks to fully implement. Consider that my offer to volunteer for such a feature :) Back to our migration. Then install the J! 4 versions of your extensions. Finally make all the necessary finishing touches, assuming that the routing changes result in a different URL structure on the site. It’s not as simple as one click upgrade but it’s not the pain and agony of 1.0 to 1.5 or 1.5 to 2.5 “migration” either.

That said, we can work on data schema changes –if necessary– in Joomla! 5. It’s best to tackle one side of the legacy system (code, data) at a time. Less moving parts equals better chance of success.

Nicholas K. Dionysopoulos

[Roberto Segura]

OMG you are breaking B/C there. We cannot go that way!!

Joking :D

I love it Michael. Thanks for taking the lead ;)

[Leo Lammeirnk]

Finally somebody who dares to put this entire B/C stuff to the table for discussion... Discussion? I see no discussion..... Michael did put the lid off the can of B/C-worms and Nicholas already gave a very good assessment with a great offer to create the migration facilities so how do we go on from here so we can get this moving on? F*** B/C for sure I agree with Nicholas when we need to think of the future of Joomla (!)  Great Michael ..... how we proceed? what could be the next steps?

Leo 8)

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send email to joomla-...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-cms.
For more options, visit https://groups.google.com/d/optout.

[Vic Drover]

B/C in Joomla 4 was part of the last release document IIRC, so this should not be a surprise to folks. I'm interested to see this move forward.

Cheers,

Victor Drover
Founder and CEO, Anything Digital
Co-founder, Watchful.li & jInbound.com
262-309-4140
Facebook: AnythingDigital | watchfulli | JInbound
Twitter: @VicDrover | @AnythingDig | @watchfulli | @JoomlaInbound

[Nils Rückmann]

HI Michael, glad to see someone is thinking beyond the current state. IMO there is a lot more B/C breaking necessary to get Joomla in a new age of CMS.

[Bakual]

That was my thought Nicholas. I just wanted to say that since code would not be compatible we need two instances and need to migrate the data over in some easy way for the user. Be that using webservices or an export/import function isn't important currently. It's just important that the user isn't feared already by the thought of a B/C. Once finished, we will have to provide an easy way, like it was promised for 2.5 but never happend.

Obviously, if we can make the code B/C less breaking, that would be awesome. Maybe even do it in steps with an intermediate "compatibility" version. But that's for later.

First see where to go :)

[brian teeman]

As long as we dont go the 1.5->1.6 route of ignoring the migration process completely and leaving that to 3rd parties I am all for progress. 

[Michael Babker]

The repo itself started with just the skeleton of a CMS install.  I hadn't really thought far into the "merge site and admin apps" piece yet (obviously).

--

[Michael Babker]

On Mon, Mar 23, 2015 at 5:41 AM, Nicholas Dionysopoulos <niko...@gmail.com> wrote:

I fully agree with you, Michael. The current architecture is seriously outdated. What you're doing is really good. I have one question and one observation.

Question: Are we still allowed to circumvent the renderer and provide our very own data representation, be it HTML or anything else (CSV, JSON, Office XML, bespoke XML, video, image, audio, something none of us can think of)? I am asking both for objective –what about all the data representations we can't think of– and selfish –I have a framework which outputs pre-rendered HTML, JSON and CSV– reasons.

The install app which is the only thing with concrete code is pretty stuck to the "new" stuff.  But that's perfectly fine.  In the main app, everything should be flexible.  Even the DocumentFactory has support for loading custom Document objects that aren't in the Joomla namespace, a huge step forward from JDocument::getInstance().  Same should apply to the renderer implementations; if your RAD framework builds direct from the Framework's base classes then you can inject in whatever you want that meets the interface requirements.  The interface (https://github.com/joomla-framework/renderer) was designed mostly around Twig, Mustache, and Symfony's templating component, but as the repo shows JLayout can be made fully compliant with it too.  My thought process is to give options where practical without locking into hard implementations.

Observation: While I agree 100% that JFactory is evil you could provide a bridge class in Joomla! 4, removed in Joomla! 5, to ease the transition pain. The main index.php will be creating a DI container, so why not set it by reference to a JFactory class which exposes its services? Calls to these old methods would produce a deprecated log entry. This would help convert legacy code. I'm speaking from experience. I understand it's a bit architecturally problematic, but it's the easiest way to tell developers to, let's say, go through the application object and get its container to grab the user object instead of going through JFactory. Mark the class as deprecated, put the new method in the docblocks, add the deprecated log message and there you go.

Very easy to accomplish.  Actually, in a work project, I have a Factory object (https://gist.github.com/mbabker/665da3e15ecc1479ba07) serving this type of role.

[Michael Babker]

Migrating the core platform between major versions is the responsibility of the core project.  Whether that be built right into it or a standalone tool (I actually prefer the later, it's easier to upgrade/migrate an application when you aren't actually executing that same code), it must be something that is provided by Joomla.  Failure to do that and we can all start learning how to build on WordPress.

On Mon, Mar 23, 2015 at 8:39 AM, brian teeman <joom...@googlemail.com> wrote:

As long as we dont go the 1.5->1.6 route of ignoring the migration process completely and leaving that to 3rd parties I am all for progress. 

--

[Nikolaos K. Dionysopoulos]

Awesome! My framework has its own DI container, so I could be exposing the Joomla! container as a service of it. Since everything is extensible I am a happy bunny :)

Nicholas K. Dionysopoulos

Director, Akeeba Ltd

https://www.AkeebaBackup.com

--
You received this message because you are subscribed to a topic in the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/joomla-dev-cms/Zhkilq_uRNI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to joomla-dev-cm...@googlegroups.com.

[dgt41]

Excellent initiative Michael and thank you for that!

I have a question on renderer: do you really need a head.php? Providing the arrays of data to template isn’t a better approach? What I don’t like with the current CMS and something I won’t like to see it in the next version is the imposed ways of doing things. My point of view is that all the head (or not head ) links of assets metas etc should be template’s responsibility, not core functions, or am I wrong here?

I totally agree on all the comments about the upgrade path!

Thanks again!

[Michael Babker]

Still something to think through.  If we went the route of the template manually rendering the head and just providing it the raw storage arrays, we'd also need a proper API for registering media.  If my com_helloworld layout uses some feature from jQuery UI, I need a way to tell the application to use this script from this location whether that be from a JHtml like helper or $doc->addScript().  It should be an API that in theory is only accessible to the rendering parts of the system (Document and Renderer packages), but it is one that is needed in some form.

--

[Nikolaos K. Dionysopoulos]

Michale,

I am very strongly against passing raw arrays to templates. My experience tells me that if there is one way for the template developers to not respect the extension developers’ media inclusion they will do it, breaking our components and not even accepting the responsibility even when we provide them with hard evidence. I’m going to great lengths to make sure that their use of plugin events to post-process the generated HTML doesn’t end up removing my Javascript because its name contains the letters “jq” (don’t laugh, it’s a far more serious issue than it sounds). If you give them easier access to screw up my scripts or, God forbid, rearrange them on the page we’ll end up with every developer using their own way to include scripts on the page.

Unless, of course, the Joomla! project can FINALLY exercise some control on template developers so that they stop breaking Joomla! with their ill-thought code. In this case I don’t care what you give them to work with, as long as all templates FINALLY render media inclusion in the same, predictable way.

[Michael Babker]

That's why I say it's still something to think about.  I could see some sort of asset management API integrated into the Document classes and things continue to be left to the <jdoc> parsers to handle outputting.  This asset API should also be able to render scripts to the body, something that's been requested a few times in the JDocument API.  I don't necessarily think it's a good idea to provide them raw data access, which is in part why the LayoutRenderer isolates the scope of the layout it's actually rendering to prevent getting to the class' API except for where we want it to.

[Justin Seliga]

If the idea of a "rewrite" without changing the database structure is on the table, I'm a proponent of throwing every idea at the wall, especially those that challenge the current paradigm of the Joomla! CMS, see what sticks, or spawns as a result. The ideas I introduce might be perceived as "feature requests" since I lack the experience to truly speak towards code architecture, but my intention is to spark some thought about maintainability and barriers to entrance not only from attracting new developers but also new users. Keep in mind, my perspective is of a novice developer, who has been a Joomla! user since 1.5 and started testing the development waters in 3.2, so please excuse any misrepresentations of the technical details or nomenclature abuse.

Is there still a need for two interfaces? The more pertinent question there, as a developer, why does Joomla! push towards having essentially two copies of the same code (front-end/back-end MVCs)? Take 3rd-party developers off the table for a moment, is the inefficiency of duplicating every feature or change in the front-end and back-end of core components worth whatever benefits it provides in separating them? Wouldn't it free up time for new feature development, and in turn growth of the CMS, if there wasn't so much duplication? After all we would expect the same (model) dataset to be returned regardless of the interface we request a list of articles from. These all-in-one "front-end only" components could include additional MVCs for the configuration/administrative tasks of that component, potentially eliminating the headache inducing inconsistencies like a plugin state returning true/false in the front-end but string 0/1 in the back-end. https://github.com/joomla/joomla-cms/pull/6443

Perhaps there is a way to meld the concept of an administrator interface into the front-end. The Document could inject a menu module (top navigation bar or off-canvas side bar) that looks and operates identically to the current administrator navigation bar. This of course wouldn't be exposed to template developers. All component installs would register their menu items as a child to the "Components" menu item of a menu type of say "admin" (handled by the installer logic) that can't ever be deleted.

This condensed interface approach might also be more attractive for new users. With the pending release of a completely re-written Drupal, Joomla! still has an opportunity to match Drupal in entering a newer generation of technology, while simplifying other aspects to grab some of the Wordpress market share as it lags behind.

[Michael Babker]

You're on the money when it comes to pitching ideas.  I want to be in a state where we're doing just that, throwing ideas around and challenging the status quo.  And I for one agree on the separated admin/site applications; we really should be able to accomplish this with one application, making efficient use of the ACL system, and a well defined interface in the application and MVC layers.  Consolidating what is essentially management and display of the components to a single set of classes kills off a good chunk of duplicate code and development time for extensions.  A component should not in essence be two separate extensions (site component and admin component).  And a module should be usable regardless of the app it is installed in, again not a site/admin thing.

--

[Mathew Lenning]

@Michael,

I'm glad to see that you haven't been lynched for opening up this conversation. I must have worn the mob out =^P

The Icarus repo was created just for this type of experiment, so you might consider opening a branch there.

Icarus is actually supposed to be a playground to promote a little gamification in the development process.

Rather than everyone just saying "You can't do that" or trying to tell you "You have to do it this way" (which is actually why issues were turned off in the first place).

Each branch is under complete control of the team behind it. 

Each team builds their version of Joomla without regard to outside opinion, other peoples B/C concerns, or any of the other road blocks that have prevented Joomla from evolving over the years.

Then eventually when/if an approach proves it has potential, the project can be opened up for community review/debate. Which by then the development team will probably have enough evidence to support their approach. 

It also gives us a chance to move the conversation out of "Theory" and into working possibilities. Because as we all know the gap between theory and reality is big.

Anyway I hope that you'll consider joining the Icarus challenge. It would be really nice to see some leadership taking the initiative in Icarus. 

It might even encourage some of the community members who have been silenced by the mob over the years to join in and share their ideas.

Happy Joomla!ng

[Fedir]

what about drop Modules, and allow render components on that positions? :)

I noticed that some module developers on JED tries accomplish what component should do

вівторок, 24 березня 2015 р. 08:36:52 UTC+2 користувач Mathew Lenning написав:

[Nikolaos K. Dionysopoulos]

Modules are a great concept. You don’t need to remove modules to achieve what you want (placing components in arbitrary places). What you need is HMVC which allows you to render a component’s view anywhere, be it the main component area, a module, or even used inside a plugin (think about custom HTML email templates). Having used HMVC in FOF since 2011 I can tell you that yes, this is exactly what you’re looking for.

Prerequisites to HMVC:

• Do not use the JPATH_COMPONENT* constants. Using them means that only one component can be loaded at a time on a page.
• Do not use the application JInput object directly. Each component instance needs its own.
• In other words, you’d better use a DI container for the component or suffer forever (hint hint)
• Use a dispatcher for each component. Right now the default controller you get in the component’s main directory fills this role. It just doesn’t know how to handle custom JInput objects.

Look, it’s not that hard to do that. The biggest challende is getting rid of the constants and using a DI container to grab the input. Once you get there HMVC is simple as pie.

[Alan Sparkes]

This looks like the most exciting Joomla discussion of the last few years.  Im not hardcore enough to write hmvc but I get the concepts and this really looks like the beginning of modernising Joomla. Watching with interest:)

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send email to joomla-...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-cms.
For more options, visit https://groups.google.com/d/optout.

--

Alan Sparkes

Joomkit Ltd
Web Design, Branding,UX & Development
CMS Consulting

online:    www.joomkit.com
contact:  in...@joomkit.com
support: sup...@joomkit.com
phone:  +44 (0)845 680 9513
mobile: +44 (0)7875189513

Company no :6682016
VAT no: 947906870

[Dmitry Rekun aka b2z]

Agree with Nicholas.

Dmitry

[Fedir]

thanks Nicholas for explanation,

that would be cool, 

вівторок, 24 березня 2015 р. 10:56:59 UTC+2 користувач Nicholas Dionysopoulos написав:

[Chris Davenport]

Before embarking on any radical change I think it is instructive to try to get a handle on what we're trying to achieve.   What are the high-level goals?  What characteristics of the current version are considered essential to maintain and what are the desirable characteristics that we're looking for in a new version?  Why is the current version not delivering those characteristics?

This will make it easier to explore different options because we will have an objective set of criteria against which each proposal can be measured.

Probably should start a new thread for that though.

I would encourage as many people as possible to put forward their experiments as Michael has done.  We all know that a new architecture is needed so let's explore the boundaries of what can be achieved.

Chris.

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send email to joomla-...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-cms.
For more options, visit https://groups.google.com/d/optout.

--

Chris Davenport
Joomla Production Leadership Team

[Fedir]

and one more thing in my head :)

better Schema management / migrations, for whole CMS: core + components,

and maybe some orm

/hide

[Nikolaos K. Dionysopoulos]

Fedir,

Schema management and migration is already included in Joomla! 3.4 in the form of FOFDatabaseInstaller. We were discussing with George a few weeks ago if we should merge that as a JDatabase feature in 3.5. AFAIK this is pending for discussion in the PLT. If that goes in then yes, it will be used by the core as well and ease the pain (I know exactly what you have in mind – inconsistent database schema and schema version in the #__schemas table; been there, got burned, wrote my own schema updater).

As for ORM, it means different things to different people. If you need real, full-featured ORM it’s feasible to integrate Doctrine with Joomla!. Herman Peeren has been doing that since 2011 if memory serves. If that’s what we want we can ask the man for some help. If you want to implement something like Laravel’s Eloquent which is basically Active Record plus some ORM-like features I have already done it (that’s my second, more successful attempt).

There’s code for everything you asked for so far, it’s GPLv2 and I’ll be happy to help Michael integrate it into Joomla! if that’s the consensus :)

Nicholas K. Dionysopoulos

Director, Akeeba Ltd

https://www.AkeebaBackup.com

--
You received this message because you are subscribed to a topic in the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/joomla-dev-cms/Zhkilq_uRNI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to joomla-dev-cm...@googlegroups.com.

[Chris Davenport]

Whilst having the option of migrating data at the database level can be useful, I think that a general approach to migration across (major) Joomla versions should be done at a higher level.  The reason being that migrating at the database level can bypass "business rules" that are supposed to guarantee the integrity of the data.  By using a hypermedia API approach to migration, where you pull data from the old API and post it to the new one, you are ensuring that the data always respects the rules and integrity constraints, even if the rules and constraints have themselves changed across versions.

Chris.

--

You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.

To post to this group, send email to joomla-...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-cms.
For more options, visit https://groups.google.com/d/optout.

[Mathew Lenning]

It is strange how we went from experimenting to integrating in Joomla so quickly. 

Wouldn't it be better to keep experiments out of 3.x for the time being? At least until there is a shared vision of Joomla next or 4.0 or whatever we want to call it.

Sincerely,

Mathew Lenning

Babel-university.com

P.S. This message was sent via iPhone, so please forgive any errors

[Troy]

Nicholas for dummys like me would you please explain what a "di containter" is?

Bear

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2015.0.5856 / Virus Database: 4315/9372 - Release Date: 03/24/15

[Nikolaos K. Dionysopoulos]

Hello Chris,

Data migration is not to be perceived as an exercise in academic cleanliness of code and methods but as a practical tool in real world conditions and servers. This means that any solution has to work on real world sites and (usually cheap, low quality, shared) hosts. Which brings me to the following problems I have run across over the years.

JSON data may not always be JSON data. Using a web services API supposes that a request with format=json in the URL or an Accept: application/json HTTP header will result in valid JSON data. This may not be possible in the following circumstances:

• 3PD plugins spitting out PHP Notices and Warnings on hosts where Error Reporting is set to anything other than None. This is, unfortunately, the norm.
• 3PD plugins idiotically assuming that all requests are HTML and injecting their own output before or after the component’s output, usually by hooking the onAfterRender event.
• 3PD plugins idiotically assuming that all requests are HTML, trying to inject media to the JDocument, causing a Fatal error.
• Several free hosts or proxies between the site and browser injecting hosts or other HTML data irrespective of the Content Type of the request

Mind you that I’ve been seeing that even in back-end requests which are usually less susceptible to the 3PD plugin issues. In the front-end the effects of plugins are far more pronounced. Even worse, the misgivings of the plugins will be transparent to the site owner considering that the vast majority or totality of the requests they are exposed to are of the HTML kind.

Maximum execution time and request, CPU usage, memory usage and I/O limits. The maximum execution time can be as low as 3 to 10 seconds and the memory limit can be as low as 24Mb (we see that on way too many live hosts, I’m afraid). This means that very small quantities of data can be processed at any one time. Therefore you need vast amounts of requests which will be very likely to cause the process to fail because of request limits or CPU usage limits. We do see several of these limitations in live hosts, e.g. not being able to hit the same or a similar or any URL from the same IP more than X times in Y seconds (typically 2 times in 1 second, can be as low as 2 times in 10 seconds). Even when there are no request limits there are usually CPU limits which kick in with multiple requests.

Firewalls. Using web services means that the target site has to do server-side HTTP requests to the source site. In most cases there is a server firewall which prevents that. More often than not trying to perform an HTTP request to the same server –what would be required in most migration scenarios– is forbidden. Some shared hosts don’t even allow a change on their firewall due to their setup.

Database query limits. Even if you don’t hit on any of these issues you are very likely to hit on database query limits. Under a web services data transformation methodology you’d have to perform several database queries per generated item. For example, transferring a single article requires half a dozen queries: create the article, create the asset record, check if a tag exists and create it if it doesn’t, create the UCM entry, create the tag to article entry. You can easily go way past the query limits using this method which can be easily demonstrated by using com_overload to create over a few thousand records.

Speaking of which performance limitations of JTableNested can hit time and CPU usage limits on moderately big sites. The time JTableNested requires to insert data is exponentially linked to the number of rows already present (O(N^2)). As a rule of thumb, over 10,000 rows in the assets table incur significant performance penalties which will end up killing the migration process because you’ll be hitting the PHP execution time, Apache maximum request time or sever's CPU usage limits.

As you understand all of the above are not baseless fear-mongering and wild speculation. These are real world issues which we’ve run into while developing Akeeba Backup (primarily) and our other software (secondarily). We have found workarounds for all of them but they require direct transferring of database tables in rather big chunks of multi-row INSERTs. That’s why I proposed using this methodology.

If we make the migration process solely dependent on web services I’m afraid that a lot of sites will be impossible to migrate, having an adverse and lasting effect on Joomla!’s reputation (people are still moaning about the migration from Joomla! 1.5). While I’d prefer the web service methodology as it’s cleaner, more flexible and architecturally correct I am more inclined to treat it as a practical issue in desperate need of a solution instead of an academic research subject. My experience with migration methods depending on similar web service methodologies in the past –even more lightweight than what I can infer you have in mind– has made me extremely wary of their possibility of success with real world sites running on real world servers.

If anything, it would be best to offer two migration routes. One with web services and one with direct database import and schema update. The first is cleaner, more flexible and architecturally correct but may fail on shared hosts whereas the latter is dirtier and less flexible but is known to work for the last nine or so years and could serve as the last resort solution. Just like we offer Joomla! updates through both com_joomlaupdate and installing the update as a package we should do the same for migrations, making sure that nobody gets left behind, blaming Joomla! for it.

Nicholas K. Dionysopoulos

Director, Akeeba Ltd

https://www.AkeebaBackup.com

[Nikolaos K. Dionysopoulos]

Hello Troy,

A "DI Container” is shorthand for Dependency Injection Container. To put it in simple terms –sacrificing technical correctness over clarity– you have an object which “carries” with it the knowledge to provide the appropriate “core" objects that other “worker” objects in the architecture require. Basically, instead of having everyone look up the same reference book it is possible to have specific objects use a custom reference book.

Think of it as a black box which can give you the correct reference to the application object, user object, input object, database object and so on. Unlike JFactory, you’re not forced to use the same application, user, input and database objects everywhere. A Controller could be passed a specially crafted Container which has a custom input object that’s different from the application’s input object. This means that this controller can act on a different data set than the one passed in the request. Therefore our components could start acting more like Drupal modules and have many of them placed on the same page. There are other more esoteric and very convenient uses, but they’re out of scope of your question. Let me just say that it’s no wonder that pretty much every major PHP framework uses a DI container.

For everyone else: please put your pitchforks away! I know that my explanation of a DI Container is very naïve and not technically correct. I am just trying to explain a hardcore development concept in a way a non-developer can (hopefully) grasp :)

Nicholas K. Dionysopoulos

Director, Akeeba Ltd

https://www.AkeebaBackup.com

You received this message because you are subscribed to a topic in the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/joomla-dev-cms/Zhkilq_uRNI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to joomla-dev-cm...@googlegroups.com.

[Justin Seliga]

There is without question a time and place for the business end of this conversation but Michael posited the idea, much more elegantly than I'm about to, that the Joomla! CMS is getting old and starting to show it's age. I don't think the premise behind the original post implied the idea that a rogue group would immediately embark on radical change. With that in mind, trying to wrap this conversation up into higher-level goals and characteristics that can be maintained in the next major release makes a lot of assumptions that pigeonhole the potential of this discussion. As I mentioned before, I would suggest every possible idea be thrown at the wall as if there isn't a single boundary. Let's not even confine this conversation to questioning what the current version lacks. No idea, no matter how extreme is bad. They might not be practical, but they could provide a perspective which sparks another "better" idea.

No one is denying how much application design has changed since 2008, but sometimes you become desensitized to the breadth of that gap when you're exposed to and learning the technology every day. The CMS in many ways is closed off from observing these trends, which can be seen in a code base that isn't ideally compatible with newer and arguably more "standardized" ways of approaching some of these problems. Looking at the landscape of the Joomla! marketplace I think many of the well-known Joomla! development organizations have been singing that loud and clear.

The most successful template providers for Joomla! have invested heavily in their own "frameworks" (which requires quotes as it has an array of meanings depending on the provider), and it's been happening for years. It's a very clear and real problem, especially for our end users, they shouldn't be forced to learn an entirely different administrative template interface just because they switched template providers. The marketplace has Gantry Joomla! templates, T3 Joomla! templates, etc. The CMS needs to provide an infrastructure that is conducive to simply just Joomla! templates without the qualifiers.

Standardizing output with Bootstrap 2 was one step in solving this epidemic, but that seemed to be more of a band-aid, than a remedy. Boil everything down, and the biggest goal of these "template frameworks" is about translating the logic of the template layer to site administrative users in an intuitive and efficient interface for management. In the end these are nothing more than a collection of form fields. Sure Joomla! provides a host of form field types that can easily be defined in a manifest file but they're extraordinarily rigid, and Joomla! makes a ton of assumptions of how the developer will use these. Joomla! making assumptions seems to be very common. As a result the lack of features (and the lack of documentation) ends in the community having n number of extensions, all who wrote a field type that essentially accomplishes the same goal.

HTML has a finite number of input types, each having a very specific set of possible data that could be available as options. Joomla! can and should provide extremely flexible form field types. Flexible to the point where it's more conducive for a developer to contribute back to the project than it is to layer yet another framework on top to achieve their goals. The only difference between a list of categories and a list of directories is the source of the data. Instead of writing two separate classes that extend JFormFieldList why not allow the manifest to specify a source for the data (database, file, json/xml string, etc) and depending on the source type allow specific filtering parameters. As a practical example, say we want a field that adds a fontawesome icon in front of menu items. A really easy way to get all of the available options and provide them to the site administrator, in a way that doesn't require adding new option entries in the manifest file everytime fontawesome adds a new icon is to simply read their CSS file for a specific string pattern and return it to an array. Couldn't JFormFieldList accept source type = 'file', location = '/path/to/file.css', pattern = 'expression' and preg_match giving the field an array of options? Instead of having a JFormFieldAccessLevel and JFormFieldLanguages wouldn't it be great if we could just give it a type = 'model', component = 'languages', name = 'languages', method = 'getList' or likewise for a list of access levels ? (I likely could be wrong, but the best I can tell is this type of flexibility doesn't exist.  At least not within documentation, and the fact that JFormFieldAccessLevels and JFormFieldLanguages exist implies just that).

All of this is just a really long way of saying, the CMS has had it's way of doing things for years, that is not to say it's bad. However, sometimes to move forward and embrace new methodologies it's necessary to avoid as many constructs as possible and approach the problem as if the only option is to write a CMS from the ground up. To me there is really only two certainties in the construct of this discussion, the Joomla! Framework and the current table structures, and they should be the only boundaries in this conversation. Can we really create goals if all the options aren't on the table first? I couldn't agree more (though sometime in the future after this discussion) that an objective set of criteria is absolutely necessary for real decisions to be made.

On Tuesday, March 24, 2015 at 6:26:39 AM UTC-4, Chris Davenport wrote:

Before embarking on any radical change I think it is instructive to try to get a handle on what we're trying to achieve.   What are the high-level goals?  What characteristics of the current version are considered essential to maintain and what are the desirable characteristics that we're looking for in a new version?  Why is the current version not delivering those characteristics?

This will make it easier to explore different options because we will have an objective set of criteria against which each proposal can be measured.

Probably should start a new thread for that though.

I would encourage as many people as possible to put forward their experiments as Michael has done.  We all know that a new architecture is needed so let's explore the boundaries of what can be achieved.

Chris.

[Paul Orwig]

I think in the early stages, and in a project where everyone is a volunteer and can put their time/effort into what they choose, there can be room for both discussions to take place at the same time:

• discussions and proof of concept work on underlying architecture
  
• discussions about high level goals/objectives

So I think the discussion in this thread is great, and I think it would also be great to start another discussion in another thread (as Chris suggested) about high level goals/objectives.

Thanks,

paul

 

--

[Justin Seliga]

My intention wasn't to disregard anyone's time or effort, and far be it for me to suggest that anyone else's idea is less important than my own. My apologies to Chris if it was perceived that way.  These types of discussions can be easily sidetracked or muddied with unnecessary obstacles. Only hoped to avoid that.

[Troy]

Me thinks I'm a dinosaur and needs to be set out to pasture :-[

Bear

[Michael Babker]

Consider this an extension of Icarus of sorts.  Actually, this as a standalone repo is much cleaner than using the existing repo.  Especially when the first commit would be "remove everything we've come to know" which would kill off any ability to view a code diff.  For concepts that plug into the existing infrastructure, such as your MVSC code, that repo is a great working hub.

[Michael Babker]

So for me, there are a few high level goals:

- An application structure that natively supports any web request format utilizing what we know today as JDocument as a base interface; native support for HTML, JSON, RSS, etc. without needing hacks or needless method_exists checks (in the end, you should still check $doc->getType() to make sure you have an expected format type and not just assume an HTML document, I would call this a best practice).

- An application structure that natively supports both command line and web operations without the need for creative tricks to get around hardcoded application dependencies.  Working with JUser from command line requires a JApplicationCms and JSession object registered to JFactory (the former being a web application class).

- Extensibility of the base API without the need to overload core classes or even hack the core API.  The existing JDocument::getInstance() method lacks some flexibility; all classes are assumed to be prefixed as JDocument and only the core libraries folder is scanned for class objects (a behavior in itself that should be deprecated in favor of just autoloading everything).  With my proposed DocumentFactory::getDocument() method one can inject a custom namespace into the options array and load a custom document object.

- Improved testability through less singleton object caches.  The tricks we have in the unit test suite to deal with JFactory references are rather interesting at times.

- Implementing the Framework code.  A lot of top level code is duplicated in a B/C breaking manner right now (because of class hierarchy it's pretty impossible to just replace JDatabase or JForm with the Framework counterparts).  By building anew using this code as the base foundation of the application, we can begin using the Framework code in more practical examples and use this experiment as a way to validate our existing Framework code.  What structures in the Framework need improvement and what structures work fine as they are?  So far the main API I've had to hack into from the Framework is the Form (JForm) class and this was only to provide injectable dependencies to form fields.

[Michael Babker]

In general, we are in desperate need of a proper schema management API.  It's a mild joke that any changes to SQL structure requires updating several SQL dumps and creating proper format specific queries for all supported platforms.  We should be at a point where our API should be able to manage this; updating single files that can be read and converted to the proper format would be a much more preferable solution.  Any code that moves us in that direction is hugely welcome by me.

--

You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.

[Michael Babker]

On Tue, Mar 24, 2015 at 1:55 PM, Justin Seliga <justin...@gmail.com> wrote:

The most successful template providers for Joomla! have invested heavily in their own "frameworks" (which requires quotes as it has an array of meanings depending on the provider), and it's been happening for years. It's a very clear and real problem, especially for our end users, they shouldn't be forced to learn an entirely different administrative template interface just because they switched template providers. The marketplace has Gantry Joomla! templates, T3 Joomla! templates, etc. The CMS needs to provide an infrastructure that is conducive to simply just Joomla! templates without the qualifiers.

It's great for their businesses but terrible for the ecosystem.  The heart of their frameworks should enable those companies to quickly develop and deploy templates, and the frameworks themselves usable for more experienced folks who are comfortable spinning up a customized template on their favorite framework.  From what I've seen, their platforms have grown to the point that they could practically be applications in and of themselves.  They have a lot of options crammed into the UI that some of us would expect to be handled in CSS or JS files, but in doing so they make it easier for users to customize their templates.  It does make templates more difficult to adapt when you have to check extension compatibility with them; in reality templates shouldn't cause this much headache for users.

Standardizing output with Bootstrap 2 was one step in solving this epidemic, but that seemed to be more of a band-aid, than a remedy. Boil everything down, and the biggest goal of these "template frameworks" is about translating the logic of the template layer to site administrative users in an intuitive and efficient interface for management. In the end these are nothing more than a collection of form fields. Sure Joomla! provides a host of form field types that can easily be defined in a manifest file but they're extraordinarily rigid, and Joomla! makes a ton of assumptions of how the developer will use these. Joomla! making assumptions seems to be very common. As a result the lack of features (and the lack of documentation) ends in the community having n number of extensions, all who wrote a field type that essentially accomplishes the same goal.

I envision a platform where the MVC code could care less what template or framework you're using.  The initial concepts in my structure remove layouts completely from within the MVC folder structure; the ONLY place to find a layout is a layout folder.  And when the tools are in place, the ONLY place to manipulate the appearance of that layout or the template it gets absorbed into will be within the layout or the template it's absorbed into.  I envision an API that deals strictly with data objects being injected into our view class and the view class only being responsible for taking the injected data object and converting it for output.  Core templates and layouts can be built on a common UI standard, but ultimately the entire application is no longer vendor locked to specific UI elements.

HTML has a finite number of input types, each having a very specific set of possible data that could be available as options. Joomla! can and should provide extremely flexible form field types. Flexible to the point where it's more conducive for a developer to contribute back to the project than it is to layer yet another framework on top to achieve their goals. The only difference between a list of categories and a list of directories is the source of the data. Instead of writing two separate classes that extend JFormList why not allow the manifest to specify a source for the data (database, file, json/xml string, etc) and depending on the source type allow specific filtering parameters. As a practical example, say we want a field that adds a fontawesome icon in front of menu items. A really easy way to get all of the available options and provide them to the site administrator, in a way that doesn't require adding new option entries in the manifest file everytime fontawesome adds a new icon is to simply read their CSS file for a specific string pattern and return it to an array. Couldn't JFormField accept source type = 'file', location = '/path/to/file.css', pattern = 'expression' and preg_match giving the field an array of options? Instead of having a JFormFieldCategories and JFormFieldLanguages wouldn't it be great if we could just give it a type = 'model', component = 'categories', method = 'getList' or likewise for a list of Languages? (I likely could be wrong, but the best I can tell is this type of flexibility doesn't exist.  At least not within documentation, and the fact that JFormCategories and JFormLanguages exist implies just that).

In truth, I don't think I've seen anyone make such a proposal.  It would definitely be interesting to see something like that, however I'd also caution against making the class too complex by adding all of these dynamic generator options into it.  I think there would need to be some balance here, but I don't see any reason why it couldn't support some basic features to do as you suggest.

[Chris Davenport]

Hi Nik,

Much of your criticism of the web services approach seems to be based on the assumption that the migrator would execute on the target host.  Whilst that is certainly possible, it needn't be required.  Provided the migrator has read access to the source and read/write access to the target, it can run anywhere.  Indeed, it's not inconceivable that a migration service could be setup to do migrations remotely.  Allowing the migrator to be rate-limited would also solve a lot of your expected issues with slow or inadequate hosts, I would suggest.

Database-level migration aims to help low quality sites at the expense of making life harder for the moderate-to-good sites, which I would guess is probably the majority.

I would have no objection in principle to offering both migration routes although using the hypermedia API would be the preferred route for the majority of sites.  As you say, it's cleaner, more flexible and architecturally correct.

Chris.

[ssnobben]

Great initiative Michael!

I think its good you show examples have creativity, energy, excitement starting up people and the community. Great!

I  also agree with Paul O, Chris and some of what Justin posted about planning and how to go forward.

This should be a planning process that also is documented how things should be proceed in the right order within the Joomla community so we know we are doing the right things in the right order for the right purpose(s). Overview and break it down....

I talked before long long time ago with Louise L about having a Joomla dev strategy that could be continuously updated/changed and describe the Joomla dev method but what come out from this finally was a kind of strategy dev doc. For me the vision, strategy and execution is good planning steps on every level of Joomla (OSM).  Drupal and Wordpress have another steering control dev process for good and bad bcs someone is responsible for the priority and direction.

Paul wrote
- discussions about high level goals/objectives

What, How, When, Who, Where, and Why?

One idea would be to also start thinking from higher level Joomla existence and then break this down to all other aspects that is needed, required down to the coding.

One other idea is that it would be good to split up the discussion later into manageable chunks that is mapped to how Joomla structure would be developed a f t e r an overview analyse, discussion is done and with accepted with consensus. Create visualize objectives and write down mock-ups, wiring diagrams that is shared and further developed into some core dev people that continuously develop them as project going forward. Set target, goals and plan execution and what is required split up and plan the project in small chunks of dev focus.

I also like the conept of Zackman framework thinking of why Joomla should exist and being the first thoughts of how to also break this down even to why certaint Joomla architecture should exist or not. So there is an answer to What, How, When, Who, Where, and Why to thinking, decision and forward going. Its normall for enterprises but can also be good to use for an organisation like Joomla community I guess. You may use part of and adopt for Joomla for planning, discuss set up goals and execute.

"...derived from the philosophical concept of reification, the transformation of an abstract idea into an instantiation."

http://en.wikipedia.org/wiki/Zachman_Framework

Well hope it will be good and have the right focus of users - how are they defined?, the market - what is Joomlas different market(s)? and all be fine to develop something that is needed and used in future digital world.

What, How, When, Who, Where, and Why?

Maybe Michael you can create a document that can be updated to set this dev process and share with others to discuss improve?

Cheers!

[Nikolaos K. Dionysopoulos]

Chris,

No. I do not suppose that the migration runs on the target server, that would be impossible because of server timeouts. I suppose that there is a controller app (more on that later) which communicates either a. with both the source and target server over HTTP or b. with the target server only over HTTP (and the target server communicates with the source server directly). The controller app could be a Javascript application running on your browser, a web service running on a third party server or a misguided human doing HTTP requests by hand. This doesn’t change the nature of any of the problems I have described.

Do keep in mind that I found out about the issues I mentioned when developing JoomlaPack and its successor Akeeba Backup. We were not calling it “web services” back in ’06 when I started this development, we were calling it “AJAX”, but nothing changes except the name and the internal data structure. In essence there were always two applications: a controller application and a workhorse PHP application running on the server. The controller application could be moved to any location be it Javascript running in the browser (typical back-end backup), a desktop OS (see the old Akeeba Remote Control), a remote service (myJoomla, Watchful, Joomla! Commander), a CLI CRON job (Akeeba Remote Control CLI) or a generic service such as WebCRON (see front-end backup mode). The problems I mentioned occur irrespective of the HTTP call source. They are internal to the server (Joomla!) portion of the methodology, NOT the controller part! Telling me that moving the controller part as a service would make any difference is just plain wrong. Not to mention that this precludes migrating sites on local servers which are not publicly visible on the Internet.

As for rate limiting, don’t think it’s a panacea. Yes, you DO need rate limiting but the point is how much you can do in its step. You’d be transferring one article every 10 seconds or so just to not hit the request limits. With 1000 articles that’s three and a half hours (my site has already reached double as much Joomla! articles and just think that I barely use the core content management). With 10,000 articles that’s over a day. And that’s just articles. Trying to sell the easy migration to users what we’d be telling them? “Oh, don’t worry, the data migration will only take two days and if you screw up you can start it all over again”. Eh, sorry, I don’t think that sounds good. Direct database migration would be much faster –in the order of a dozen minutes, not hours– and less flexible, but it doesn’t take a lifetime and a half to migrate your site.

IMHO web services are super cool but for a completely different use: synchronising content between sites. That’s the very frequently requested feature which nobody thought of tackling in the obvious way, using web services. This also does away with the other frequently asked feature, multisite, considering that it’s almost always mentioned in the context of a staging/live site pair. If you had content sync you could easily set up staging/live site pairs. But didn’t I just say that web services would be really bad for content transfer? It depends on the context. The content sync can be reasonably expected to be used by a small subset of our user base which has the knowledge, resources and properly set up sites and servers to make use of it. The site migration is supposed to be used by millions of people of every technological competence level (from incompetent to guru) who are currently using Joomla!. The former can have catches and asterisks requiring expert resolution. The latter must be a turnkey solution.

Nicholas K. Dionysopoulos

Director, Akeeba Ltd

https://www.AkeebaBackup.com