1.5 to 1.6 upgrade process

[Andrew Eddie]

In my physical and virtual travels, the question of, and concern
about, the process of upgrading (or worse, migrating) from 1.5 to 1.6.
Now, I am reminding people there is no urgency to upgrade because 1.5
is on a long-term-release plan, but I think we still need a solution.

Hitherto, Joomla has not had to an SQL upgrade for a *long* time - our
community is not really familiar with the process. Ideally we need
some sort of "thing" that takes the pain out of a 1.5 to 1.6 upgrade,
just as we did with the installation in the early Mambo days.

So, given this question is on the communities mind, it seemed like
this would be a nice, self-contained mini-project to start thinking
about what the process could look like and then what the technical
difficulties - and hopefully a couple of people can put their hand up
to run with it.

Thanks in advance.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

[dukeofgaming]

Hello Andrew,

As you might already know, there is a handful of people already
interested in giving this a solution over here
http://www.alltogetherasawhole.org/group/j15to16migration. We are
looking forward to use Doctrine's migration capabilities (http://
www.doctrine-project.org/documentation/manual/1_1/en/migrations) and
for that we have started looking into the database.

The first step, as suggested by Júlio, was to make the database
diagrams, which are sort of ready and available over here:
http://rapidshare.com/files/299365808/j16migration_diagrams.rar
(specific topic here: http://www.alltogetherasawhole.org/group/j15to16migration/forum/topics/database-diagram).

And well, through the process I found some details that might help
polish the db a little, and well, as I pointed out over there, I think
Joomla needs db naming conventions even though there seems to be
already some implicit ones (I couldn't find any over here
http://docs.joomla.org/Coding_style_and_standards). As suggested by
Amy, I'll quote those details over here. Hope they are of some actual
use and not just noise:

- Not all PKs are called id even though most are, e.g.
messages.message_id, banner.bid, bannerclient.cid
- Not all tables have the PK in the first place, e.g. newsfeeds
- Some "FKs" are abbreviated, some are not, e.g.
access_rules.section_id is nice, while banner.cid is not
- I "think" not all data types are optimized (though most are), e.g.
components.enabled is TINYINT(4)
- There is a sections table?
- And last but not least, no db level referential integrity (even
though MyISAM just supports the notation and not the behaviors that
could be nice enough... and perhaps future support for InnoDB?, it has
side benefits! [such as nulls actually not using storage space])

I personally think that database conventions are absolutely non-
trivial and actually very necessary to if we ever look forward to
having an ORM in Joomla and/or composable extensions.

Regards,

David

> Andrew Eddiehttp://www.theartofjoomla.com- the art of becoming a Joomla developer

[dukeofgaming]

[BTW, I find CakePHP's database conventions rather good and easy to
predict/memorize: http://book.cakephp.org/view/24/Model-and-Database-Conventions]

On 30 Oct, 05:03, dukeofgaming <dukeofgam...@gmail.com> wrote:
> Hello Andrew,
>
> As you might already know, there is a handful of people already

> interested in giving this a solution over herehttp://www.alltogetherasawhole.org/group/j15to16migration. We are
> looking forward to use Doctrine's migration capabilities (http://www.doctrine-project.org/documentation/manual/1_1/en/migrations) and

> for that we have started looking into the database.
>
> The first step, as suggested by Júlio, was to make the database
> diagrams, which are sort of ready and available over here:http://rapidshare.com/files/299365808/j16migration_diagrams.rar

> (specific topic here:http://www.alltogetherasawhole.org/group/j15to16migration/forum/topic...).

>
> And well, through the process I found some details that might help
> polish the db a little, and well, as I pointed out over there, I think
> Joomla needs db naming conventions even though there seems to be

> already some implicit ones (I couldn't find any over herehttp://docs.joomla.org/Coding_style_and_standards). As suggested by

> > Andrew Eddiehttp://www.theartofjoomla.com-the art of becoming a Joomla developer

[Sam Moffatt]

Hi David,

I have the distinct feeling you're trying to over complicate something
in the most part. The majority of the changes are minor with the
exception of ACL which will require some significant attention. The
nested categories is pretty simple since we really only have a two
level structure at the moment, some shuffling might be required
temporarily and a tree rebuild once its done to get the thing working
properly. But if it works, at the end of the day I guess it works.

> And well, through the process I found some details that might help
> polish the db a little, and well, as I pointed out over there, I think
> Joomla needs db naming conventions even though there seems to be
> already some implicit ones (I couldn't find any over here
> http://docs.joomla.org/Coding_style_and_standards). As suggested by
> Amy, I'll quote those details over here. Hope they are of some actual
> use and not just noise:
>
> - Not all PKs are called id even though most are, e.g.
> messages.message_id, banner.bid, bannerclient.cid

My own belief of calling everything some ubiquitous "id" field ends up
hurting you later when you end up starting to do table joins -
especially when they start to become complicated. When you start to
look at composite keys you end up wanting to use more than just 'id'
as part of the key anyway, so I personally find that it is easier to
just do it by default. This also makes determining what the foreign
key field name should be easy: the same as the primary key of the
table you're linking to.

> - Not all tables have the PK in the first place, e.g. newsfeeds

It took me a while to work out what you mean here, perhaps lost in
translation somewhere. You mean that the first column of the table
isn't the primary key? I'd suggest this is a legacy thing.

> - Some "FKs" are abbreviated, some are not, e.g.
> access_rules.section_id is nice, while banner.cid is not

Most new ones should be in a longer form, banner.cid is an ancient field.

> - I "think" not all data types are optimized (though most are), e.g.
> components.enabled is TINYINT(4)

What do you suggest it should be instead?

> - There is a sections table?

What is this in reference to? There might be a legacy FK to it in the
content table and I'm sure that will remain at least for this release.
It will be useful for at least upgrading a 1.5 site to 1.6 in the
early stages, after that point it might not be useful and I'm sure we
can then mark it as deprecated and schedule its removal from the
schema.

> - And last but not least, no db level referential integrity (even
> though MyISAM just supports the notation and not the behaviors that
> could be nice enough... and perhaps future support for InnoDB?, it has
> side benefits! [such as nulls actually not using storage space])

It probably hasn't been done because it really doesn't mean anything.
Almost every table create last time I checked set the engine to MyISAM
anyway. This is perhaps more of a legacy habit that hasn't been
followed because it has no major benefit overall at this point.

If you're going through all of the tables it would be great if you
could document stuff here:
http://docs.joomla.org/Tables

That way we start to get stuff up there at least the base line and
work towards writing all of this knowledge down and have a place we
can record changes against as well until we work out the ORM path.

A lot of items are broken because they've been broken for a while and
unbreaking things causes compatibility issues. Even small changes,
such as a field rename, can trip people up

Sam Moffatt
http://pasamio.id.au

[Andrew Eddie]

Hi David

Look, I don't want to stifle innovation and trying new things, but the
upgrade process, to me, is pretty simple.

1. We need to make sure that the diff sql is correct and includes all
changes (I think it's missing a few things).
2. We try and incorporate all SQL upgrades that we possibly can in SQL
land (for example, merging the sections and categories tables).
3. Assuming the diff runs successfully, mop up any schema and
conversion changes in PHP land (eg, you are going to have to do a
table rebuild on all nested sets).
4. Remove deprecated files (which probably needs a manifest).

We just need something simple roughly in line with the installation
application. This is for the user so it needs to be simple and fool
proof. Anatomically correct is not a priority at the moment.
Probably need to think about an option to backup and rollback as well.

I suggest you look at the workflow (how does a user get it, install
it, use it, finish) and from there get a prototype working (by the end
of November would be cool if possible).

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer




2009/10/30 dukeofgaming <dukeof...@gmail.com>:

[dukeofgaming]

Hi Andrew, its ok, I know this is about the migration script but just wanted to point out that Joomla has no database conventions and some of the cases such conventions might get straight. Is there actual interest for database conventions?. I was preparing a reply to Sam, but I'll leave that to the event we actually discuss them.

The only thing that I still think is worth talking about are those foreign keys. The idea here (http://www.alltogetherasawhole.org/group/j15to16migration) is to use Doctrine as a tool to help us automate some part of the process and get that SQL generated easier, since Doctrine is a relational tool it needs those relations to exist to make the most of it (autogeneration of record and migration classes) and for us to work in specialized cases (object navigability instead of queries).

So here is the question: since putting them does not affect MyISAM behavior and would be a step towards better formalization of the database and a step towards an ORM too... do you think it would be a problem to leave them in the final database?.

Regards,

David

[Hannes Papenberg]

Hello folks,
as I hinted earlier, I'd like us to have an upgrader component. That
component is not really complicated. As Sam already pointed out, it just
has to do some SQL queries and copy some files around. The big issue
however is, that it will run into the PHP execution time limit. And not
just once. That is why that component will need to use quite a lot of
ajax to periodically trigger these different steps.

This is how I envision the upgrade process:
1. The user downloads the upgrade package from joomlacode and directly
installs it via the component installer onto his system. Depending on
the size of the component package, we might have to install the upgrader
component and in there have upload fields for smaller packages with the
actual files for the upgrade in there. At the end of this step, we
assume that all files for the upgrade are uploaded in archives to that
server.
2. The component checks the checksum of the core files against a
checksum file. Depending on the file, changes in it trigger different
kinds of alarms. For example changes to the templates will only be a
notice, changes to the /libraries folder will be a big warning. We also
check if the folders and files are writable. At this point we are asking
if the user wants to go on or not. If possible, we should be able to
make a backup of all files and the DB. This would be the icing on the cake.
3. Now we start with the actual modifications to the system by switching
the site offline and replacing the frontend components with the new
files from the upgrade_components_frontend.tar archive. In the next ajax
triggered page call we replace the files in the /images, /modules,
/languages, /media, /templates and /includes folder in the frontend.
From this point onwards we need to use mootools 1.2, so we have to
switch the JS used in the component. These steps should be possible in
one page call, although this is already pretty much for the set limit.
Notice that we haven't changed the plugins folder yet.
4. Now we switch over the backend components and in the next page call,
we switch the rest of the backend folders.
5. At this stage we start with database changes and change all
non-framework tables to the new system, introduce the #__extensions
table, etc. This also means that we create the ACL tables and fill them up.
6. Now we change all files in the /libraries folder that we wont call
when reloading the upgrade component and do the same in another pagecall
for the /plugins folder.
7. This step is critical, since it can break the whole upgrade process.
At this point, we change the rest of the files, e.g. in /libraries,
/administrator/includes, /plugins and the rest of the tables in the
database. Now the upgrade is basically done.
8. In this last step, we do some clean up, remove the upgrader component
and some last stray files.

This is very ambitious, but for the enduser, it means an only four step
upgrade process: Install the upgrader, upload the file packages in the
component itself, check writability and that no files were changed, kick
of upgrade process. It will not solve frontend problems with "old"
templates and general problems with outdated plugins (which would have
to be disabled all by default during the upgrade process)

If some people could start work on this, it would be really awesome.

Hannes

Andrew Eddie schrieb:

[Andrew Eddie]

David, what I'd do is start another thread about schema refactoring
and chip away at it one field (figuratively) at a time.

I've been doing a bit with each component I refactor but more work is
needed, there is no question of that. We are talking code style on
the framework list so I've no issue with bringing up schema here while
we've still got time. I'll start a new thread for that.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

2009/10/31 dukeofgaming <dukeof...@gmail.com>:

[Andrew Eddie]

Thanks Hannes

I think a component would be ok (I'd actually been thinking this would
be separate, but that's ok). So it will do all the preliminary checks
and balances and possibly compute the delta for the files. I think
you'll then have to break out of Joomla into a small upgrader
application (part of the component package but like /installation/)
otherwise you'll have files and schema changing under you. Does that
sound right?

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

2009/11/1 Hannes Papenberg <hack...@googlemail.com>:

[Hannes Papenberg]

I know that files and schema are going to change under you, as you say.
I think that we would be able to create a component that still stays in
Joomla, but changes all these files from within. It is possible with out
framework and it is technically possible. However, at one point we have
to make a bunch of changes that need to be done in one page call and I'm
not sure if those will finish on all servers. It's something that I
think we just have to test....

Hannes

Andrew Eddie schrieb:

[Andrew Eddie]

Sounds fair.

David/Julio, if you want a branch in the SVN to start playing, just
say the word. Sam will set you up :)

[dukeofgaming]

Sure, one thing comes to my attention though; Doctrine is LGPL, and even while this might just be a temporary tool, would this collide with GPL and/or cause any trouble by having it on the repository?.

Thanks,

David

[Andrew Eddie]

LGPL is fine for that purpose.

Can you give us the executive summary of what Doctrine is?

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

2009/11/2 dukeofgaming <dukeof...@gmail.com>:

[dukeofgaming]

Doctrine is a very robust object-relational mapper that allows for automatic model generation (http://www.doctrine-project.org/documentation/manual/1_1/en/introduction-to-models#generating-models:existing-databases:making-the-first-import) and even automatic migrations (http://www.doctrine-project.org/documentation/manual/1_1/en/migrations#writing-migration-classes:generating-migrations).

[For those not familiar with an ORM, in a nutshell, it allows to work with tuples as objects that can be saved (insert or update) and deleted with a method, and in addition enables object level navigability, e.g. 

        $content->User->Session->time 

The previous line gets us the time from the session of the user that created some content; we just jumped through 3 tables with that little effort]

The idea here, me thinks, would be that once we get the migration right with Doctrine, we can use Doctrine's query profiling for the whole process and use the output SQL to make the migration component.

[Hannes Papenberg]

I'm not sure that I see the benefit in this tool at this stage. In most
cases, the database migration will be either a very simple query (UPDATE
#__content SET access = access + 1) or it will need quite some
processing as with the ACL management. I doubt that it will be possible
to have this done automatically by this tool...

Let me give you one example:
In 1.5 the templates store their parameters in a params.ini. In 1.6
migration, we need to get the data from the #__templates_menu table,
load it into memory, get the params.ini files, too, re-encode them to
JSON, rearrange the data from #__templates_menu, alter the #__menu table
and the #__templates_menu table and write all that data back.

Would it be possible to do this with doctrine?

Hannes

dukeofgaming schrieb:

> Doctrine is a very robust object-relational mapper that allows for
> automatic model generation
> (http://www.doctrine-project.org/documentation/manual/1_1/en/introduction-to-models#generating-models:existing-databases:making-the-first-import)
> and even automatic migrations
> (http://www.doctrine-project.org/documentation/manual/1_1/en/migrations#writing-migration-classes:generating-migrations).
>
>
> [For those not familiar with an ORM, in a nutshell, it allows to work
> with tuples as objects that can be saved (insert or update) and
> deleted with a method, and in addition enables object level
> navigability, e.g.
>
> $content->User->Session->time
>
> The previous line gets us the time from the session of the user that
> created some content; we just jumped through 3 tables with that little
> effort]
>
> The idea here, me thinks, would be that once we get the migration
> right with Doctrine, we can use Doctrine's query profiling for the
> whole process and use the output SQL to make the migration component.
>
> On Sun, Nov 1, 2009 at 6:16 PM, Andrew Eddie <mamb...@gmail.com
> <mailto:mamb...@gmail.com>> wrote:
>
>
> LGPL is fine for that purpose.
>
> Can you give us the executive summary of what Doctrine is?
>
> Regards,
> Andrew Eddie
> http://www.theartofjoomla.com - the art of becoming a Joomla developer
>
>
>
>
> 2009/11/2 dukeofgaming <dukeof...@gmail.com

> <mailto:dukeof...@gmail.com>>:

> > Sure, one thing comes to my attention though; Doctrine is LGPL,
> and even
> > while this might just be a temporary tool, would this collide
> with GPL
> > and/or cause any trouble by having it on the repository?.
> > Thanks,
> > David
> >
> > On Sun, Nov 1, 2009 at 3:27 PM, Andrew Eddie
> <mamb...@gmail.com <mailto:mamb...@gmail.com>> wrote:
> >>
> >> Sounds fair.
> >>
> >> David/Julio, if you want a branch in the SVN to start playing, just
> >> say the word. Sam will set you up :)
> >>
> >> Regards,
> >> Andrew Eddie
> >> http://www.theartofjoomla.com - the art of becoming a Joomla
> developer
> >>
> >>
> >>
> >>
> >> 2009/11/1 Hannes Papenberg <hack...@googlemail.com

> <mailto:hack...@googlemail.com>>:

> <mailto:hack...@googlemail.com>>:

[Andrew Eddie]

Thanks David.

I'll begin by saying Doctrine looks very cool. However, my fear is
that it will blow the scope/scale out too much (but if it proves
technologies to help me as a 3PD do DB upgrades between component
versions, that would be a significant ++).

My "wish" would be to have a prototype working by the end of November.
Is that feasible (regardless of the SQL upgrade method)? I don't
even mind if 2 or 3 teams trial different methods and we pick the
"better" one to run with. The main thing is it has to be finish-able.
My hope is to completely finish with backend tidy-ups by the end of
Nov myself and also have the frontend to a point where the HTML/CSS
people can argue amongst themselves. Would be good to be able to
fine-tune a lot of stuff between sessions at the Dev conference in Dec

:)

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

2009/11/2 dukeofgaming <dukeof...@gmail.com>:

[Sam Moffatt]

If people need access to branches, mail me Joomlacode usernames (this
address is fine), who you are and why and I'll go from there.

Sam Moffatt
http://pasamio.id.au

[dukeofgaming]

Hannes, take a look here: http://www.doctrine-project.org/documentation/manual/1_1/en/utilities#facade; I don't see a problem doing what you say even if we were to make complex stuff such as that in one big encapsulated step with the aid of Doctrine, either how you describe or using temporal tables.

Andrew, take a look here: http://www.doctrine-project.org/documentation/manual/1_1/en/migrations#performing-migrations, I think it is exactly what you just described for the significant ++, besides, everything with Doctrine gets loaded by demand, so no actual overhead there.

BTW, the only automatic step would be the diffing, other more specialized stuff such as what Hannes just described we would do working with Doctrine Records and Collections, and to have object navigability we need to specify foreign keys, that way we can save some queries.

About the dates, I'm currently under a big load of work (even if it doesn't look like it =P), so not really sure I can commit to a lot, but as far as helping with what I know about Doctrine (including its integration with Joomla) and some ideas regarding the component, you can count on me =).

[Andrew Eddie]

> Andrew, take a look
> here: http://www.doctrine-project.org/documentation/manual/1_1/en/migrations#performing-migrations,
> I think it is exactly what you just described for the significant ++,
> besides, everything with Doctrine gets loaded by demand, so no actual
> overhead there.

That looks pretty darn cool.

> About the dates, I'm currently under a big load of work (even if it doesn't
> look like it =P), so not really sure I can commit to a lot, but as far as
> helping with what I know about Doctrine (including its integration with
> Joomla) and some ideas regarding the component, you can count on me =).

Basically we just need a couple of keen devs to own this and get it
done (following code standards of course, hehe). I'll around to
provide guidance as well but I'd like some others to drive this.

[Matias Aguirre]

Hi all,

Im interested to help in this component to make the migration. Just
tell me how i can to start and how i can to upload an example/dev
component then you can guideme/critical this project.

What is the better way to do this? What about the 3rd party
extensions? all of them is compatible with Joomla 1.6?

I see the the j16diagram and in the j16 diagram appears the
jos_sections table. This is fine? i cannot found that table in the
lastest svn.

Sorry by my english.
Take care

On Nov 1, 10:32 pm, Andrew Eddie <mambob...@gmail.com> wrote:
> > Andrew, take a look

> > here: http://www.doctrine-project.org/documentation/manual/1_1/en/migration...,

> > I think it is exactly what you just described for the significant ++,
> > besides, everything with Doctrine gets loaded by demand, so no actual
> > overhead there.
>
> That looks pretty darn cool.
>
> > About the dates, I'm currently under a big load of work (even if it doesn't
> > look like it =P), so not really sure I can commit to a lot, but as far as
> > helping with what I know about Doctrine (including its integration with
> > Joomla) and some ideas regarding the component, you can count on me =).
>
> Basically we just need a couple of keen devs to own this and get it
> done (following code standards of course, hehe).  I'll around to
> provide guidance as well but I'd like some others to drive this.
>
> Regards,

> Andrew Eddiehttp://www.theartofjoomla.com- the art of becoming a Joomla developer

>
>
>
> > On Sun, Nov 1, 2009 at 6:48 PM, Andrew Eddie <mambob...@gmail.com> wrote:
>
> >> Thanks David.
>
> >> I'll begin by saying Doctrine looks very cool.  However, my fear is
> >> that it will blow the scope/scale out too much (but if it proves
> >> technologies to help me as a 3PD do DB upgrades between component
> >> versions, that would be a significant ++).
>
> >> My "wish" would be to have a prototype working by the end of November.
> >>  Is that feasible (regardless of the SQL upgrade method)?  I don't
> >> even mind if 2 or 3 teams trial different methods and we pick the
> >> "better" one to run with.  The main thing is it has to be finish-able.
> >>  My hope is to completely finish with backend tidy-ups by the end of
> >> Nov myself and also have the frontend to a point where the HTML/CSS
> >> people can argue amongst themselves.  Would be good to be able to
> >> fine-tune a lot of stuff between sessions at the Dev conference in Dec
> >> :)
>
> >> Regards,
> >> Andrew Eddie

> >>http://www.theartofjoomla.com- the art of becoming a Joomla developer
>
> >> 2009/11/2 dukeofgaming <dukeofgam...@gmail.com>:

> >> > Doctrine is a very robust object-relational mapper that allows for
> >> > automatic
> >> > model generation
>

> >> > (http://www.doctrine-project.org/documentation/manual/1_1/en/introduct...)
> >> > and even automatic migrations
>
> >> > (http://www.doctrine-project.org/documentation/manual/1_1/en/migration...).

> >> > [For those not familiar with an ORM, in a nutshell, it allows to work
> >> > with
> >> > tuples as objects that can be saved (insert or update) and deleted with
> >> > a
> >> > method, and in addition enables object level navigability, e.g.
> >> >         $content->User->Session->time
> >> > The previous line gets us the time from the session of the user that
> >> > created
> >> > some content; we just jumped through 3 tables with that little effort]
>
> >> > The idea here, me thinks, would be that once we get the migration right
> >> > with
> >> > Doctrine, we can use Doctrine's query profiling for the whole process
> >> > and
> >> > use the output SQL to make the migration component.
>

> >> > On Sun, Nov 1, 2009 at 6:16 PM, Andrew Eddie <mambob...@gmail.com>

> >> > wrote:
>
> >> >> LGPL is fine for that purpose.
>
> >> >> Can you give us the executive summary of what Doctrine is?
>
> >> >> Regards,
> >> >> Andrew Eddie

> >> >>http://www.theartofjoomla.com- the art of becoming a Joomla developer
>
> >> >> 2009/11/2 dukeofgaming <dukeofgam...@gmail.com>:

> >> >> > Sure, one thing comes to my attention though; Doctrine is LGPL, and
> >> >> > even
> >> >> > while this might just be a temporary tool, would this collide with
> >> >> > GPL
> >> >> > and/or cause any trouble by having it on the repository?.
> >> >> > Thanks,
> >> >> > David
>

> >> >> > On Sun, Nov 1, 2009 at 3:27 PM, Andrew Eddie <mambob...@gmail.com>

> >> >> > wrote:
>
> >> >> >> Sounds fair.
>
> >> >> >> David/Julio, if you want a branch in the SVN to start playing, just
> >> >> >> say the word.  Sam will set you up :)
>
> >> >> >> Regards,
> >> >> >> Andrew Eddie

> >> >> >>http://www.theartofjoomla.com- the art of becoming a Joomla
> >> >> >> developer
>
> >> >> >> 2009/11/1 Hannes Papenberg <hackwa...@googlemail.com>:

> >> >> >> >>http://www.theartofjoomla.com- the art of becoming a Joomla
> >> >> >> >> developer
>
> >> >> >> >> 2009/11/1 Hannes Papenberg <hackwa...@googlemail.com>:

> ...
>
> read more »

[dukeofgaming]

Hi Matias, yes I was wondering about that, I got the script from the Alpha 2 install and reverse engineered it, I'll update the diagram some time tomorrow hopefully.

[Matias Aguirre]

HI again,

Please check if this (http://matware.com.ar/j16-svn-diagram.mwb)
diagram is ok. I think is a good idea to separate the database with
groups to make the job more easy and do the migration with 'stages'
and prevent timeouts.

take care

> > Andrew Eddiehttp://www.theartofjoomla.com-the art of becoming a Joomla developer

>
> > > On Sun, Nov 1, 2009 at 6:48 PM, Andrew Eddie <mambob...@gmail.com> wrote:
>
> > >> Thanks David.
>
> > >> I'll begin by saying Doctrine looks very cool.  However, my fear is
> > >> that it will blow the scope/scale out too much (but if it proves
> > >> technologies to help me as a 3PD do DB upgrades between component
> > >> versions, that would be a significant ++).
>
> > >> My "wish" would be to have a prototype working by the end of November.
> > >>  Is that feasible (regardless of the SQL upgrade method)?  I don't
> > >> even mind if 2 or 3 teams trial different methods and we pick the
> > >> "better" one to run with.  The main thing is it has to be finish-able.
> > >>  My hope is to completely finish with backend tidy-ups by the end of
> > >> Nov myself and also have the frontend to a point where the HTML/CSS
> > >> people can argue amongst themselves.  Would be good to be able to
> > >> fine-tune a lot of stuff between sessions at the Dev conference in Dec
> > >> :)
>
> > >> Regards,
> > >> Andrew Eddie

> > >>http://www.theartofjoomla.com-the art of becoming a Joomla developer

> > >> >>http://www.theartofjoomla.com-the art of becoming a Joomla developer

>
> > >> >> 2009/11/2 dukeofgaming <dukeofgam...@gmail.com>:
> > >> >> > Sure, one thing comes to my attention though; Doctrine is LGPL, and
> > >> >> > even
> > >> >> > while this might just be a temporary tool, would this collide with
> > >> >> > GPL
> > >> >> > and/or cause any trouble by having it on the repository?.
> > >> >> > Thanks,
> > >> >> > David
>
> > >> >> > On Sun, Nov 1, 2009 at 3:27 PM, Andrew Eddie <mambob...@gmail.com>
> > >> >> > wrote:
>
> > >> >> >> Sounds fair.
>
> > >> >> >> David/Julio, if you want a branch in the SVN to start playing, just
> > >> >> >> say the word.  Sam will set you up :)
>
> > >> >> >> Regards,
> > >> >> >> Andrew Eddie

> > >> >> >>http://www.theartofjoomla.com-the art of becoming a Joomla

> > >> >> >> >>http://www.theartofjoomla.com-the art of becoming a Joomla

> ...
>
> read more »

[Compass]

I thought I might just add a small comment in the discussion...

In my experience of migrating a bunch of sites, and helping Joomla
users do the same, it seems like most of the pain involved boiled down
to:

1. URL's changing (i.e. killing your site in google)
2. 3rd party extensions.

I think the things discussed above, are very valuable, but not the
major things that the average end user is dealing with.

I'd also through in there that Louis' 404 reddrect feature might be
the "killer app" for migrations!

Regards,

Barrie North
www.compassdesigns.net
www.joomlashack.com

[Marco Giorgetti]

I'm strongly agree with Barrie.

--
Marco Giorgetti
http://www.marcogiorgetti.com

[dukeofgaming]

Some wild idea for 3PDs: suppose we document "perfectly" the migration and we provide the means and standards for 3PDs to develop a migration plugin for this component. Now suppose we ask all 3PDs to register their plugin's URL along with the compatible versions for migration. Now suppose the migration component comes with a manifest containing the previous, so it can check the extensions you have agains the ones you can migrate and then downloads the plugins from the URLs given, then it executes each downloaded plugin to migrate its extension.

Now suppose it actually works, lol.

[Andrew Eddie]

I suppose that's a great idea :)

Barrie, we can certainly do a best attempt to pre-register redirects
that we know about in the core - that can be part of the process. For
the 3PD's, send out the word that now is the time to help.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer




2009/11/3 dukeofgaming <dukeof...@gmail.com>:

[Matias Aguirre]

Why the Joomla 1.6 database have a table like #__components and the
replacement #__extensions with the same data? or #__components is
still used?

Someone have documentation or link of how to works the #__assets
table?

Thanks

[Andrew Eddie]

Hi Matias

2009/11/3 Matias Aguirre <fast...@gmail.com>:

>
> Why the Joomla 1.6 database have a table like #__components and the
> replacement #__extensions with the same data? or #__components is
> still used?

#__components is used to in the backend menu to store additional
internal links/shortcuts for the component. The extensions table is a
master list of all installed extensions.

> Someone have documentation or link of how to works the #__assets
> table?

We don't have that documented yet, sorry.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

[mic]

As always (and this happend most of the times in the past): 3PD
developers should do their 'homework'.
As seen to many times, the development of an extension is made, but if
a new Joomla version comes out, nothing else.

A good sample was the move from j.1.0.x to j.1.5.x: how many
developers did provide an ETL to the migrator (I have seen not so
many ...)?

Same here: if an 'upgrade component' would be published, those
developers should provide their informations for upgrading.
But (and this is addressed to the core developers): an absolutely
'must' is the correct - and early enough - published 'howto/changes'
of what is todo or what I (as 3PD developer) have to look for!

[Andrew Eddie]

I think there are carrots you can hold out to encourage developers
(like "1.6 ready" stickers on the JED) but we can worry about that
when we see some progress on the actually system.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer




2009/11/4 mic <michael...@gmail.com>:

[Matias Aguirre]

Hi all,

Im working on a new version of mtwmigrator that will have support for
Joomla 1.6. I think that this is the better/fastest way to knows the
logic of the migration/update.

The methods discussed in this topic says that the component needs to
migrate into the current production site, if i understand. I think
that the better way to do the migration is to make a component that
install a new fresh Joomla 1.6 into a temporary directory and do all
the migration in that directory. Then if everything is ok and all
works, the user can to rename/move the root directory and use the new
version.

What can I do is work to advance mtwmigrator some work, pending a
decision as it should be the official migrator.

Here an alpha version of mtwmigrator with Joomla 1.6 support
(migration not yet): http://matware.com.ar/downloads/mtwmigrator/

Then when i have the logic of migration i will post here the results.

Take care

On Nov 2, 12:09 pm, dukeofgaming <dukeofgam...@gmail.com> wrote:

[Andrew Eddie]

Just following up.

Anyone given this (Joomla 1.5 to 1.6 upgrade process) any traction?

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer

[Matias Aguirre]

Andrew,

Im trying to do the migration progress. Users and groups is done. Now
im working with categories and i have some doubts, if you or any can
to guide me is apreciate.

My problem is with the #__assets table. When Joomla adds a new
category, it add a new row in this table. I need to know what is the
rule of the `name` field. To make this more understandable:

+----+------------------------+----------------------------+
| id | name | title |
+----+------------------------+----------------------------+
| 29 | .category.3 | Original Category Test 1:1 |
| 28 | com_content.category.2 | Original Category Test 1 |
+----+------------------------+----------------------------+

When i add a new category with root parent, it add name with
com_content, and if i add a category with parent of the last cat, it
not add a com_content. Its a bug of Joomla or i need to follow this
rule?

My plan to do this is to migrate all the section with ROOT parent and
the categories following the sections parent.

Take care

On Nov 16, 4:54 am, Andrew Eddie <mambob...@gmail.com> wrote:
> Just following up.
>
> Anyone given this (Joomla 1.5 to 1.6 upgrade process) any traction?
>
> Regards,

> Andrew Eddiehttp://www.theartofjoomla.com- the art of becoming a Joomla developer

[Matias Aguirre]

sorry but I forget to mention that im using the latest svn revision
13514.

any other help with "level", "lft" and "rgt" fields will be helpful.

Thanks

> > Andrew Eddiehttp://www.theartofjoomla.com-the art of becoming a Joomla developer

[Wilco Jansen]

Matias, I wrote a piece about level, lft and rgt here --> http://jfoobar.org/blog/324-hierarchical-data-nodes-and-nested-sets-in-16.html

Links also contain the deeper information on the theory behind this all.

Regards, Wilco

[Andrew Eddie]

Hi Matias.

Thanks so much for having a go at this.

Regarding the name convention, it follows this hierarchy:

root.1 (reserved for the root node of the tree)
- com_something (component)
- - com_something.category.{id} (category)
- - - com_something.{contentType}.{id} (content)

The root node is obviously reserved for the system, and this node hold
all the permissions that you see in Global Configuration.

The next level down (parent to root) is for the component and the name
is simply the name of the component folder. This node holds the
permissions that you see in the "Options" popup.

Within the component you start to add nodes for categories and,
optionally, content. The naming, for example using Newsfeed, is:

com_newsfeed.category.123

Where 123 is the id of the category. This node parents to the
component, that is and in this case, the node with a name of
"com_newsfeeds".

The fourth level is for actual content and the only place this occurs
is in com_content articles. The name for these are in the form:

com_content.article.123

Where 123 is the id of the article. This parents to the containing
category. I guess uncategorised articles will parent directly to the
com_content node or possibly com_content.category.0 (I haven't really
addressed this yet in the core so I'll have to let you know what to do
with them).

So, for com_content, you should get a tree happening like:

root.1
- com_content
- - com_content.category.1
- - - com_content.article.1

Components like com_weblinks, com_newsfeeds, etc that support
categories do not go to that fourth level. They stop at categories.

Where there are nested categories, it will be something like:

root.1
- com_content
- - com_content.category.1
- - - com_content.category.2
- - - - com_content.article.1

So your procedure for the assets table is going to be something like:

1. The upgrade SQL fires to create jos_assets.
2. Root node and core components are added manually via SQL with
default permissions.
3. Scan the components table for non-core components and add them.
4. Add all the sections as the first level of categories under the
com_content node.
5. Add all the com_content categories under the nodes that where just
built from the sections.
6. Add all the article assets under their respective category.
7. Add all the categories assets in from all the other components.

You don't have to worry about level, lft and rgt as you are building
the data. Once you are finished, all you need to do is run the
rebuild method on the asset table, eg:

$table = JTable::getInstance('asset');
$table->rebuild();

That will work it all out for you.

Does that help? If you have any other questions, don't be afraid to ask.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer




2009/11/17 Matias Aguirre <fast...@gmail.com>:

[Matias Aguirre]

Thanks you very much for the help.

Here you can test the users and categories migration:
http://matware.com.ar/downloads/mtwmigrator/ (0.3.1alpha version)

I can to create a new separate component for the migration to joomla
1.6 if you like. Just tell me what steps i need to follow to do this.

All of the code that do the migration is in the models/j15-j16.php
file. You are free to use the functions. Its not a beautiful code, but
we can to make it better.

Tomorrow i will to do the content migration and the other stuff.

Thanks again for the help and info and i hope that this code helps.

Regards

> Andrew Eddiehttp://www.theartofjoomla.com- the art of becoming a Joomla developer
>

> 2009/11/17 Matias Aguirre <fastsl...@gmail.com>:

>
>
>
> > Andrew,
>
> > Im trying to do the migration progress. Users and groups is done. Now
> > im working with categories and i have some doubts, if you or any can
> > to guide me is apreciate.
>
> > My problem is with the #__assets table. When Joomla adds a new
> > category, it add a new row in this table. I need to know what is the
> > rule of the `name` field. To make this more understandable:
>
> > +----+------------------------+----------------------------+
> > | id | name                   | title                      |
> > +----+------------------------+----------------------------+
> > | 29 | .category.3            | Original Category Test 1:1 |
> > | 28 | com_content.category.2 | Original Category Test 1   |
> > +----+------------------------+----------------------------+
>
> > When i add a new category with root parent, it add name with
> > com_content, and if i add a category with parent of the last cat, it
> > not add a com_content. Its a bug of Joomla or i need to follow this
> > rule?
>
> > My plan to do this is to migrate all the section with ROOT parent and
> > the categories following the sections parent.
>
> > Take care
>
> > On Nov 16, 4:54 am, Andrew Eddie <mambob...@gmail.com> wrote:
> >> Just following up.
>
> >> Anyone given this (Joomla 1.5 to 1.6 upgrade process) any traction?
>
> >> Regards,

> >> Andrew Eddiehttp://www.theartofjoomla.com-the art of becoming a Joomla developer

[Andrew Eddie]

Hi Matias.

Do you want a branch to work on in the Joomla SVN repo? Easier to
coach you on how to make beautiful code then :)

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer




2009/11/17 Matias Aguirre <fast...@gmail.com>:

[Matias Aguirre]

Sure, my username on forge.joomla.org is maguirre.

I need to know the name of the new component and some other small
indications. I think the best and simplest way is to do all the
migration in a separated directory like j16upgrade or something like
that. When the migration is done, the user only needs to move this
directory to the root of the webserver.

Let me know what do you think about this.

Take care

On Nov 16, 11:47 pm, Andrew Eddie <mambob...@gmail.com> wrote:
> Hi Matias.
>

> Do you want a branch to work on in the Joomla SVN repo?  Easier to
> coach you on how to make beautiful code then :)
>
> Regards,

> Andrew Eddiehttp://www.theartofjoomla.com- the art of becoming a Joomla developer
>
> 2009/11/17 Matias Aguirre <fastsl...@gmail.com>:
>
>
>

> > Thanks you very much for the help.
>
> > Here you can test the users and categories migration:

> >http://matware.com.ar/downloads/mtwmigrator/(0.3.1alpha version)

> >> >> Andrew Eddiehttp://www.theartofjoomla.com-theart of becoming a Joomla developer

[Andrew Eddie]

Thanks Matias

I've asked Sam to set you up with access.

I think I'd go with a folder name of "jupgrade" but only to avoid
numbers (with luck in the future it can just work out the required
upgrades) but whatever you think is fine.

Regards,
Andrew Eddie
http://www.theartofjoomla.com - the art of becoming a Joomla developer




2009/11/17 Matias Aguirre <fast...@gmail.com>:

[Sam Moffatt]

Access created, bit behind on my inbox.

Sam Moffatt
http://pasamio.id.au

[Matias Aguirre]

Hi Sam,

a little stupid question: which directory i can use to import the new
component?

Take care

On Nov 18, 9:07 pm, Sam Moffatt <pasa...@gmail.com> wrote:
> Access created, bit behind on my inbox.
>

> Sam Moffatthttp://pasamio.id.au

>
>
>
> On Tue, Nov 17, 2009 at 1:49 PM, Andrew Eddie <mambob...@gmail.com> wrote:
>
> > Thanks Matias
>
> > I've asked Sam to set you up with access.
>
> > I think I'd go with a folder name of "jupgrade" but only to avoid
> > numbers (with luck in the future it can just work out the required
> > upgrades) but whatever you think is fine.
>
> > Regards,
> > Andrew Eddie

> >http://www.theartofjoomla.com- the art of becoming a Joomla developer
>
> > 2009/11/17 Matias Aguirre <fastsl...@gmail.com>:
>
> >> Sure, my username on forge.joomla.org is maguirre.
>
> >> I need to know the name of the new component and some other small
> >> indications. I think the best and simplest way is to do all the
> >> migration in a separated directory like j16upgrade or something like
> >> that. When the migration is done, the user only needs to move this
> >> directory to the root of the webserver.
>
> >> Let me know what do you think about this.
>
> >> Take care
>
> >> On Nov 16, 11:47 pm, Andrew Eddie <mambob...@gmail.com> wrote:
> >>> Hi Matias.
>
> >>> Do you want a branch to work on in the Joomla SVN repo?  Easier to
> >>> coach you on how to make beautiful code then :)
>
> >>> Regards,

> >>> Andrew Eddiehttp://www.theartofjoomla.com-the art of becoming a Joomla developer
>
> >>> 2009/11/17 Matias Aguirre <fastsl...@gmail.com>:
>

[Sam Moffatt]

Hi Matias,

I just created a /development/branches/jupgrade directory that you can
use to put stuff in.

Cheers,

Sam Moffatt
http://pasamio.id.au

[Matias Aguirre]

HI all,

Here a beta version of jUpgrade: http://matware.com.ar/downloads/jupload/
it only support users and categories migration for now, but i like to
show you the method to do the migration and get some feedback too.

A few things to know:

# The download function points to my webserver because there are a
snapshop of the current svn version and im working with this version.
please let me know where i can to upload a new snapshot.
# Im using jQuery, i know that the preferred javascript library for
Joomla is mootools but jquery is more easy for me. There are a initial
support for mootools but i discontinued this (too much errors).

now im importing the files to the svn branch so you can access with
svn too.

Any doubts please tell me.

Take care

On Nov 20, 12:14 am, Sam Moffatt <pasa...@gmail.com> wrote:
> Hi Matias,
>
> I just created a /development/branches/jupgrade directory that you can
> use to put stuff in.
>
> Cheers,
>

> Sam Moffatthttp://pasamio.id.au

>
> On Fri, Nov 20, 2009 at 9:07 AM, Matias Aguirre <fastsl...@gmail.com> wrote:
>
> > Hi Sam,
>
> > a little stupid question: which directory i can use to import the new
> > component?
>
> > Take care
>
> > On Nov 18, 9:07 pm, Sam Moffatt <pasa...@gmail.com> wrote:
> >> Access created, bit behind on my inbox.
>
> >> Sam Moffatthttp://pasamio.id.au
>
> >> On Tue, Nov 17, 2009 at 1:49 PM, Andrew Eddie <mambob...@gmail.com> wrote:
>
> >> > Thanks Matias
>
> >> > I've asked Sam to set you up with access.
>
> >> > I think I'd go with a folder name of "jupgrade" but only to avoid
> >> > numbers (with luck in the future it can just work out the required
> >> > upgrades) but whatever you think is fine.
>
> >> > Regards,
> >> > Andrew Eddie

> >> >http://www.theartofjoomla.com-the art of becoming a Joomla developer
>
> >> > 2009/11/17 Matias Aguirre <fastsl...@gmail.com>:
>
> >> >> Sure, my username on forge.joomla.org is maguirre.
>
> >> >> I need to know the name of the new component and some other small
> >> >> indications. I think the best and simplest way is to do all the
> >> >> migration in a separated directory like j16upgrade or something like
> >> >> that. When the migration is done, the user only needs to move this
> >> >> directory to the root of the webserver.
>
> >> >> Let me know what do you think about this.
>
> >> >> Take care
>
> >> >> On Nov 16, 11:47 pm, Andrew Eddie <mambob...@gmail.com> wrote:
> >> >>> Hi Matias.
>
> >> >>> Do you want a branch to work on in the Joomla SVN repo?  Easier to
> >> >>> coach you on how to make beautiful code then :)
>
> >> >>> Regards,

[Matias Aguirre]

http://matware.com.ar/downloads/jupgrade/

sorry

> > >> >http://www.theartofjoomla.com-theart of becoming a Joomla developer
>
> > >> > 2009/11/17 Matias Aguirre <fastsl...@gmail.com>:
>
> > >> >> Sure, my username on forge.joomla.org is maguirre.
>
> > >> >> I need to know the name of the new component and some other small
> > >> >> indications. I think the best and simplest way is to do all the
> > >> >> migration in a separated directory like j16upgrade or something like
> > >> >> that. When the migration is done, the user only needs to move this
> > >> >> directory to the root of the webserver.
>
> > >> >> Let me know what do you think about this.
>
> > >> >> Take care
>
> > >> >> On Nov 16, 11:47 pm, Andrew Eddie <mambob...@gmail.com> wrote:
> > >> >>> Hi Matias.
>
> > >> >>> Do you want a branch to work on in the Joomla SVN repo?  Easier to
> > >> >>> coach you on how to make beautiful code then :)
>
> > >> >>> Regards,

> > >> >>> >> Andrew Eddiehttp://www.theartofjoomla.com-theartofbecoming a Joomla developer

[Matias Aguirre]

Sorry my server was down the last 12 hours. you can get the file now.

take care

> > > >> >http://www.theartofjoomla.com-theartof becoming a Joomla developer
>
> > > >> > 2009/11/17 Matias Aguirre <fastsl...@gmail.com>:
>
> > > >> >> Sure, my username on forge.joomla.org is maguirre.
>
> > > >> >> I need to know the name of the new component and some other small
> > > >> >> indications. I think the best and simplest way is to do all the
> > > >> >> migration in a separated directory like j16upgrade or something like
> > > >> >> that. When the migration is done, the user only needs to move this
> > > >> >> directory to the root of the webserver.
>
> > > >> >> Let me know what do you think about this.
>
> > > >> >> Take care
>
> > > >> >> On Nov 16, 11:47 pm, Andrew Eddie <mambob...@gmail.com> wrote:
> > > >> >>> Hi Matias.
>
> > > >> >>> Do you want a branch to work on in the Joomla SVN repo?  Easier to
> > > >> >>> coach you on how to make beautiful code then :)
>
> > > >> >>> Regards,

> > > >> >>> >> Andrew Eddiehttp://www.theartofjoomla.com-theartofbecominga Joomla developer