Joomla 3.6.2/Smart search - utf8_unicode_ci vs utf8_general_ci

[Sergio Manzi]

After upgrading from 3.4.8 to 3.6.2 all my tables have been converted to the utf8mb, and that's expected, but I noticed that there are differences as far as regard collation: most tables have been set to utf8_unicode_ci but all "#_finder_*" tables ("Smart search", if I'm not mistaken) have been set as utf8_general_ci.

AFAIK utf8_general_ci does not supports ligatures (ß == ss) and characters substitution (ÀÁÅåāă == a).

Is it OK the way it is?

Thanks,

Sergio

[George Wilson]

It's not really OK - but it causes massive mysql issues - see this tracker for more information https://github.com/joomla/joomla-cms/pull/9387 and we couldn't see any resolution other than forcibly purging and reindexing during the update - which could cause issues for people, obviously.

If anyone wants to look back at it feel free though!

Kind Regards,

George

[Sergio Manzi]

Thanks, George, I've read the "issue" you pointed me at and of which I was unaware.

AFAIU, and in theory (I'm not using Smart Search right now), it would be OK to:

• be sure that Weblink is upgraded or disabled (I don't have that either...)
• empty all smart search tables
• assign them utf8_unicode_ci collation
• reindex the whole enchilada

right?

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send email to joomla-...@googlegroups.com.
Visit this group at https://groups.google.com/group/joomla-dev-cms.
For more options, visit https://groups.google.com/d/optout.

[Chris Davenport]

That should work, although I think you mean utf8mb4_unicode_ci in step 3.

Chris.

On 8 August 2016 at 01:15, Sergio Manzi <s...@smz.it> wrote:

Thanks, George, I've read the "issue" you pointed me at and of which I was unaware.

AFAIU, and in theory (I'm not using Smart Search right now), it would be OK to:

• be sure that Weblink is upgraded or disabled (I don't have that either...)
• empty all smart search tables
• assign them utf8_unicode_ci collation
• reindex the whole enchilada

right?

On 2016-08-08 01:02, 'George Wilson' via Joomla! CMS Development wrote:

It's not really OK - but it causes massive mysql issues - see this tracker for more information https://github.com/joomla/joomla-cms/pull/9387 and we couldn't see any resolution other than forcibly purging and reindexing during the update - which could cause issues for people, obviously.

If anyone wants to look back at it feel free though!

Kind Regards,

George

On Sunday, August 7, 2016 at 9:11:46 PM UTC+1, Sergio Manzi wrote:

After upgrading from 3.4.8 to 3.6.2 all my tables have been converted to the utf8mb, and that's expected, but I noticed that there are differences as far as regard collation: most tables have been set to utf8_unicode_ci but all "#_finder_*" tables ("Smart search", if I'm not mistaken) have been set as utf8_general_ci.

AFAIK utf8_general_ci does not supports ligatures (ß == ss) and characters substitution (ÀÁÅåāă == a).

Is it OK the way it is?

Thanks,

Sergio

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cms+unsubscribe@googlegroups.com.
To post to this group, send email to joomla-dev-cms@googlegroups.com.

Visit this group at https://groups.google.com/group/joomla-dev-cms.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cms+unsubscribe@googlegroups.com.
To post to this group, send email to joomla-dev-cms@googlegroups.com.

Visit this group at https://groups.google.com/group/joomla-dev-cms.
For more options, visit https://groups.google.com/d/optout.

--

Chris Davenport
Joomla Production Leadership Team

[Sergio Manzi]

On 2016-08-08 10:48, Chris Davenport wrote:

That should work, although I think you mean utf8mb4_unicode_ci in step 3.

Of course, yes, utf8mb4_unicode_ci! Sorry for the mix-up...

Sergio

===

P.S.: Chris, did you got the mail I sent you about a week ago? In case not, can you please ping me from an email address you normally read? tnx!

[George Wilson]

Weblinks should be unrelated I *think* but yes for the rest of your points. The problem is doing this for all people is almost impossible. And if you do it manually then database fixer will shout at you. It's probably something that will get resolved in a Joomla 4 - where we'll ask people to do this as part of that update/migration.

On Monday, August 8, 2016 at 1:15:24 AM UTC+1, Sergio Manzi wrote:

Thanks, George, I've read the "issue" you pointed me at and of which I was unaware.

AFAIU, and in theory (I'm not using Smart Search right now), it would be OK to:

• be sure that Weblink is upgraded or disabled (I don't have that either...)
• empty all smart search tables
• assign them utf8_unicode_ci collation
• reindex the whole enchilada

right?

On 2016-08-08 01:02, 'George Wilson' via Joomla! CMS Development wrote:

It's not really OK - but it causes massive mysql issues - see this tracker for more information https://github.com/joomla/joomla-cms/pull/9387 and we couldn't see any resolution other than forcibly purging and reindexing during the update - which could cause issues for people, obviously.

If anyone wants to look back at it feel free though!

Kind Regards,

George

On Sunday, August 7, 2016 at 9:11:46 PM UTC+1, Sergio Manzi wrote:

After upgrading from 3.4.8 to 3.6.2 all my tables have been converted to the utf8mb, and that's expected, but I noticed that there are differences as far as regard collation: most tables have been set to utf8_unicode_ci but all "#_finder_*" tables ("Smart search", if I'm not mistaken) have been set as utf8_general_ci.

AFAIK utf8_general_ci does not supports ligatures (ß == ss) and characters substitution (ÀÁÅåāă == a).

Is it OK the way it is?

Thanks,

Sergio

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cms+unsubscribe@googlegroups.com.
To post to this group, send email to joomla-dev-cms@googlegroups.com.

[Sergio Manzi]

I didn't thought about the database fixer complaining about that... right!

I got that Weblinks can be nastily involved if you uninstalled them (and hence subsequently you didn't got upgrades for them) and you forgot to disable Weblinks Smart Search plugin (see: https://github.com/joomla/joomla-cms/pull/9387#issuecomment-195988347 and following related posts).

One more question: as part of my sites upgrade process from 3.4.8 to 3.6.2 I also manually changed the "database collation" to utf8mb4_unicode_ci. This has no effect for already defined tables, being just "a default", a "hint" for newly created tables, but afterward I came to think that probably this is not a good idea as this would mean that when installing any new (hypothetical) extension not explicitly defining its DB tables collation, those tables will be created as uf8mb4_unicode_ci, but the extension itself could not correctly handle the fact (but on the other hand I don't see how this could happen...). What's your opinion about this?

I found a nice article about this matter: https://mathiasbynens.be/notes/mysql-utf8mb4

Regards,

Sergio

To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send email to joomla-...@googlegroups.com.

[George Wilson]

Ahh ok. I understand if you uninstalled the package for weblinks to start with you should be fine. It's just if you didn't uninstall everything.

In relation to you changing the default database collation. It ought to be just fine. Although obviously it depends on each extension. But if they are using JFilter it should be just fine.

That articles good. We consulted it extensively when doing all the utf8mb4 fixes during the long 3.5 beta period.