japanese, chinese and...IMEs on the spot on QT

[johnsonj]

I have manged to succeed in implementing one liner ime on qscintilla2
speedy and simple.

scintillaQT do not know MoveimeCarets(); multi carets.
How can I port these ime functions to scintillaQT?

[Neil Hodgson]

johnsonj:

QScintilla2 is an external platform layer maintained by Phil Thompson / Riverbank Computing. You could try the QScintilla mailing list if there is no reply here.
http://www.riverbankcomputing.com/software/qscintilla/intro

Neil

[johnsonj]

I failed to compile scintillaQt

So I used qscintilla2 to implement ime on Qt.
solved to move imeFuncitons to editor.cxx and editor.h

It works very good but indicators problem bothers me.
My intention is on scintillaQt.
but there is no SciTEQT like SciteWin SciteGtk.
I think scitillaqt needs to be enhanced.

[Neil Hodgson]

johnsonj:

I failed to compile scintillaQt

   The core Qt platform layer has two products: ScintillaEditBase which is simpler and ScintillaEdit that has an API with an method for each Scintilla feature. You can build ScintillaEditBase either by loading the qt/ScintillaEditBase/ScintillaEditBase.pro file into QtCreator and running the build command or using the qmake command in that directory. Sometimes the correct qmake command is called qmake-qt4.

   I thought ScintillaEditBase already used an inline IME.

   There is a small demonstration application for ScintillaEditBase called Haven:

http://scintilla.org/haven.zip

   Neil

[johnsonj]

Thank you for Haven.
I implemented ime for scintillaQt thanks to Haven.

I moved imefunctions to editor.cxx and editor.h.
I could not solve inOverstrike and language.
but It works well.
Please take a look.
following is the patch;

void ScintillaEditBase::inputMethodEvent(QInputMethodEvent *event)
{
    if (sqt->pdoc->TentativeActive()) {
        sqt->pdoc->TentativeUndo();
    } else {
        // No tentative undo means start of this composition so
        // fill in any virtual spaces.
        //bool tmpOverstrike = inOverstrike;
        //inOverstrike = false;  // not allow to be deleted twice.
        sqt->AddCharUTF("", 0);
        //sci->inOverstrike = tmpOverstrike;
    }
   
    bool compstrExist = false;
    int imeCursorPos = 0;
    sqt->view.imeCaretBlockOverride = false;
   
    if (!event->commitString().isEmpty()) {
        QString commitStr = event->commitString();
        unsigned int commitStrLen = commitStr.length();
        compstrExist = (commitStrLen == 0);

        for (unsigned int i = 0; i < commitStrLen; i++) {
            QString sub =  commitStr.mid(i, 1);
            QByteArray subBytes = sqt->BytesForDocument(sub);
            sqt->AddCharUTF(subBytes.data(), subBytes.length());
        }

    } else if (!event->preeditString().isEmpty()) {
        QString preeditStr = event->preeditString();
        unsigned int preeditStrLen = preeditStr.length();
        compstrExist = (preeditStrLen != 0);

        if ((preeditStrLen == 0) || (preeditStrLen > 200)) {
            return;
        }
        sqt->pdoc->TentativeStart();

        foreach (QInputMethodEvent::Attribute attr, event->attributes()) {
            if (attr.type == QInputMethodEvent::Cursor) {
                if (attr.length) {
                    imeCursorPos = attr.start;
                }
            } else if (attr.type == QInputMethodEvent::Language) {
                //QString qlang = QLocale::languageToString(attr.value<Langguage><QLocale>())
            } else if (attr.type == QInputMethodEvent::TextFormat) {
                QTextFormat format = attr.value.value<QTextFormat>();
                QTextCharFormat charFormat = format.toCharFormat();
                // Apply attributes to the preedit char.
                int indicator= SC_INDICATOR_UNKNOWN;
                if (charFormat.underlineStyle() == QTextCharFormat::NoUnderline) {
                    indicator = SC_INDICATOR_TARGET;
                    imeCursorPos = attr.start;
                } else {
                    indicator = SC_INDICATOR_INPUT;
                }

                QString sub =  preeditStr.mid(attr.start, attr.length);
                QByteArray subBytes = sqt->BytesForDocument(sub);
                int subLength = subBytes.length();
                bool recording = sqt->recordingMacro;
                sqt->recordingMacro = false;
                sqt->AddCharUTF(subBytes.data(), subLength);
                sqt->recordingMacro = recording;
               
                sqt->DrawImeIndicator(indicator, subLength);
            }
        }
       
        int imeCharPos[200] = {0};
        unsigned int numBytes = 0;
        for (unsigned int i = 0; i < preeditStrLen; i++) {
            QString sub =  preeditStr.mid(i, 1);
            QByteArray subBytes = sqt->BytesForDocument(sub);
            int subLength = subBytes.length();
            numBytes += subLength;
            imeCharPos[i+1] = numBytes;
        }
        bool koreanIme = true;
        if (koreanIme) {
            sqt->view.imeCaretBlockOverride = true;
        }
        sqt->MoveImeCarets(-imeCharPos[preeditStrLen]);
        sqt->MoveImeCarets(imeCharPos[imeCursorPos]);
    }
    sqt->ShowCaretAtCurrentPosition();
}

[johnsonj]

can not make korean ime be triggered.
can not find the function to set preedit rect on linux.

------------------------------------ here is the patch -----------------------------
void ScintillaEditBase::inputMethodEvent(QInputMethodEvent *event)
{
    //copy & paste by johnsonj with a lot of helps of Neil
    //Great thanks for my foreruners, jiniya and BLUEnLIVE

    if (sqt->pdoc->TentativeActive()) {
        sqt->pdoc->TentativeUndo();
    }

    int imeCursorPos = 0;
    int imeCharPos[sqt->maxLenInputIME] = {0};
    int imeIndicator[sqt->maxLenInputIME] = {SC_INDICATOR_UNKNOWN};

    sqt->view.imeCaretBlockOverride = false;

    if (!event->commitString().isEmpty()) {
        QString commitStr = event->commitString();
        unsigned int commitStrLen = commitStr.length();

        for (unsigned int i = 0; i < commitStrLen; i++) {
            QString sub =  commitStr.mid(i, 1);
            QByteArray subBytes = sqt->BytesForDocument(sub);
            sqt->AddCharUTF(subBytes.data(), subBytes.length());
        }

    } else if (!event->preeditString().isEmpty()) {
        QString preeditStr = event->preeditString();
        unsigned int preeditStrLen = preeditStr.length();

        if ((preeditStrLen == 0) || (preeditStrLen > sqt->maxLenInputIME)) {
            sqt->ShowCaretAtCurrentPosition();
            return;
        }

        sqt->pdoc->TentativeStart(); //TentativeAcitive() from now on

        QLocale::Language inputLang;

        foreach (QInputMethodEvent::Attribute attr, event->attributes()) {

            if (attr.type == QInputMethodEvent::TextFormat) {
                QTextFormat format = attr.value.value<QTextFormat>();
                QTextCharFormat charFormat = format.toCharFormat();

                int indicator = SC_INDICATOR_UNKNOWN;
                switch (charFormat.underlineStyle())
                {
                case QTextCharFormat::DashUnderline:
                    indicator = SC_INDICATOR_INPUT; //normal input
                    break;
                case QTextCharFormat::NoUnderline:
                    indicator = SC_INDICATOR_TARGET; //target input
                    imeCursorPos = attr.start;
                    break;
                }

                for (int i = attr.start; i < attr.start+attr.length; i++) {
                    imeIndicator[i] = indicator;
                }
            } else if (attr.type == QInputMethodEvent::Cursor) {

                if (attr.length) {
                    imeCursorPos = attr.start;
                }
            } else if (attr.type == QInputMethodEvent::Language) {

                inputLang = attr.value.value<QLocale>().language();

            }
        }

        unsigned int numBytes = 0;
        for (unsigned int i = 0; i < preeditStrLen; i++) {
            QString sub =  preeditStr.mid(i, 1);
            QByteArray subBytes = sqt->BytesForDocument(sub);
            int subLength = subBytes.length();
            numBytes += subLength;
            imeCharPos[i+1] = numBytes;

            bool recording = sqt->recordingMacro;
            sqt->recordingMacro = false;
            sqt->AddCharUTF(subBytes.data(), subLength);
            sqt->recordingMacro = recording;

            sqt->DrawImeIndicator(imeIndicator[i], subLength);
        }

        if (inputLang == QLocale::Korean) {

[johnsonj]

how about moving ime functions to some other place.

[Neil Hodgson]

johnsonj:

> how about moving ime functions to some other place.

Once they are moved to the platform-independent code then its much more difficult to change them since other platform layers will come to depend on their behaviour never changing.

Neil

[johnsonj]

I see.
attached the patch

[johnsonj]

attriubutes has no QInputMethodEvent::Language.
I checked it on linux.

Patch was tested on winXp and linux.

[johnsonj]

cleaner.

* problems on linux Qt:
1 linux Qt do not allow caret go between preedit string.
2 need to set preedit box position on lunux Qt.

I found My patch has a weak point: need lock?
what if input comes indo before tentativeundo has completed.

[johnsonj]

I have been learning lots of things about ime from Neil's kind explanations.
Thanks Neil.
I will keep going on within Qt for whom are interested.
this patch may not work outside Qt.

copied selector bubble codes from original ones for testing.
fixed preeditPos for retangular selection.

I found bugs in scintilla and Qt on linux while trying to set candidate window position.
scintilla part : always return pt.x=0 pt.y=0.
      Point pt = sqt->LocationFromPosition(startPos);
Qt4.8.6 part : always positioned near bottom left. Setting position manually also does not work.
      QRect(pt.x, pt.y, width, height)

Weird, I do not understand.

[johnsonj]

I detected the source of problem.
Installing ibus-qt returns correct ime curosr position.
I remember it worked qt4.8.3 on linux

I have experineced a lot of changes of setting.
Now I reached Qt4.8.6 on debian jessie and the same on lubuntu 14.10 on Xp virtualbox.
it introduces another bug.
Qt returns only one single underline style.

So I decide not to depend underlie style.
Here follows my assumptions.

just for a review
---------------------------------------------------------------------

void ScintillaEditBase::inputMethodEvent(QInputMethodEvent *event)
{
    // copy & paste by johnsonj with a lot of helps of Neil
    // Great thanks for my foreruners, jiniya and BLUEnLIVE

    if (sqt->pdoc->TentativeActive()) {
        sqt->pdoc->TentativeUndo();

    } else {
        // No tentative undo means start of this composition so

        // Fill in any virtual spaces.
        sqt->FillVirtualSpace();

    }

    sqt->view.imeCaretBlockOverride = false;

    if (!event->commitString().isEmpty()) {

        const QChar *imeChars = event->commitString().constData();
        unsigned int imeCharsLen = event->commitString().length();
        for (unsigned int i = 0; i < imeCharsLen; i++)
        {
            QByteArray oneChar = sqt->BytesForDocument(imeChars[i]);
            sqt->AddCharUTF(oneChar.data(), oneChar.length());
        }
    } else if (!event->preeditString().isEmpty()) {
        const QChar *imeChars = event->preeditString().constData();
        unsigned int imeCharsLen = event->preeditString().length();

        if ((imeCharsLen == 0) || (imeCharsLen > sqt->maxLenInputIME)) {

            sqt->ShowCaretAtCurrentPosition();
            return;
        }

        sqt->pdoc->TentativeStart(); //TentativeAcitive() from now on

        int imeIndicator[sqt->maxLenInputIME] = {SC_INDICATOR_UNKNOWN};

        int imeCaretPos = 0;
        foreach (const QInputMethodEvent::Attribute &attr, event->attributes())

        {
            if (attr.type == QInputMethodEvent::TextFormat) {

                // Get ime attibutes, overlapping not allowed

                for (int i = attr.start; i < attr.start+attr.length; i++) {

                    imeIndicator[i] = attr.start; //Mark block

                }
            } else if (attr.type == QInputMethodEvent::Cursor) {

                    imeCaretPos = attr.start;

            }
        }

        int imeCharPos[sqt->maxLenInputIME] = {0};

        unsigned int numBytes = 0;

        bool recording = sqt->recordingMacro;
        sqt->recordingMacro = false;

        for (unsigned int i = 0; i < imeCharsLen; i++)
        {
            QByteArray oneChar = sqt->BytesForDocument(imeChars[i]);
            int oneCharLen = oneChar.length();

            sqt->AddCharUTF(oneChar.data(), oneCharLen);

            if (imeIndicator[i] == imeCaretPos) {
                DrawImeIndicator(SC_INDICATOR_TARGET, oneCharLen);
            } else {
                DrawImeIndicator(SC_INDICATOR_INPUT, oneCharLen);
            }

            // Record character positions for moving ime caret
            numBytes += oneCharLen;
            imeCharPos[i+1] = numBytes;
        }
        sqt->recordingMacro = recording;

        if (IsHangul(imeChars[0])) {
            sqt->view.imeCaretBlockOverride = true;
            MoveImeCarets(- imeCharPos[imeCharsLen]);
        } else {
            MoveImeCarets(- imeCharPos[imeCharsLen] + imeCharPos[imeCaretPos]);
        }

        //Set candidate box position for Qt::ImMicroFocus
        preeditPos = sqt->sel.MainCaret();
    }
    sqt->ShowCaretAtCurrentPosition();
}

[johnsonj]

perfect, cool!

[johnsonj]

I think it can handle accented input.
May be the last patch!

[johnsonj]

TentiveUndo() eats up

        case Qt::ImFont:
        case Qt::ImCursorPosition:
        case Qt::ImSurroundingText:
        case Qt::ImCurrentSelection:

==========================================================
void ScintillaEditBase::inputMethodEvent(QInputMethodEvent *event)
{
    // Copy & paste by johnsonj with a lot of helps of Neil

    // Great thanks for my foreruners, jiniya and BLUEnLIVE

    if (sqt->pdoc->TentativeActive()) {
        sqt->pdoc->TentativeUndo();
    } else {
        // No tentative undo means start of this composition so
        // Fill in any virtual spaces.
        sqt->FillVirtualSpace();
    }

    sqt->view.imeCaretBlockOverride = false;

    if (!event->commitString().isEmpty()) {

        unsigned int const commitStrLen = event->commitString().length();
        for (unsigned int i = 0; i < commitStrLen; i++)
        {
            QByteArray oneChar = sqt->BytesForDocument(event->commitString().at(i));

            sqt->AddCharUTF(oneChar.data(), oneChar.length());
        }
    } else if (!event->preeditString().isEmpty()) {

        unsigned int const preeditStrLen = event->preeditString().length();
        if ((preeditStrLen == 0) || (preeditStrLen > sqt->maxLenInputIME)) {
            sqt->ShowCaretAtCurrentPosition();
            return;
        }

        sqt->pdoc->TentativeStart(); // TentativeAcitive() from now on.

        // Get underline types, segments and ime caret position.
        unsigned int imeIndicator[sqt->maxLenInputIME] = {0};
        unsigned int imeCaretPos = 0;
        unsigned int imeLinePattern[sqt->maxLenInputIME] = {0};
        unsigned int attrLen = 0;
        bool koreanIme = false;

        foreach (QInputMethodEvent::Attribute attr, event->attributes())

        {
            if (attr.type == QInputMethodEvent::TextFormat) {

                QTextFormat format = attr.value.value<QTextFormat>();
                QTextCharFormat charFormat = format.toCharFormat();

                unsigned int indicator = SC_INDICATOR_UNKNOWN;
                switch (charFormat.underlineStyle())
                {

                case QTextCharFormat::NoUnderline:
                    indicator = SC_INDICATOR_TARGET; //target input

                    break;
                case QTextCharFormat::SingleUnderline:
                    attrLen += attr.length; // for Detecting ibus-Qt bug

                case QTextCharFormat::DashUnderline:
                    indicator = SC_INDICATOR_INPUT; //normal input
                    break;

                case QTextCharFormat::DotLine:
                case QTextCharFormat::DashDotLine:
                case QTextCharFormat::WaveUnderline:
                case QTextCharFormat::SpellCheckUnderline:
                    indicator = SC_INDICATOR_CONVERTED;
                    break;

                }
                for (int i = attr.start; i < attr.start+attr.length; i++) {

                    imeIndicator[i] = indicator;
                    imeLinePattern[i] = attr.start; // Mark one segment for ibus-Qt bug

                }
            } else if (attr.type == QInputMethodEvent::Cursor) {
                imeCaretPos = attr.start;

            } else if (attr.type == QInputMethodEvent::Language) {
                // It never reaches here as of Qt 4.8.6 on XP and Linux
                QLocale qlocale = attr.value.value<QLocale>();
                QLocale::Language qlang = qlocale.language();
                koreanIme = (qlocale.language() == QLocale::Korean);
            }
        }

        // Display preedit characters one by one.

        int imeCharPos[sqt->maxLenInputIME] = {0};

        int numBytes = 0;

        bool recording = sqt->recordingMacro;
        sqt->recordingMacro = false;

        for (unsigned int i = 0; i < preeditStrLen; i++)
        {
            QByteArray oneChar = sqt->BytesForDocument(event->preeditString().at(i));

            int oneCharLen = oneChar.length();

            sqt->AddCharUTF(oneChar.data(), oneCharLen);

            DrawImeIndicator(imeIndicator[i], oneCharLen);

            bool ibusQtBug = (attrLen == preeditStrLen);
            // It means only one single underline comes in.
            if (ibusQtBug) {  // Detect the target input segment with imeCaretPos
                if (imeLinePattern[i] == imeCaretPos) {

                    DrawImeIndicator(SC_INDICATOR_TARGET, oneCharLen);
                } else {
                    DrawImeIndicator(SC_INDICATOR_INPUT, oneCharLen);
                }
            }

            // Record character positions for moving ime caret.

            numBytes += oneCharLen;
            imeCharPos[i+1] = numBytes;
        }
        sqt->recordingMacro = recording;

        // Move IME carets.
        if (koreanIme || IsHangul(event->preeditString().at(0))) {
            sqt->view.imeCaretBlockOverride = true;
            MoveImeCarets(- imeCharPos[preeditStrLen]);
        } else {
            MoveImeCarets(- imeCharPos[preeditStrLen] + imeCharPos[imeCaretPos]);
        }

        // Set candidate box position for Qt::ImMicroFocus.

        preeditPos = sqt->sel.MainCaret();
    }
    sqt->ShowCaretAtCurrentPosition();
}

QVariant ScintillaEditBase::inputMethodQuery(Qt::InputMethodQuery query) const
{
    int pos = send(SCI_GETCURRENTPOS);
    int line = send(SCI_LINEFROMPOSITION, pos);

    switch (query) {
        case Qt::ImMicroFocus:
        {
            int startPos = (preeditPos >= 0) ? preeditPos : pos;

            Point pt = sqt->LocationFromPosition(startPos);

            int width = send(SCI_GETCARETWIDTH);
            int height = send(SCI_TEXTHEIGHT, line);
            return QRect(pt.x, pt.y, width, height);
        }

        case Qt::ImFont:
        case Qt::ImCursorPosition:
        case Qt::ImSurroundingText:
        case Qt::ImCurrentSelection:

        default:
            return QVariant();
    }
}

[Neil Hodgson]

The source code for Qt is open so can be examined to determine how the IME works.
The QInputMethodEvent::Language attribute never seems to be set (in Qt 4.8.2) but QInputContext::language() is implemented.
The X 11 version is, for example, in src\gui\inputmethod\qximinputcontext_x11.cpp.
However, this is Qt 4.x-specific, since Qt 5 uses QPlatformInputContext instead of QInputContext.

Since Qt 5 will be increasing in use, changes should be compatible with it.

Here are the implementations of QInputMethodEvent::Language for Win32, OS X, Nokia phones, and X 11.

QString QWinInputContext::language()
{
return QString();
}

QString QMacInputContext::language()
{
return QString();
}

QString QCoeFepInputContext::language()
{
TLanguage lang = m_fepState->LocalLanguage();
const QByteArray localeName = qt_symbianLocaleName(lang);
if (!localeName.isEmpty()) {
return QString::fromLatin1(localeName);
} else {
return QString::fromLatin1("C");
}
}

QString QXIMInputContext::language()
{
QString language;
if (xim) {
QByteArray locale(XLocaleOfIM(xim));

if (locale.startsWith("zh")) {
// Chinese language should be formed as "zh_CN", "zh_TW", "zh_HK"
language = QLatin1String(locale.left(5));
} else {
// other languages should be two-letter ISO 639 language code
language = QLatin1String(locale.left(2));
}
}
return language;
}

[johnsonj]

I think Qt should give inputCodepage()-like function for programmers.
Windows give inputCodepage(), Gtk provide pangoScript
and so Qt should return QLocale::Language.

inputCodepage() is in need only for korean Ime.
Without QLocale::Language, ime has no problem.

I have not such a big picture as you have.
If Qt5 has not backward compatibility with Qt4,
I think it is bettter to start again from scratch for Qt5.

[Neil Hodgson]

johnsonj:

> If Qt5 has not backward compatibility with Qt4,
> I think it is bettter to start again from scratch for Qt5.

Qt5 currently builds. Any changes should not break this.

Neil

[johnsonj]

I am testing qscintilla for qt5 on linux.
I can not build haven, so testing with qscintilla2.
IME supporting qt5 is only fcitx.
fcitx-qt5 has a bug which returns only one dashline style like ibus-qt4 which returns only one single underline.
qscintilla2 demo on Qt5 on linux works cool!

So I wanted to build haven to test, but I got an error like this on qt4 and qt5 on linux
How can I handle this?
===================================================================================================================================
g++ -Wl,-rpath,\$ORIGIN/../scintilla/bin -Wl,-O1 -o ../bin/Haven main.o mainwindow.o moc_mainwindow.o    -L/usr/lib/i386-linux-gnu -L../scintilla/bin -lScintillaEditbase3 -lQtGui -lQtCore -lpthread
/usr/bin/ld: cannot find -lScintillaEditbase3
collect2: error: ld returned 1 exit status
Makefile:105: recipe for target '../bin/Haven' failed
make: *** [../bin/Haven] Error 1

[Neil Hodgson]

johnsonj:

> fcitx-qt5 has a bug which returns only one dashline style like ibus-qt4 which returns only one single underline.

So its the same code in Scintilla or it has to recognise different types of underline?

> qscintilla2 demo on Qt5 on linux works cool!
>
> So I wanted to build haven to test, but I got an error like this on qt4 and qt5 on linux
> How can I handle this?
> ===================================================================================================================================
> g++ -Wl,-rpath,\$ORIGIN/../scintilla/bin -Wl,-O1 -o ../bin/Haven main.o mainwindow.o moc_mainwindow.o -L/usr/lib/i386-linux-gnu -L../scintilla/bin -lScintillaEditbase3 -lQtGui -lQtCore -lpthread
> /usr/bin/ld: cannot find -lScintillaEditbase3

Linux is case-sensitive and the capitalisation in the project file is “ScintillaEditBase”.

Neil

[johnsonj]

Thank you.
ibus-qt4 returns only one dashline, fcitx-qt5 returns only one dottedline.
they never return nounderline for target input (Background color).
review the patch attaced.

I changed haven.pro like this.
----------------------------------------------------------------------------
macx {
    LIBS += -framework ScintillaEditBase
    QMAKE_LFLAGS += -F../scintilla/bin
} else {
    unix:!macx {
       LIBS += -L../scintilla/bin -lScintillaEditBase
    } else {
        LIBS += -L../scintilla/bin -lScintillaEditBase3
    }
}
----------------------------------------------------------------------------
Now the ime patched scintillaqt was tested with haven for qt4 and qt5 on windows and linux.

                     qt4                        qt5                     note
windows          o                           x                   it  always returns the candidate box position (0,0)
linux                o                           o                   ibus-qt4 and fcitx-qt5 were used.

The untouched scintillaqt suffers one line type bug

[johnsonj]

I removed redundant codes drastically
+        // Target input segment is indicated with NoUnderline on Windows,
+        // SingleUnderline  on ibus-Qt4 and DashUnderline  on fcitx-Qt5.
+        // Moreover only one  underline style makes things worse.
+        // So mark segments with attr.pos respectedly,
+        // Detect the target input segment with ime caret postion.

[Neil Hodgson]

   In Qt classes the “slots” and “signals” sections are special and should only include methods that are slots or signals. They shouldn’t contain other methods like IsHangul. The new methods probably belong in the “private” section.

  Scintilla's Qt code is indented with tabs and uses K&R indentation style, so new code in these files should be the same. Comparing code is made more difficult with different brace positions and whitespace characters.

http://en.wikipedia.org/wiki/Indent_style#K.26R_style

   The image at http://scintilla.org/IMEs.png shows the appearance of a Japanese IME for the text “shimonoseki” with the caret moved back. GTK+ on Linux (Fedora 19) looks good with the whole composition marked and the caret obvious. Qt 5.4 on Win32 also looks OK. I don’t know what is going wrong with Qt 4.8.6 since the underline looks completely wrong (wrong colour, wrong shape underline), like its not being drawn by Scintilla. Maybe this is still using a windowed IME. It also fails when there are 2 carets with extra text (typing “sh” produces “ssh\nssh”).

   Also on Fedora 19 with GTK+ 2.24, the windowed IME option (SciTE: ime.interaction=0) doesn’t currently work well. The IME window remains with the text over Scintilla after Enter is pressed.

   Neil

[johnsonj]

Thank you for letting me understand why my diff file was irreglar.
It reminded me converting tab to spaces before making diff file.
I thought it has no problme since I set diff option -ubB.
I do not understand why I put ime functions to signal section instead of public one.

----------------------------------------------------------------------------------------------------------------

"""   Also on Fedora 19 with GTK+ 2.24, the windowed IME option (SciTE: ime.interaction=0) doesn’t currently work well. The IME window remains with the text over Scintilla after Enter is pressed."""

I do not like the existing ime.
I remember windowed ime has not work well since long time ago.
                    
windowed ime       windows             gtk
functionally         good              bad
visually             bad               bad

----------------------------------------------------------------------------------------------------------------

"""  I don’t know what is going wrong with Qt 4.8.6 since the underline looks completely wrong (wrong colour, wrong shape underline), like its not being drawn by Scintilla. Maybe this is still using a windowed IME. It also fails when there are 2 carets with extra text (typing “sh” produces “ssh\nssh”)."""

It is very complicated for Qt testing
Qt/win underline styles may be intended and correct visually.
Underline Styles follow Visually Qt/win without ibus-qt4 or fcitx-qt5.
But Have to install ibus-qt4 and fcitx-qt5 to get correct caret position

Heres I found.
Note: The only ime supporting Qt5 is fcitx.
                 qt4qt5/windows    ibus-qt4/linux    fcitx-qt5/linux
normal input      DashUnderline    SingleUnderline   DashUnderline
target input      Nounderline      SingleUnderline   DashUnderline
---------------------------------------------------------------------------------------------------------------

[johnsonj]

unabridged  for explaining qt ime mechanism.
-----------------------------------------------------------------------------------------------------------------

void ScintillaEditBase::inputMethodEvent(QInputMethodEvent *event)
{
    // Copy & paste by johnsonj with a lot of helps of Neil
    // Great thanks for my foreruners, jiniya and BLUEnLIVE

    if (sqt->pdoc->TentativeActive()) {
        sqt->pdoc->TentativeUndo();
    } else {
        // No tentative undo means start of this composition so
        // Fill in any virtual spaces.
        sqt->FillVirtualSpace();
    }

    sqt->view.imeCaretBlockOverride = false;

    if (!event->commitString().isEmpty()) {
        unsigned int const commitStrLen = event->commitString().length();
        for (unsigned int i = 0; i < commitStrLen; i++)
        {
            QByteArray oneChar = sqt->BytesForDocument(event->commitString().at(i));
            sqt->AddCharUTF(oneChar.data(), oneChar.length());
        }
    } else if (!event->preeditString().isEmpty()) {
        unsigned int const preeditStrLen = event->preeditString().length();
        if ((preeditStrLen == 0) || (preeditStrLen > sqt->maxLenInputIME)) {
        sqt->ShowCaretAtCurrentPosition();
            return;
        }

        sqt->pdoc->TentativeStart(); // TentativeAcitive() from now on.

        // Get underline types, segments and ime caret position.
        unsigned int imeIndicator[sqt->maxLenInputIME] = {0};
        unsigned int imeCaretPos = 0;

        unsigned int singleLen = 0;
        unsigned int dashLen = 0;

        unsigned int imeLinePattern[sqt->maxLenInputIME] = {0};

        bool koreanIme = false;

        foreach (QInputMethodEvent::Attribute attr, event->attributes())
        {
            if (attr.type == QInputMethodEvent::TextFormat) {
                QTextFormat format = attr.value.value<QTextFormat>();
                QTextCharFormat charFormat = format.toCharFormat();

                unsigned int indicator = SC_INDICATOR_UNKNOWN;
                switch (charFormat.underlineStyle())
                {
                case QTextCharFormat::NoUnderline:

                    // ibus-Qt4 and ibus-Qt5 do not use NoUnderline for target input

                    indicator = SC_INDICATOR_TARGET; //target input
                    break;
                case QTextCharFormat::SingleUnderline:

                    indicator = SC_INDICATOR_INPUT; //normal input

                    singleLen += attr.length; // ibus-Qt4 shares SingleUnderline for target input
                    break;

                case QTextCharFormat::DashUnderline:
                    indicator = SC_INDICATOR_INPUT; //normal input

                    dashLen += attr.length; // fcitx-Qt5 shares DashUnderline for target input

                    break;
                case QTextCharFormat::DotLine:
                case QTextCharFormat::DashDotLine:
                case QTextCharFormat::WaveUnderline:
                case QTextCharFormat::SpellCheckUnderline:
                    indicator = SC_INDICATOR_CONVERTED;
                    break;

                default:
                    indicator = SC_INDICATOR_UNKNOWN;

                }
                for (int i = attr.start; i < attr.start+attr.length; i++) {
                    imeIndicator[i] = indicator;
                    imeLinePattern[i] = attr.start; // Mark one segment
                }

            } else if (attr.type == QInputMethodEvent::Cursor) {
                imeCaretPos = attr.start;
            } else if (attr.type == QInputMethodEvent::Language) {

                // Here never reaches as of Qt4.8.6 on XP and Linux and Qt5.3 on Linux

                QLocale qlocale = attr.value.value<QLocale>();

                koreanIme = (qlocale.language() == QLocale::Korean);
            }
        }

        // Display preedit characters one by one.
        int imeCharPos[sqt->maxLenInputIME] = {0};
        int numBytes = 0;

        bool recording = sqt->recordingMacro;
        sqt->recordingMacro = false;
        for (unsigned int i = 0; i < preeditStrLen; i++)
        {
            QByteArray oneChar = sqt->BytesForDocument(event->preeditString().at(i));
            int oneCharLen = oneChar.length();

            // Record character positions for moving ime caret.
            numBytes += oneCharLen;
            imeCharPos[i+1] = numBytes;

            sqt->AddCharUTF(oneChar.data(), oneCharLen);
            DrawImeIndicator(imeIndicator[i], oneCharLen);
            

            // Folling codes shoud be removed if Qt/linux returns consistent underlines
            // Qt4 and Qt4 on windows uses NoUnderline for target input.
            // ibus-Qt4 shares SingleUnderline, fcitx shares DashUnderline with normal input.
            // segment marked above with ime caret position is the target segment.
            bool ibusQt4 = (singleLen == preeditStrLen);
            bool fcitxQt5 = (dashLen == preeditStrLen);
            //if (ibusQt4) {fprintf(stderr, "ibusQt4");}
            //if (fcitxQt5) {printf(stderr, "fcitxQt5");}
            if (ibusQt4 || fcitxQt5) {

                if (imeLinePattern[i] == imeCaretPos) {
                    DrawImeIndicator(SC_INDICATOR_TARGET, oneCharLen);
                } else {
                    DrawImeIndicator(SC_INDICATOR_INPUT, oneCharLen);
                }
            }
        }

        sqt->recordingMacro = recording;

        // Move IME carets.
        if (koreanIme || IsHangul(event->preeditString().at(0))) {
            sqt->view.imeCaretBlockOverride = true;
            MoveImeCarets(- imeCharPos[preeditStrLen]);
        } else {
            MoveImeCarets(- imeCharPos[preeditStrLen] + imeCharPos[imeCaretPos]);
        }

        // Set candidate box position for Qt::ImMicroFocus.
        preeditPos = sqt->sel.MainCaret();
    }
    sqt->ShowCaretAtCurrentPosition();
}

QVariant ScintillaEditBase::inputMethodQuery(Qt::InputMethodQuery query) const
{
    int pos = send(SCI_GETCURRENTPOS);

    switch (query) {
        case Qt::ImMicroFocus:
        {
            int startPos = (preeditPos >= 0) ? preeditPos : pos;
            Point pt = sqt->LocationFromPosition(startPos);

            int width = sqt->vs.caretWidth;
            int height = sqt->vs.lineHeight;

[johnsonj]

Is the image produced with my patch attached?
The second item(qt4/linux) uses dashed underline which means the existing ime.
Other items use dotted underline which means my patch.

[johnsonj]

patch attached. simplest!
I report my experiece to scintilla ime.
-----------------------------------------------------------------------------------------------------------------------------------------
                                                                                   inline  existing  windows   Gtk2.0     
　　　　　ime functional comparison                             ime     Qt ime  windowed  windowed
1. Can it show target input different from normal input?      o          x            o            o
2. Can it go between preedit characters?                          o          o            o            x
3. Can it show candidate box on under caret position?       o          x            x            x
4. Can it handle overstrike mode?                                    o          x            x            x
5. Can it handle multiple carets?                                      o          x            o            o

[Neil Hodgson]

johnsonj:

> Is the image produced with my patch attached?

I tried again with imeqt0113.patch and the underlines are correct blue dots. Since imeqt0113.patch is really the same as imeqt0111.patch except for whitespace, its possible I was seeing a stale copy of the old code.

Neil

[johnsonj]

Yes same code.
fixed as you instructed.

-----------------------------------------------------------------

"""I don’t know what is going wrong with Qt 4.8.6 since the underline looks completely wrong (wrong colour, wrong shape underline), like its not being drawn by Scintilla. Maybe this is still using a windowed IME. It also fails when there are 2 carets with extra text (typing “sh” produces “ssh\nssh”)."""

I guess the second item in the image must run on linux.
I think that is about the existing ime.
It shows dash underline which indicates the existing ime.
Have you ibus-qt4 installed which returns correct ime caret position?

[johnsonj]

Can you be more specific?
I have no problem about qt4q5 on linux.

[Neil Hodgson]

johnsonj:

> Have you ibus-qt4 installed which returns correct ime caret position?

That name isn’t offered by the Fedora Software app. It has ibus-qt-1.3.3-1.fc19 which I hadn’t installed. Installing it made no difference to Qt or GTK+.

Neil

[johnsonj]

I have tested qt4 and qt5 with haven and Qscintilla2 demo and partially pyqt.
                          on windows xp and lubuntu and mint on virtualbox.

Heres my testing ime. it took me a long time to find out.
     ibus, fcitx, uim, scim.
qt4   o      o     o     x
qt5   x      o     x     ?

qt4 and qt5 ocasionallly interfere each other.(especially for building pyqt)
so only one qt should be installd.

ibus and fcitx may be kinds of imes.
They are diehard and never go well with other imes.
so only one ime should be installd.

I have lubuntu on virtualbox for qt4 and qt5 repectedly.
       mint on virtual box for qt5.
       and crunchbang (upgraded jessey) for qt4

Now I have installed fedora21 on virtualbox for qt4 testing.
I installed qt4.8.6.
builded haven untouched.
nounderline for target input , and dashline for normal input.
Korean ime has no caret shown.
Japenese ime do not allow editing preedit string.
Just as expected already.

builded haven patched.
and install ibus-qt.xxx
background color for target input, dotted underline for normal input.
Korean ime caret is shown as I want.
Caret can go between preedit string in Japanese ime.
It results in All as I expected, as I wanted and as I reported.

Now enough for win and linux!

[johnsonj]

Haven qt4.8.6 on lubuntu and fedora.

[johnsonj]

Haven with Qt4.8.6 on Windows.

[johnsonj]

haven with Qt4.8.6 on fedora21

[johnsonj]

Haven with Qt5.3 on lubuntu

[Neil Hodgson]

johnsonj:

TentiveUndo() eats up

        case Qt::ImFont:
        case Qt::ImCursorPosition:
        case Qt::ImSurroundingText:
        case Qt::ImCurrentSelection:

   ImSurroundingText and ImCursorPosition occur on Linux (Fedora 19 ibus) with Qt 4.8 Korean.

   There is an outstanding request to support surrounding text on GTK+ so it shouldn’t be removed on Qt.

https://sourceforge.net/p/scintilla/feature-requests/1066/

   Neil

[Neil Hodgson]

   While I haven’t tested it thoroughly on OS X, it appears to work. Here is a picture of the IME with Japanese input on OS X.

http://scintilla.org/IMEQtOSX.png

   Neil

[johnsonj]

How beautiful it is?
Thank you!
I think "accented input" must work well.

I am struggling with the candidate box position on Qt5.4/winXP.
I guess it may be also true on Qt5.4/windows7 but I am not sure.

[Neil Hodgson]

johnsonj:

> I think "accented input" must work well.

Dead key accents work: that is where you enter the accent key then the main character. Holding down the main character key to see the accent menu doesn’t work either now or before.

There are two places where arrays are dimensioned with [sqt->maxLenInputIME]. The Clang compiler in Qt Creator on OS X 10.10 does not like this.

../ScintillaEditBase/ScintillaEditBase.cpp:435:42: error: variable-sized object may not be initialized
unsigned int imeUnderlineSegment[sqt->maxLenInputIME] = {0};
^~~~~~~~~~~~~~~~~~~
../ScintillaEditBase/ScintillaEditBase.cpp:448:24: error: variable-sized object may not be initialized

int imeCharPos[sqt->maxLenInputIME] = {0};

^~~~~~~~~~~~~~~~~~~

inputMethodQuery receives the ImCurrentSelection query on OS X although I couldn’t find any behaviour that depends on this. inputMethodQuery should be restored to its previous code as there appears to be no reason to remove it and it certainly is active.

Neil

[Neil Hodgson]

johnsonj:

I am struggling with the candidate box position on Qt5.4/winXP.

I guess it may be also true on Qt5.4/windows7 but I am not sure.

   Qt 5.4 Windows 7 appears OK:

http://www.scintilla.org/CandidateBox7.png

   Neil

[johnsonj]

On Qt5/XP, patche ime put Candidate box always on the last position before inputmethodEvent.
While existing ime Candidate box always pops up on the first position of preedit string.
Candidate box should follow caret position.
So, the image do not tell which is which.

Caret has to move more right to figure it out.

[Neil Hodgson]

johnsonj:

On Qt5/XP, patche ime put Candidate box always on the last position before inputmethodEvent.
While existing ime Candidate box always pops up on the first position of preedit string.
Candidate box should follow caret position.
So, the image do not tell which is which.

   Yes, the window is not following the caret on Windows 7 either:

http://www.scintilla.org/CandidateBox7Caret.png

Caret has to move more right to figure it out.

   More of the queries may need to be processed.

http://doc.qt.io/qt-5/qt.html#InputMethodQuery-enum

   Neil

[johnsonj]

Any position related calls (byte or point whatever) within inputMethodEvent() fails
So http://doc.qt.io/qt-5/qt.html#InputMethodQuery-enum does not help.

Currently the best way I think may be to follow the first preedit charcter instead of caret like the existing ime.
-----------------------------------------------------------------------------------------------------------------------------

I happend to type Ctrl+Shift+U to switch among imes in inline ime mode.
I saw "U" with dotted underline which was astonishing to me.
I have never thought ascii characters can come into preedit routine.

I have finally got understand "accented input".
If a  key is pressed a little longer,
IME returns the character into preedit routine with the accompanying candidate box.

Not only accentable characters but also any (even ime) characters can be preeditfied.
So  I'd like to call it "user defined preedit character input", rather than "accented input"

I think TentativeUndo() can handle it like Ctrl+Shift+U.

[johnsonj]

Thank you for your instruction.
I misunderstood "surrounding text".
those are intended to be provided for ime.
"M17n input methods also support surrounding text, consequently, languages such as Thai and IMs such as plain Zhuyin that require this feature are supported through ibus-m17n.

[johnsonj]

It proves calls from inputmethodevent does not fail.
Qt5/win querys before inputmethodevent instead of right after inputmethodevent,
consequently uses states of last inputmethodevent instead of ones of current inputmethodevent.

I am at a loss.

[johnsonj]

On Qt5/win, Qscintilla2 goes a wrong way.
Qscintilla2 totally breaks korean IME.

[johnsonj]

Do not get me wrong. I mean the existing ime breaks.
patched ime not.

[Neil Hodgson]

johnsonj:

> It proves calls from inputmethodevent does not fail.
> Qt5/win querys before inputmethodevent instead of right after inputmethodevent,
> consequently uses states of last inputmethodevent instead of ones of current inputmethodevent.

Should preeditPos be reinitialised at some point? While it is initialised to -1, it never reverts to this so previous values will continue to be used.

I don’t think that the Qt IME code is ready so release 3.5.3 should not have the updated code.

Neil

[johnsonj]

On Qt5/win preeditPos always should be -1 to depend on pos to set candidate position.
It is better to set preeditPos -1 when scintillaqt initialize.
But It is not bad for preeditPos to sit on the place to show Qt5/win bug definitely.
I think there may be no way for scintilla to cope with the bug. But it may be not.
This situation is also true to the existing ime.

[johnsonj]

Qt5.4 on windows xp

[johnsonj]

updateMicroFocus solved it

[Neil Hodgson]

johnsonj:

<imeqt0118.patch>

   To keep all the compilers happy, in IsHangul, the declaration of hangul should be on the line it is set, otherwise there is a value (false) assigned but never read. So:

  bool hangul = (HangulJamo | HangulCompatibleJamo  | HangulSyllable
 | HangulJamoExtendedA | HangulJamoExtendedB);

   or just

  return (HangulJamo | HangulCompatibleJamo  | HangulSyllable
  | HangulJamoExtendedA | HangulJamoExtendedB);

updateMicroFocus solved it

   Looks good on Windows 7/Qt 5.4.

   On OS X, the wrong part of the preedit string is getting the blue background. In this situation, QInputMethodEvent::Cursor is (start:1, length:1) and the  imeUnderlineSegment is {0, 1, 1} which is dividing the string into 2 parts, the first with 1 character and the second with 2. Its the first part that changes with menu selections so should have the blue background to be like Windows.

   Neil

[johnsonj]

I think this shoud work.

[johnsonj]

I am sorry for bothering you.
My marking segment approch exposes weak points.
If imeCaretPos == preeditStrLen on OSX, it draws as target segment.
If imeCaretPos == 0 on other Os, it also treats as target input.
Qt is too capricious.
I can not find a way.

[Neil Hodgson]

johnsonj:

   Yes, Qt is very difficult as it is spread over so many platforms and there are also 2 major versions of Qt to target.

   I have been looking closer at the attributes reported by Qt 5.4 on OS X 10.10 and comparing it to more native code. Here are images of a complex preedit string.

TextEdit, the basic text editor that is included with OS X.

SciTE:

Haven using thin/thick underlines and changed logic:

   Qt only ever reports a maximum of 3 TextFormat segments so it isn’t possible to emulate the native look completely. The TextFormats only include underline information and the difference between the two types is that thin underline segments have colour #3F3F3F and the thick underline segment (the target) has colour #000000. However, when initially entered and there is only 1 segment, the colour is #3F3F3F although native applications show this as a thick underline.

   The logic based on current Hg (not the imeqt* patches) looks like this:

 if (charFormat.underlineStyle() != QTextCharFormat::NoUnderline) {
    // Set temporary indicator for underline style.
    QColor uc = charFormat.underlineColor();
    int style = INDIC_COMPOSITIONTHIN;
    if (uc.lightness() == 0)
        style = INDIC_COMPOSITIONTHICK;
    const int rgb = uc.rgb() & 0xFFFFFF;
    send(SCI_INDICSETSTYLE, INDIC_INPUTMETHOD + indicNum, style);
    send(SCI_INDICSETFORE, INDIC_INPUTMETHOD + indicNum, rgb);
    send(SCI_SETINDICATORCURRENT, INDIC_INPUTMETHOD + indicNum);
    send(SCI_INDICATORFILLRANGE, caretPos, subLength);
    indicNum++;
}

   The thin underline used in the above Haven is a new indicator visual (INDIC_COMPOSITIONTHIN) since INDIC_PLAIN isn’t in quite the right place.

     } else if (style == INDIC_COMPOSITIONTHIN) {
        PRectangle rcComposition(rc.left+1, rcLine.bottom-2, rc.right-1, rcLine.bottom-1);
        surface->FillRectangle(rcComposition, fore);

   By the way, Korean on OS X has word as well as syllable input so this is possible:

 

   Neil

[johnsonj]

"""However, when initially entered and there is only 1 segment, the colour is #3F3F3F although native applications show this as a thick underline."""

That breaks input semantics.
thick underline means target or normal?
That is because TextEdit does not use background color.
background color exposes explicitly this weak point.
I think underline color difference you suggested can handle it.
but Is it sure Qt will always return different underline colors?
Qt even returns caret positions differently.

I bet second item is on OSX, on which Qt returns caret at the end of target segment instead of the start of one.

It is impossible to follow Qt rules.
I think it is better to follow Scintilla ime semantics as other platforms.
"""background for target input and dotted underline for normal input"""

[johnsonj]

If you change SC_INDICATOR_TARGET to SC_INDICATOR_COMPOSITIONTHICK on my patch,
You wll see the same visual effects as TextEdit I guess.

[johnsonj]

Korean IME should show block caret, character by character.
If not, koreans think it as imcompeleteness.

[johnsonj]

adopted my assumption only to Linux.

If Qt returns caret length 0 to hide caret on target input mode as manual says,
My assumption could be adopted simply to all platforms.
Unfortunately not.

[Neil Hodgson]

johnsonj:

> <imeqt0121.patch>

Was this one supposed to work on OS X? It shows from the start of the preedit to the end of the target with dotted underlines and the remainder with background colour. The presence of ‘onLinux’ seems to show it is meant to work everywhere but treat Linux specially. It works OK on Windows.

The new preeditStrOld field worries me: every time state is added there is something more to go wrong when operations occur in an unexpected sequence.

A platform-independent version of IsHangul could be added to CharacterCategory since it looks like it may be needed by multiple platforms with exactly the same behaviour:

diff -r 7a4cd6fc5883 lexlib/CharacterCategory.h
--- a/lexlib/CharacterCategory.h Tue Jan 20 09:31:51 2015 +1100
+++ b/lexlib/CharacterCategory.h Thu Jan 22 11:35:06 2015 +1100
@@ -24,6 +24,8 @@

CharacterCategory CategoriseCharacter(int character);

+bool IsHangul(int character);
+
#ifdef SCI_NAMESPACE
}
#endif
diff -r 7a4cd6fc5883 lexlib/CharacterCategory.cxx
--- a/lexlib/CharacterCategory.cxx Tue Jan 20 09:31:51 2015 +1100
+++ b/lexlib/CharacterCategory.cxx Thu Jan 22 11:35:06 2015 +1100
@@ -3299,6 +3299,18 @@
return static_cast<CharacterCategory>(*(placeAfter-1) & maskCategory);
}

+bool IsHangul(int character) {
+ // Korean character ranges used for preedit chars.
+ // http://www.programminginkorean.com/programming/hangul-in-unicode/
+ const bool HangulJamo = (0x1100 <= character && character <= 0x11FF);
+ const bool HangulCompatibleJamo = (0x3130 <= character && character <= 0x318F);
+ const bool HangulJamoExtendedA = (0xA960 <= character && character <= 0xA97F);
+ const bool HangulJamoExtendedB = (0xD7B0 <= character && character <= 0xD7FF);
+ const bool HangulSyllable = (0xAC00 <= character && character <= 0xD7A3);
+ return HangulJamo || HangulCompatibleJamo || HangulSyllable ||
+ HangulJamoExtendedA || HangulJamoExtendedB;
+}
+
#ifdef SCI_NAMESPACE
}
#endif


Neil

[johnsonj]

Separating linux ime from others helps understanding.

[johnsonj]

Sorry, I made a logical error.
Pardon, I am confused due to a lot of environments

"""The presence of ‘onLinux’ seems to show it is meant to work everywhere but treat Linux specially"

Yes, ime on Qt/Linux is totally buggy as of now. it should be treated specially and removed easily when fixed.
I am forced to use preporcessor macro to separate it clearly.

"""The new preeditStrOld field worries me"""

Can I use preeditString?

"""A platform-independent version of IsHangul could be added""

No, IsHangul is intended for temporally use.
So the name is not KoreanIME.
We have unicode category.

[johnsonj]

Tidy up for easy reading.

[johnsonj]

+    QString preeditStrOld;

suprisingly the above variable I added compromise Windows XP to die slowly, showing heap memory error.
as you worried.

Why does not the existing preeditString cause error?

[johnsonj]

        preeditString = preeditStr;


Is it bad allocation?
If so, how can I save QString for later use?

[Neil Hodgson]

johnsonj:

> + QString preeditStrOld;
>
> suprisingly the above variable I added compromise Windows XP to die slowly, showing heap memory error.
> as you worried.

This is looking confusing with three variables: preeditString, preeditStrOld, and preeditStr. QString should be quite safe. Its possible that there is some trouble about bounds and writing outside allocations.

Check that qmake has been run and everything has been recompiled so that there is no out-of-date code.

Neil

[johnsonj]

thank you.
I use the existing preeditString only
logic looks more clear

[johnsonj]

patch attached

ime bugs on Linux make us have two schemes.

                   standard rule              marking scheme
linux                   x                                 o
win32                 o                                  o

osx                    o                                  x

[johnsonj]

simplified with mark strategy

[johnsonj]

simplest with marking strategy.

[johnsonj]

Here is my assumption reported,
though I am not sure whether informations of osx are correct or not
-------------------------------------------------------------------------
                * underline style for normal input
                  win             linux                   osx
qt4            dash             single                single
qt5            dash             dash                 single
-------------------------------------------------------------------------
                * underline style for target input
                  win             linux                   osx
qt4             no               single                  no
qt5             no               dash                   no
--------------------------------------------------------------------------
                * caret length  when target input mode (hide caret)
                  win             linux                   osx
qt4              0                    1                     0
qt5              0                    1                     0
--------------------------------------------------------------------------
                * caret position  when target input mode
                   win               linux                   osx
qt4            attr.start       attr.start       attr.start+attr.length
qt5            attr.start       attr.start       attr.start+attr.length
--------------------------------------------------------------------------
* Current state of Qt on linux is buggy due to ibus-qt.
  it shares one underline style with normal input and target input.
  it returns caret length==1 although target input mode.
  I think this situation should be fixed sooner or later

[Neil Hodgson]

johnsonj:

> --------------------------------------------------------------------------
> * caret length when target input mode (hide caret)
> win linux osx
> qt4 0 1 0
> qt5 0 1 0
> --------------------------------------------------------------------------
> * caret position when target input mode
> win linux osx
> qt4 attr.start attr.start attr.start+attr.length
> qt5 attr.start attr.start attr.start+attr.length

On OS X, attr.length appears to always be 1.

Neil

[johnsonj]

Thank you for your tips.
You are right.
I was not careful to the images you showed.
So I guess imeqt0123.patch or imeqt0123-2 should work.

here is informations of qt ime coreccted
-------------------------------------------------------------------------

               * underline style for normal input
                  win             linux                   osx
qt4            dash             single                single
qt5            dash             dash                 single
-------------------------------------------------------------------------
                * underline style for target input
                  win             linux                   osx
qt4             no               single                  no
qt5             no               dash                   no

--------------------------------------------------------------------------
                * caret length  when target input mode (hide caret)
                  win             linux                   osx

qt4              0                    1                     1
qt5              0                    1                     1

--------------------------------------------------------------------------
                * caret position  when target input mode
                   win               linux                   osx
qt4            attr.start       attr.start       attr.start+attr.length
qt5            attr.start       attr.start       attr.start+attr.length

--------------------------------------------------------------------------
* Current state of Qt on linux is buggy due to ibus-qt4 and fcitx-qt5.

  it shares one underline style with normal input and target input.

[Neil Hodgson]

johnsonj:

> So I guess imeqt0123.patch or imeqt0123-2 should work.

imeqt0123-2 has been working for me and the code appears simpler than imeqt0123.

Do you think it is ready for committing?

Neil

[johnsonj]

No not yet.
If both of two work, I prefer imeqt123.patch.
Qt/linux codes are destined to be removed.
Current ime state on Qt/linux is abnormal.
I need more time to test them.
I will submit it again with names refined.

[johnsonj]

OSX may not accept marking starategy.
imeqt123-2.patch assumes that attr.len is 0 when target input mode.
but it proves not.
If one segment and imeCaretPos == preedtStrLen,
it do not tell it is for target or normal
Normal typing do not show problem.
Press space when one segment, and you will always see normal underlines.

WINDOWS and OSX seems to work with standard starategy.
LINUX seems to work with marking strategy.

                   standard          marking
WINDOWS        O                    O
OSX                O                    X
LINUX              X                    O

It has to treat LINUX exceptionlly since the ime situations are very abnormal.
So thinking it to be removed when the bug be fixed,
I submit this patch based on imeqt123.patch.

[johnsonj]

Current state of IMEs for Qt on LINUX
            * standard method
              uim          ibux       fcitx
qt4         o                x             x
qt5         x                 x             x

            * marking method
              uim          ibux       fcitx
qt4         o                o            o
qt5         x                 x            o

* It shows Qt has no problem.
* ibus(fcitx) causes a problem.
* It shows only fcitx is ready for Qt5.

Qt IME on linux is in need to be fixed

[johnsonj]

fixed to support unicode over 0xffff.
started again with latest git source.

only tested with Ctrl+Shift+u.
no unicode font, no input method.
sorry.

[johnsonj]

U+1d11e can be input through ime with Ctrl+shift+u

[johnsonj]

to keep pace with windows for emomji input.

[johnsonj]

tidy up.
naming, indentation....
almost at last.

[Neil Hodgson]

johnsonj:

> tidy up.
> naming, indentation....
> almost at last.

There are some 64/32-bit issues on 64-bit Windows: QString uses int (32-bits) for string indexes where std::string uses size_t (64-bit). ucWidth and the i loop variable just before can be unsigned int which matches the commitStrLen and preeditStrLen which i is compared to.

There are now 2 maxLenInputIME symbols, one an enum in ScintillaBase and one a preprocessor define in ScintillaEditBase. To prevent confusion and follow the convention, the preprocessor version and all uses in ScintillaEditBase should be in upper case MAXLENINPUTIME.

IsHangul doesn’t use ‘this’ so should be static.

I haven’t seen any bugs on the platforms I have tried: Qt 5.4 on Windows or OS X.

Neil

[johnsonj]

Thank you for your instructions.
changed all but,
compiler says 'static' can not be set in member function in file level.

[johnsonj]

ScintillaEditBase.h.
static bool IsHangul(const QChar qchar);

[Neil Hodgson]

johnsonj:

> ScintillaEditBase.h.
> static bool IsHangul(const QChar qchar);

OK. Do you think this version should be committed?

Neil

[johnsonj]

absolutely yes.

[Neil Hodgson]

   Committed with some spelling corrected in comments:

http://sourceforge.net/p/scintilla/code/ci/ba747b6d4ea15d5a1a6cf01726df86eb96ebfd8f/

   Neil

[johnsonj]

Thank you for your hard works and kind instructions.
Thank you for TentativeUndo().
I was a very long journey.