RECONVERT STRING on Qt

[johnsonj]

inputMethodQuery method appears not be called when arrow keys type in.
so It works only after when a key types in.

There is likely to put 'update method' somewhere I can not find.

[johnsonj]

I managed to make Qt answer RECONVERT STRING.
Overstrike will be handled under AddCharUTF() like this adding one variable.

if (InOvertrike && allowOverstrike) { }

[johnsonj]

tested on lubuntu 15.04 qt5.4 with fcitx.

It will not work on windows XP qt5.4.0 with japanese ime 2002 and 2010.

[johnsonj]

demonstration:

https://www.youtube.com/watch?v=jP7OrAtUaWA

[johnsonj]

It works cool!

[johnsonj]

removed support for overstrike.
extended surrounding to paragraph.
supported multicarets.

[johnsonj]

strimlined

[johnsonj]

The existing code returned byte length which is not with QString.
-            return pos - paraStart;

Now Qstring length appears to perform reconversion.
The existing codes most are restored.

[johnsonj]

presume main ranges map to:

     dwCompStr -> win
     between Qt::ImCursorPosition and Qt::ImAnchorPosition -> Qt
     primary selection -> gtk

gtk selection event appears to be equal to Qt::ImCurrentSelection:
To my suprise, It turend out Qt::ImCurrentSelection is not related with Reconversion.
what Qt::ImCurrentSelection: is for?

[johnsonj]

demonstration.
1. reconversion when composing on Korean IME.
2. recnversion without selection on Korean IME.
3. reconversion in overstrike mode on Japanese IME

overstriked japanese with hangul to show that overstrike mode is on.
https://www.youtube.com/watch?v=Il0dYJ3HHBM

[johnsonj]

replaced case Qt::ImSurroundingText:

[johnsonj]

reporting:

I managed to apply ScintillaQt for reconversion.

but, To my surprise,
Qscintilla2 2.9 reconversion enters inputMethodEvent with selection thrown away.
ScintillaQt reconversion code will not work in Qscintilla2.

Even worse, I can not make Qscintilla2 for reconversion by itself in overstrike.
I wonder where Qscintilla2 and ScintillaQt have diverged.

[johnsonj]

Qscintilla2 works for reconversion without updateMicroFocus() specified.
I guess hidden updateMicroFocus works even for Qscintilla2 reconversion key, without being filtered.

[johnsonj]

If microUpdateFocus() be added as same as ScintillaQt,
Qscintilla 2.8 can share reconversion code with ScintillaQt.

Qscintilla 2.9 needs not updateMicroFocus(), but throws away selection and enters reconversion.

so It might be OK to reconcentrate on ScintillaQt.

[johnsonj]

even ScintillaQt have got to enter into reconverstion without selection.
I do not understand where and when it went away off.

[johnsonj]

limit one line without eol.
since eol can not be overstriked.

assume LTR selection.
I think this assumption is correct.
now can get consistent positons.

[johnsonj]

found an elegant way to updateMicroFocus() from QScintilla2.91

[johnsonj]

removed connection warning.
   add updateUi() and updateMicroFocus() to ScintillaQt.

[johnsonj]

for a review.
integrated into tentativeUndo system.
streamlined.

[johnsonj]

for the recording for who are interested.
A very important thing to not is to handle backspace when composing.
Normally it comes with GCS_COMPSTR message on windows, preedit-changed signal on gtk.
but it just comes with inputMethodEvent on Qt, which can be notified.

Here three if statements do not catch the backspace event.

if (event->replacementLength() != 0)
if (!event->commitString().isEmpty())
if (!event->preeditString().isEmpty()) {

so  if (sqt->pdoc->TentativeActive()) statement shoud be put on top,
as far as any solution can not be found.

[johnsonj]

modest version following your instruction.
suppressed reconversion multiline in overstrike mode.

[johnsonj]

There appears to bo no variations left to try.
so let me sum up.

* About Reconversion one CJK ime.
=====================================================================================================
1. feature      win32                        gtk                    qt
-----------------------------------------------------------------------------------------------------
selection       preeditfied                  preeditfied            preeditfied
no selection    surrounding-preeditfied      surrounding-replaced   surrounding-replaced
   

2. variations             if overstrike    if RTLselection      if multiline       tested    note
------------------------------------------------------------------------------------------------------
1st allowOverstrike            X                 X                 X                  O
2nd oneLine                    O                 O                 O                  O
3rd selection-oriented         X                 X                 X                  X       1
4th paragraph                  O                 O                 O                  O       2

note 1:  There is ofen a case in which gtk does not answer selection.
note 2:  eol can not be overstriked. so multiline in overstrike mode breaks.

3. my opinion
The point in variations is in allowOverstrike.
I find myself prefer initial 'allowOverstrike' version.
It does not need three if statements. so simple.
and more important thing is that it does not disorder user experience.
phrase input in overstrike shrinks and enlarges to overstrke next phrase.
That does not just break characters but breaks meaning of next pharase.
I think no japanese want next pharase to be overstriked to meaningless.

Otherwise 4th paragraph version may be the second choice.
It allows paragraph for surrounding replaced, line for selection preeditfied.
I think it is better than 2nd oneline version.

[johnsonj]

continued:
          winXP    lubuntu15.04    note
-----------------------------------------
win32       O           *           1
gtk         *           O           2
qt4         X           O           3
qt5         X           O           4

note 1: perfect
note 2: muti selection at the same time has a problem.
note 3: only fcitx works.
note 4: only first one character preeditfied.

[Neil Hodgson]

johnsonj:

> modest version following your instruction.
> suppressed reconversion multiline in overstrike mode.

…
> <rc-qt-0904.patch>

SCN_UPDATEUI is an application event which should be able to change in the future to meet application needs such as filtering the updates to only those wanted by the application. Calling updateMicroFocus here places unreasonable constraints on this feature. Its unclear just what updates are needed by the IME code.

Assuming that having a replacement length means there is no preedit or commit string looks like a bad bet to me. On OS X, the native IME code has both replacing and new text in one call although this doesn’t seem to happen in Qt currently.
https://developer.apple.com/library/mac/documentation/Cocoa/Reference/NSTextInputClient_Protocol/#//apple_ref/occ/intfm/NSTextInputClient/setMarkedText:selectedRange:replacementRange:

As with GTK+, not responding to platform queries in overstrike mode may lead to more problems.

Since overstrike is complicating reconversion on GTK+ and Qt, its simpler to disallow reconversion in overstrike on these platforms. It is only on Win32 where reconversion is expected that some complexity should be allowed.

Neil

[johnsonj]

"""   SCN_UPDATEUI is an application event which should be able to change in the future
to meet application needs such as filtering the updates to only those wanted by the
application. Calling updateMicroFocus here places unreasonable constraints on this
feature. Its unclear just what updates are needed by the IME code."""

Yes, I am with you. I am trying to find an elegant way.

"""   Assuming that having a replacement length means there is no preedit or commit
string looks like a bad bet to me. On OS X, the native IME code has both replacing and
new text in one call although this doesn’t seem to happen in Qt currently."""

This will never happen in Qt. Event if it will change, scintilla also can chage.
one "return" statement will give a good reading to programmers.

"""   As with GTK+, not responding to platform queries in overstrike mode may lead to more problems."""

Yes, you are right. I would follow you.

"""   Since overstrike is complicating reconversion on GTK+ and Qt, its simpler to disallow
reconversion in overstrike on these platforms. It is only on Win32 where reconversion is
expected that some complexity should be allowed. """

No. no restrict on overstrike.
If you disallow "allowOverstrike", we have only one option left.
Limiting to one line without eol as in win32 should solve all sorts of problem.

[johnsonj]

one line without eol for reconversion.
paragraphs for replacement.

[johnsonj]

limit one line without eol according to kateviewinternal.cpp.
it works cool!.
==================================================================================================
QVariant KateViewInternal::inputMethodQuery ( Qt::InputMethodQuery query ) const
{
  switch (query) {
    case Qt::ImMicroFocus: {
      // Cursor placement code is changed for Asian input method that
      // shows candidate window. This behavior is same as Qt/E 2.3.7
      // which supports Asian input methods. Asian input methods need
      // start point of IM selection text to place candidate window as
      // adjacent to the selection text.
      KTextEditor::Cursor c = m_imPreeditRange ? m_imPreeditRange->start() : m_cursor;
      return QRect (cursorToCoordinate(c, true, false), QSize(0, renderer()->lineHeight()));
    }

    case Qt::ImFont:
      return renderer()->currentFont();

    case Qt::ImCursorPosition:
      return m_imPreeditRange ? m_imPreeditRange->start().column() : 0;

    case Qt::ImSurroundingText:
      if (Kate::TextLine l = doc()->kateTextLine(m_cursor.line()))
        return l->string();
      else
        return QString();

    case Qt::ImCurrentSelection:
      if (m_view->selection())
        return m_view->selectionText();
      else
        return QString();
    default:
      /* values: ImMaximumTextLength and ImAnchorPosition */
      break;
  }

  return QWidget::inputMethodQuery(query);
}

[johnsonj]

I do really appreciate for your caring about tiresome IME.
Let me sum up since.

             qt 4.8.6,  qt5.4.X,  gtk2
-----------------------------------------------------
        kor - replacement     jap - reconversion  
ibus           O                      X            
fcitx          O                      O            

- tested japanese - reconversion with anthy.
- tested korean - replacement with ibus-hangul and fcitx-hangul.
so the only option is currently fcitx.

I find reconversion feed should be limited from line start to main selection end on both qt and gtk.
If line end set, the difference lenght (line end - selection end) will cause problems
(especially document with emoji included).

It is OK to allow replacement with paragraphs. but that is involved in eol,
which may cause problems 'count characters among platforms'.
and eol can not be overstiked.
so multi lines in overstike mode should not be allowed for reconversion.

I am not sure, but it seems to work as I expect.

     * reconversion with preedit string        * replace with commit string                   * environment
------------------------------------------------------------------------------------------------------------------------
           RTL        LTR                    no selection      while composing        Overstrike   one line     emoji    
------------------------------------------------------------------------------------------------------------------------
qt         O           O                           O                O                    O           O            O
gtk        O           O                           O                O                    O           O            O

[johnsonj]

demo for reconvert on qt

https://youtu.be/DlcHuhm40i0

[johnsonj]

changed to feed whole one line for reconversion.

[Neil Hodgson]

Since Qt is UTF-16, CountUTF16 may be more correct than CountCharacters for emoji.

Neil

[johnsonj]

I have thought Qt uses CountUTF16() since QString is feeded for reconversion.
I assumed it is like reconversion on windows since Qstring is similar to std::wstring.
My assumption turns out false with emoji introduced.
It turned out Qt reconversion counts characters, not counts Qstring length.

and this is true between GetRelativePosition and GetRelativePositionUTF16.
After a lot of trial and error, Finally I choosed GetRelativePosition()  and CountCharacters()

[johnsonj]

Made a mistake for ommiting some code.

+        if (sqt->pdoc->TentativeActive()) {
+            sqt->pdoc->TentativeUndo();
+        }

[johnsonj]

I have managed to work reconversion of qt4 on Fedora22 with fcitx.

https://youtu.be/gHhviPQbTbM

scim turns out to be hidden behind fcitx.
ibus interferes with fcitx.
They should be removed completely for clean ime works.

[johnsonj]

Demo reconvert of PySide with Qt4 on fedora22.
Lots of warnigs occurred to me while building, but it works well.

https://www.youtube.com/watch?v=0l3EDWebzSM&feature=youtu.be

[Neil Hodgson]

johnsonj:

https://www.youtube.com/watch?v=0l3EDWebzSM&feature=youtu.be

   “This video is private.”

   Neil

[johnsonj]

sorry for that.
I have uploaded it but not posted yet.
I forgot clicking posting button.

[johnsonj]

Surprising to me, UIM has only "retrieve surrounding", no "delete surrounding".
And more it retrieve surrounding only when it receives 'reconversion key', as win32 does.

I decide to test if my understanding is right.

Test environment: scintilla3.5.4, fcitx, Qt4.8.6, lubuntu 15.10

The old ime had an ablity to handle Qt::ImSurroundingText.
but it works only for the first whole line. why?

Points to make sourounding work:
1. sqt->pdoc->CountCharacters(feedStart, pos)
2. int feedEnd = sqt->sel.Empty() ? sqt->pdoc->LineEnd(endLine) : mainEnd;

These points are also true to gtk ime.

following fixes to be considerd for completion:
1. updateMicroFocus() for Qt 5.X
2. RTL selection.


---------------------------------------------------------------------------------------------------------------------------------
void ScintillaEditBase::inputMethodEvent(QInputMethodEvent *event)
{
...
    // Insert the commit string.
    if (!event->commitString().isEmpty() || event->replacementLength()) {
        // Select the text to be removed.
        int commitPos = send(SCI_GETCURRENTPOS);
        int start = sqt->pdoc->GetRelativePosition(commitPos, event->replacementStart());
        int end = sqt->pdoc->GetRelativePosition(start, event->replacementLength());
        sqt->pdoc->DeleteChars(start, end - start);

...
    }
...

QVariant ScintillaEditBase::inputMethodQuery(Qt::InputMethodQuery query) const
{
    int pos = send(SCI_GETCURRENTPOS);
    int line = send(SCI_LINEFROMPOSITION, pos);

    int mainStart = sqt->sel.RangeMain().Start().Position();
    int mainEnd = sqt->sel.RangeMain().End().Position();
    int startLine = sqt->pdoc->LineFromPosition(mainStart);
    int endLine = sqt->pdoc->LineFromPosition(mainEnd);

    switch (query) {
...
        case Qt::ImCursorPosition:
        {
            int feedStart = sqt->pdoc->LineStart(startLine);
            return QVariant(sqt->pdoc->CountCharacters(feedStart, pos));
        }

        case Qt::ImSurroundingText:
        {
            // Limit feed to caret line without eol.
            if (startLine != endLine)
                return QVariant();

            int feedStart = sqt->pdoc->LineStart(startLine);
            int feedEnd = sqt->sel.Empty() ? sqt->pdoc->LineEnd(endLine) : mainEnd;
            std::string s = sqt->RangeText(feedStart, feedEnd);
            return QVariant(sqt->StringFromDocument(&s[0]));
        }

...
    }
}
---------------------------------------------------------------------------------------------------------------------------------

[Neil Hodgson]

johnsonj:

> The old ime had an ablity to handle Qt::ImSurroundingText.
> but it works only for the first whole line. why?

It looks like it should work on paragraphs but possibly the calling IME can’t do that.

> Points to make sourounding work:
> 1. sqt->pdoc->CountCharacters(feedStart, pos)

Sounds reasonable although it should be checked with some emoji in case its really counting UTF-16 code units.

> void ScintillaEditBase::inputMethodEvent(QInputMethodEvent *event)

> {
> ...
> // Insert the commit string.
> if (!event->commitString().isEmpty() || event->replacementLength()) {
> // Select the text to be removed.
> int commitPos = send(SCI_GETCURRENTPOS);
> int start = sqt->pdoc->GetRelativePosition(commitPos, event->replacementStart());
> int end = sqt->pdoc->GetRelativePosition(start, event->replacementLength());
> sqt->pdoc->DeleteChars(start, end - start);

If the text being deleted is the tentative addition from the preedit string there will be conflicts.

> case Qt::ImCursorPosition:
> {
> int feedStart = sqt->pdoc->LineStart(startLine);
> return QVariant(sqt->pdoc->CountCharacters(feedStart, pos));

The Qt documentation talks about the surrounding paragraph which is matched by the current code. Since kate and konsole implement this with the line, that may be what is expected.

Neil

[johnsonj]

Neil:

I just wanted to know why the old scintillaqt ime of 3.5.4 did not work.
That was because it handled surrounding in byte, instead of character.
so it worked for the first line and first input.

and it did not work if the line was not selected to lineEnd.
mainEnd should be set to feedEnd to make selection reconversion work.

That is just all I want to tell you.

"""   Sounds reasonable although it should be checked with some emoji in case its really counting UTF-16 code units."""

emoji make CountCharacters to be used.

"""   If the text being deleted is the tentative addition from the preedit string there will be conflicts."""

No problem. It can be solved anyhow. Ime can tell preedit state.

"""   It looks like it should work on paragraphs but possibly the calling IME can’t do that."""

"""   The Qt documentation talks about the surrounding paragraph which is matched by the current code. Since kate and konsole implement this with the line, that may be what is expected."""

Yes, I find it works well on multilines on qt ime and gtk ime.
gtk document also says same thing.
but I think surrounding should be limited one line without eol like win32.
eol might cause problems across platforms.

int paraStart = sqt->pdoc->ParaUp(pos)
This turnes out to be good working on gtk and qt on linux,
but not sure if it works win32 ime or win32 documents, or docouments with unicode line ends.

Even one line might be too much long over 1024 bytes.
So rather than this
QVarLengthArray<char,1024> buffer(paraEnd - paraStart + 1);

I think this is better
+            std::string s = sqt->RangeText(feedStart, feedEnd);

I think scintillaQt surrouding bugs should be fixed.
If your prefer paragraph version, I would follow you.

[Neil Hodgson]

johnsonj:

Even one line might be too much long over 1024 bytes.
So rather than this
QVarLengthArray<char,1024> buffer(paraEnd - paraStart + 1);

I think this is better
+            std::string s = sqt->RangeText(feedStart, feedEnd);

   QVarLengthArray<char,1024> isn’t really limited to 1024 bytes. What it says is that for allocations up to 1024 bytes, the value is stored on the stack but if greater than 1024 bytes, it is stored on the heap. This is just a performance optimisation since stack allocation is normally faster than heap allocation. In PlatWin.cxx, there is a VarBuffer class for the same purpose.

   It shouldn’t slow things down to use std::string instead of QVarLengthArray since this code is called when the user hits keys, not thousands of times each second.

http://doc.qt.io/qt-4.8/qvarlengtharray.html

   Neil

[johnsonj]

""'In the example above, QVarLengthArray will preallocate 1024 elements on the stack and use them unless n + 1 is greater than 1024."""

Thank you for the tip.

[johnsonj]

Two lines are changed for "retrieve surrounding" and "select reconversion" follwing your intention.
"delete surrounding" is another thing. it is beside the point.
--------------------------------------------------------------------------------------------------------------------------
        case Qt::ImCursorPosition:
        {
            int paraStart = sqt->pdoc->ParaUp(pos);
            return QVariant(sqt->pdoc->CountCharacters(paraStart, pos)); // changed
        }

        case Qt::ImSurroundingText:
        {
            int paraStart = sqt->pdoc->ParaUp(pos);
            int paraEnd = sqt->pdoc->ParaDown(pos);

            QVarLengthArray<char,1024> buffer(paraEnd - paraStart + 1);

            Sci_CharacterRange charRange;
            charRange.cpMin = paraStart;
            charRange.cpMax = sqt->sel.Empty() ? paraEnd : sqt->sel.RangeMain().End().Position(); //changed

            Sci_TextRange textRange;
            textRange.chrg = charRange;
            textRange.lpstrText = buffer.data();

            send(SCI_GETTEXTRANGE, 0, (sptr_t)&textRange);

            return sqt->StringFromDocument(buffer.constData());
        }

[johnsonj]

I find case Qt::ImAnchorPosition to have met my trial and error.
If selection, selMainEnd has an important meaning.
This is exactly as my idea what I experimeted on Gtk before.

* excerpted from http://osxr.org/qt/source/qtbase/src/widgets/widgets/qlineedit.cpp
---------------------------------------------------------------------------------------------
QVariant QLineEdit::inputMethodQuery(Qt::InputMethodQuery property) const
{
    Q_D(const QLineEdit);
    switch(property) {
    case Qt::ImMicroFocus:
        return d->cursorRect();
    case Qt::ImFont:
        return font();
    case Qt::ImCursorPosition:
        return QVariant(d->control->cursor());
    case Qt::ImSurroundingText:
        return QVariant(text());
    case Qt::ImCurrentSelection:
        return QVariant(selectedText());
    case Qt::ImMaximumTextLength:
        return QVariant(maxLength());
    case Qt::ImAnchorPosition:
        if (d->control->selectionStart() == d->control->selectionEnd())
            return QVariant(d->control->cursor());
        else if (d->control->selectionStart() == d->control->cursor())
            return QVariant(d->control->selectionEnd());
        else
            return QVariant(d->control->selectionStart());
    default:
        return QVariant();
    }
}

[Neil Hodgson]

johnsonj:

> Two lines are changed for "retrieve surrounding" and "select reconversion" follwing your intention.

> …

> return QVariant(sqt->pdoc->CountCharacters(paraStart, pos)); // changed

OK, this looks correct.

> charRange.cpMax = sqt->sel.Empty() ? paraEnd : sqt->sel.RangeMain().End().Position(); //changed

This doesn’t look as good to me. Its not what QLineEdit is doing in your next post which is to return the whole text, not the text up to the selection end:

> case Qt::ImSurroundingText:
> return QVariant(text());

Should support ImAnchorPosition but we have direct access to the anchor so shouldn’t be a complex as this:

> case Qt::ImAnchorPosition:
> if (d->control->selectionStart() == d->control->selectionEnd())
> return QVariant(d->control->cursor());
> else if (d->control->selectionStart() == d->control->cursor())
> return QVariant(d->control->selectionEnd());
> else
> return QVariant(d->control->selectionStart());

Does supporting ImAnchorPosition remove the benefit (whatever that is) for capping ImSurroundingText?

Neil

[johnsonj]

introduced Qt::ImAnchorPosition instead of sqt->sel.RangeMain().End().Position()
This works for Qt 4.8.X
=============================================================================

        case Qt::ImCursorPosition:
        {
            int paraStart = sqt->pdoc->ParaUp(pos);

            return QVariant(sqt->pdoc->CountCharacters(paraStart, pos));  // free from position
        }

        case Qt::ImAnchorPosition:

        {
            int paraStart = sqt->pdoc->ParaUp(pos);

            return QVariant(sqt->pdoc->CountCharacters(paraStart, sqt->sel.MainAnchor())); // free from eol.
        }
=============================================================================
but paraStart has two potential problems.
1. ime limits feed length. fcitx and ibus put a limit to feed within 4096 QString length.
2. Paragraphs might be selected. There can be empty lines within LTR selection. so base offeset can be changed.
paragraph or even a line should not be allowed as feed.
Feed should be less 4096 QString characters around caret.

We shoud consider gtk ime.
There is no way for Gtk anchor position to be set.
And the existing scintillQt ime has no Qt::ImAnchorPosition.

from ibusimcontext.c
=================================================================================
static guint
get_selection_anchor_point (IBusIMContext *ibusimcontext,
                            guint cursor_pos,
                            guint surrounding_text_len)
...
    guint anchor;

    if (start_index == cursor_index) {
      anchor = end_index;
    } else if (end_index == cursor_index) {
      anchor = start_index;
    } else {
      return cursor_pos;
    }
...
===============================================================
we see anchor position automatically be found by FCITX roughly as qlineedit.
I am trying to implement scintilaGtk ime without MainSelEnd,
and scintillaQt ime without Qt::ImAnchor.

[Neil Hodgson]

johnsonj:

> but paraStart has two potential problems.
> 1. ime limits feed length. fcitx and ibus put a limit to feed within 4096 QString length.
> 2. Paragraphs might be selected. There can be empty lines within LTR selection. so base offeset can be changed.
> paragraph or even a line should not be allowed as feed.
> Feed should be less 4096 QString characters around caret.

That is an unpleasant set of constraints and then there will be different trade-offs to meet the constraints. Does user code have to deal with all this or will Qt reduce the values given to the IMEs to conform?

Neil

[johnsonj]

adapted to these constraints.
----------------------------------------------------------------------------------------------------------------------

QVariant ScintillaEditBase::inputMethodQuery(Qt::InputMethodQuery query) const
{
    int pos = send(SCI_GETCURRENTPOS);
    int line = send(SCI_LINEFROMPOSITION, pos);

    //            | <---maxLenInputIME--->|<---maxLenInputIME---> |
    // lineStart  |maxStart            |Caret|              maxEnd|    lineEnd
    //            |FeedStart  |Anchor-->Caret|             FeedEnd|
    //            |FeedStart           |Caret<--Anchor|    FeedEnd|

    int lineStart = sqt->pdoc->LineStart(line);
    int lineEnd = sqt->pdoc->LineEnd(line);

    int maxStart = sqt->pdoc->GetRelativePosition(pos, - MAXLENINPUTIME);
    int maxEnd = sqt->pdoc->GetRelativePosition(pos, + MAXLENINPUTIME);

    int feedStart = (maxStart > 0) ? std::max(lineStart, maxStart) : lineStart;
    int feedEnd = (maxEnd > 0) ? std::min(lineEnd, maxEnd) : lineEnd;

    switch (query) {
        case Qt::ImMicroFocus:

...
        case Qt::ImCursorPosition:
        {

            return QVariant(sqt->pdoc->CountCharacters(feedStart, pos));
        }

        case Qt::ImAnchorPosition:
        {
            int anchor = sqt->sel.MainAnchor();
            if ((anchor < feedStart) || (feedEnd < anchor))
                return QVariant(inputMethodQuery(Qt::ImCursorPosition));
            return QVariant(sqt->pdoc->CountCharacters(feedStart, anchor));
        }

        case Qt::ImSurroundingText:
        {
            if (sqt->pdoc->TentativeActive())
                return QVariant(QString());

            QVarLengthArray<char,1024> buffer(feedEnd - feedStart + 1);

            Sci_CharacterRange charRange;
            charRange.cpMin = feedStart;
            charRange.cpMax = feedEnd;

            Sci_TextRange textRange;
            textRange.chrg = charRange;
            textRange.lpstrText = buffer.data();

            send(SCI_GETTEXTRANGE, 0, (sptr_t)&textRange);

            return sqt->StringFromDocument(buffer.constData());
        }

...
        default:
            return QVariant();
    }
}

[johnsonj]

reporting: tested under ibus 1.5.10, lubuntu 15.10, Qt5.4.X with  “Thai – kesmanee (m17n)”

https://www.youtube.com/watch?v=ELhODeIAT90&feature=youtu.be

ScintillaQt turns out to behave differently for reconversion on Qt4.6.8 and Qt5.4.2.
I think the problem is related with updateMicroFocus().
I will give it a try for Qt4.6.8 again.

[johnsonj]

just keep recording.

“Thai – kesmanee (m17n)” works for retrieve-surrouding.
Then why ibus-anthy does not answer retrieve-surrounding?

Ibus-anthy is written by python (surprisingly neither C nor C++).
And I find it does not support surrounding feature inherently.

I remember Ibus-hangul worked for surrounding feature,
but this time I see it will not work for delete-surrounding. why? Is it No fcitx installed?
I guess there might be a possiblity for fcitx to do it on behalf ibus in background.

Nonetheless, I thank ibus.
ibus together with “Thai – kesmanee (m17n)" engine shows my implementation of surrounding feature works for Thai language correctly.

[johnsonj]

just keep recording:

pacth qwindowsinputcontext.cpp.
but not tested since I can not compile it.

Leave this for qt users on win32.
==========================================================================
bool QWindowsInputContext::handleIME_Request(WPARAM wParam,
                                             LPARAM lParam,
                                             LRESULT *result)
{
    switch (int(wParam)) {
    case IMR_RECONVERTSTRING: {
        const int size = reconvertString(reinterpret_cast<RECONVERTSTRING *>(lParam));
        if (size < 0)
            return false;
        *result = size;
        return true;
    }
        break;
    case IMR_CONFIRMRECONVERTSTRING:
        return confirmReconvert(reinterpret_cast<RECONVERTSTRING *>(lParam));
    default:
        break;
    }
    return false;
}

/*!
    \brief Determines the string for reconversion with selection.

    This is triggered twice by WM_IME_REQUEST, first with reconv=0
    to determine the length and later with a reconv struct to obtain
    the string with the position of the selection to be reconverted.

    Obtains the text from the focus object and marks the word
    for selection (might not be entirely correct for Japanese).
*/

int QWindowsInputContext::reconvertString(RECONVERTSTRING *reconv)
{
    QObject *fo = qApp->focusObject();
    if (!fo)
        return 0;

    QString surroundingText;
    if (!inputMethodQuery(fo, Qt::ImSurroundingText, &surroundingText))
        return -1;
    const DWORD memSize = sizeof(RECONVERTSTRING)
            + (surroundingText.length() + 1) * sizeof(ushort);
    qCDebug(lcQpaInputMethods) << __FUNCTION__ << " reconv=" << reconv
        << " surroundingText=" << surroundingText << " size=" << memSize;
    // If memory is not allocated, return the required size.
    if (!reconv)
        return surroundingText.isEmpty() ? -1 : int(memSize);

    QString selectedText;
    inputMethodQuery(fo, Qt::ImCurrentSelection, &selectedText));
    int selStart = surroundingText.indexOf(selectedText);

    // No selection asks IME to fill target fields with its own value.
    reconv->dwSize = memSize;
    reconv->dwVersion = 0;

    reconv->dwStrLen = surroundingText.size();
    reconv->dwStrOffset = sizeof(RECONVERTSTRING);
    reconv->dwCompStrLen = selectedText.size(); // TCHAR count.
    reconv->dwCompStrOffset = selStart * sizeof(ushort); // byte count.
    reconv->dwTargetStrLen = reconv->dwCompStrLen;
    reconv->dwTargetStrOffset = reconv->dwCompStrOffset;

    ushort *pastReconv = reinterpret_cast<ushort *>(reconv + 1);
    std::copy(surroundingText.utf16(), surroundingText.utf16() + surroundingText.size(),
              pastReconv);

    ImmReleaseContext(hwnd, himc);
    return memSize;
}

bool QWindowsInputContext::confirmReconvert(RECONVERTSTRING *reconv)
{
    ushort *reconvUTF16;
    std::copy(reinterpret_cast<ushort *>(reconv + 1), reconv->dwStrLen * sizeof(ushort), reconvUTF16);
    QString QreconvText.fromUtf16(reconvUTF16);

    QString targetStart = QreconvText.mid(0, reconv->dwTargetStrOffset / sizeof(ushort));
    QString target = QreconvText.mid(tgWstart, reconv->dwTargetStrLen);

    QVector<uint> targetStartByChar = targetStart.toUcs4();
    QVector<uint> targetByChar = target.toUcs4();

    QObject *fo = qApp->focusObject();
    if (!fo)
        return false;
    int reconvCaretPos = 0;
    inputMethodQuery(fo, Qt::ImCursorPosition, &reconvCaretPos);

    int offset = targetStartByChar.size() - reconvCaretPos;
    int nchars = targetByChar.size();

    QInputMethodEvent deleteSurroundingEvent;
    deleteSurroundingEvent.setCommitString("", offset, nchars);
    QCoreApplication::sendEvent(fo, &deleteSurroundingEvent);

    return true;
}

[johnsonj]

compile succeeded, but build not succeded.
Haven will not work on my patched Qt.

from Qt\4.8.6\src\gui\inputmethod\qinputcontext_win.cpp
===========================================================================
int QWinInputContext::reconvertString(RECONVERTSTRING *reconv)
{
    QWidget *w = focusWidget();
    if(!w)
        return -1;

    Q_ASSERT(w->testAttribute(Qt::WA_WState_Created));
    QString surroundingText = qvariant_cast<QString>(w->inputMethodQuery(Qt::ImSurroundingText));
    int memSize = sizeof(RECONVERTSTRING)+(surroundingText.length()+1)*sizeof(ushort);

    // If memory is not allocated, return the required size.

    if (!reconv) {
        if (surroundingText.isEmpty())
            return -1;
        else
            return memSize;
    }

    QString selectedText = qvariant_cast<QString>(w->inputMethodQuery(Qt::ImCurrentSelection));
    int selStart = surroundingText.indexOf(selectedText);

    reconv->dwSize = memSize;
    reconv->dwVersion = 0;

    reconv->dwStrLen = surroundingText.length();

    reconv->dwStrOffset = sizeof(RECONVERTSTRING);
    reconv->dwCompStrLen = selectedText.size();

    reconv->dwCompStrOffset = selStart * sizeof(ushort);

    reconv->dwTargetStrLen = reconv->dwCompStrLen;
    reconv->dwTargetStrOffset = reconv->dwCompStrOffset;

    memcpy((char*)(reconv+1), surroundingText.utf16(), surroundingText.length()*sizeof(ushort));
    
    HIMC imc = getContext(w->effectiveWinId());
    if (!imc)
        return -1;

    if (!::ImmSetCompositionStringW(imc, SCS_QUERYRECONVERTSTRING, reconv, memSize, NULL, 0))
        return -1;

    QString targetStart = surroundingText.mid(0, reconv->dwTargetStrOffset / sizeof(ushort));
    QString target = surroundingText.mid(targetStart.size(), reconv->dwTargetStrLen);

    QVector<uint> targetStartByChar = targetStart.toUcs4();
    QVector<uint> targetByChar = target.toUcs4();

    int surroundingTextCaretPos = qvariant_cast<int>(w->inputMethodQuery(Qt::ImCursorPosition));

    int offset = targetStartByChar.size() - surroundingTextCaretPos;

    int nchars = targetByChar.size();

    QInputMethodEvent deleteSurroundingEvent;
    deleteSurroundingEvent.setCommitString("", offset, nchars);

    qt_sendSpontaneousEvent(w, &deleteSurroundingEvent);

    releaseContext(w->effectiveWinId(), imc);
    return memSize;
}

[johnsonj]

I fixed Qt ime on windwos.

DEMO QTextEdit Qt4.8.6 WinXP
https://www.youtube.com/watch?v=ImEuDyLWlmQ

DEMO : Haven Qt4.8.6 WinXP
https://www.youtube.com/watch?v=KYKWmn3neLw

[johnsonj]

ScintillaWin::ImeOnReconvert() is ported into
                     src/plugins/platform/window/qwindowsinputcontext.cpp for Qt5.4.2
================================================================================

int QWindowsInputContext::reconvertString(RECONVERTSTRING *reconv)
{
    QObject *fo = qApp->focusObject();
    if (!fo)

        return false;

    const QVariant surroundingTextV = QInputMethod::queryFocusObject(Qt::ImSurroundingText, QVariant());
    if (!surroundingTextV.isValid())
        return -1;
    const QString surroundingText = surroundingTextV.toString();

    const DWORD memSize = sizeof(RECONVERTSTRING)
            + (surroundingText.length() + 1) * sizeof(ushort);
    qCDebug(lcQpaInputMethods) << __FUNCTION__ << " reconv=" << reconv
        << " surroundingText=" << surroundingText << " size=" << memSize;

    // If memory is not allocated, return the required size.
    if (!reconv)

        return memSize;

    QVariant selectedTextV = QInputMethod::queryFocusObject(Qt::ImCurrentSelection, QVariant());
    QString selectedText = selectedTextV.toString();
    const QVariant posV = QInputMethod::queryFocusObject(Qt::ImCursorPosition, QVariant());
    const int surroundingCaret = posV.isValid() ? posV.toInt() : 0;

    // Map selection to dwCompStr.
    // No selection assumes current caret as selectedText without length.
    const int selectedStart = surroundingCaret;
    if (!selectedText.isEmpty())
        selectedStart = surroundingText.indexOf(selectedText);

    if (selectedStart < 0)
        return -1;

    reconv->dwSize = memSize;
    reconv->dwVersion = 0;

    reconv->dwStrLen = surroundingText.size();
    reconv->dwStrOffset = sizeof(RECONVERTSTRING);

    reconv->dwCompStrLen = selectedText.size(); // TCHAR count.

    reconv->dwCompStrOffset = selectedStart * sizeof(ushort); // byte count.

    reconv->dwTargetStrLen = reconv->dwCompStrLen;
    reconv->dwTargetStrOffset = reconv->dwCompStrOffset;

    ushort *pastReconv = reinterpret_cast<ushort *>(reconv + 1);
    std::copy(surroundingText.utf16(), surroundingText.utf16() + surroundingText.size(),
              pastReconv);

    if (!selectedText.isEmpty()) // Selection does not need setCommitString.
        return memSize; // Composition String for selectedText returns through WM_IME_COMPOSITION.

    // From here, IMR_CONFIRMRECONVERTSTRING equvalent.
    // No selection asks win ime to fill target fields automatically with its own value.
    const HIMC himc = ImmGetContext(hwnd);
    if (!himc)
        return -1;

    if (!::ImmSetCompositionStringW(imc, SCS_QUERYRECONVERTSTRING, reconv, memSize, NULL, 0)) {
        ImmReleaseContext(hwnd, himc);
        return -1;
    }

    // Calculate by UTF16 length for WIN32 IME.
    int targetStart = reconv->dwTargetStrOffset / sizeof(ushort);
    int targetWidth = reconv->dwTargetStrLen;
    int offset = surroundingCaret - targetStart;

    // Calculate by character length for Qt IME.
    int offsetByChar = surroundingText.mid(targetStart, offset).toUcs4().size();
    int charsToDelete = surroundingText.mid(targetStart, targetWidth).toUcs4().size();

    // replacementStart(): -offsetByChar, replacementLength(): charsToDelete;
    QInputMethodEvent deleteSurroundingEvent;
    deleteSurroundingEvent.setCommitString(QString(), -offsetByChar, ncharsToDelete);
    QCoreApplication::sendEvent(fo, &deleteSurroundingEvent);

    // Comp/Result String for target returns through WM_IME_COMPOSITION.
    ImmReleaseContext(hwnd, himc);
    return true;
}

[johnsonj]

I find reconversion on Qt for win uses QString length.
reconversion on Qt for linux uses Character length.
There is emoji between them.

I am embarassed!!!

[johnsonj]

Qt ime components should use character length from document:

===========================================================================================
int QInputMethodEvent::replacementLength() const

Returns the number of """characters""" to be replaced in the preedit string.

See also replacementStart() and setCommitString().
int QInputMethodEvent::replacementStart() const

Returns the position at which """characters"""" are to be replaced relative from the start of the preedit string.

See also replacementLength() and setCommitString().
void QInputMethodEvent::setCommitString(const QString & commitString, int replaceFrom = 0, int replaceLength = 0)
============================================================================================

But QTextEdit expects UTF16 length.

so its characters are broken on linux, while it works well on windows.

since currently qt ime for win returns UTF16 length.

Let me put off this thread for the time being.

[johnsonj]

It is very confusing.
"""character""" should be replaced with "QChar", which is never character.
============================================================================================
int QString::size() const
Returns the number of """characters""" in this string.
============================================================================================

NOW, which is right for Qt reconversion? UTF16 or Character.
Which does Qt ime use?

I find Qt ime uses "character" as "QChar", which is counted by UTF16.

but linux word appears to see "character" on Qt as is, which is counted by UCS4.

I am very confused.

I can not understand this situation!

[Neil Hodgson]

johnsonj:

> NOW, which is right for Qt reconversion? UTF16 or Character.
> Which does Qt ime use?
>
> I find Qt ime uses "character" as "QChar", which is counted by UTF16.
> but linux word appears to see "character" on Qt as is, which is counted by UCS4.

There are many UTF-16 versus character bugs in the world.

There may be a Qt mailing list or other forum for discussing this. If not, then there is the Qt bug tracker. I have used the tracker and it is monitored and may produce a fix. For maximum effectiveness, reports on the tracker should be thoroughly researched first with a reproducible case.

A bug I reported:
https://bugreports.qt.io/browse/QTBUG-19483

Neil

[johnsonj]

Thank you!
I do really appreciate you for giving valuable instructions.
I implemented Qt reconversion for windows with what I have learned from scintilla-interest group so far.

Currently reconverion on ScintillaWin tries to goes into Qt ime for windows.

https://bugreports.qt.io/browse/QTBUG-50791
https://bugreports.qt.io/browse/QTBUG-50826

[johnsonj]

I hope Fcitx-anthy may fix feedEndLimit problem.

https://groups.google.com/forum/#!topic/fcitx/o4xCbx-wSp0

[johnsonj]

reporting:
Emoji exposes an unnoticed miscommunication between Qt and fcitx.

[qt] fix inconsistency when handling qstring and ucs4 string
https://github.com/fcitx/fcitx/commit/14faccfbb0d87e06c25d182ae842808d18be3dc7

[qt5] Fix inconsistency between qstring and ucs4 string when handle surrounding
https://github.com/fcitx/fcitx-qt5/commit/31ecc9f2f9c8eb77082044944bbb6740d35ae7c3

[johnsonj]

Fcitx changed to use QChar length for setCommitString.

+ nchar = QString::fromUcs4(ucsText.mid(cursor + offset, nchar).data()).size()
+ offset = QString::fromUcs4(ucsText.mid(start, len).data()).size() * (offset >= 0 ? 1 : -1);

+ event.setCommitString("", offset, nchar);

very embarassing!
I have thought offset and nchar mean characters.
Surrounding feature for ScintillaQt likely have to wait for next fcitx release.

[johnsonj]

adapted to what has been found so far.

[johnsonj]

-    if (event->commitString().isNull() && event->preeditString().isNull()) {
+    if (event->commitString().isEmpty() && event->preeditString().isEmpty()) {

Tested on lubuntu15.10 without emoji.

[johnsonj]

retrieve surroudding for the old scintillqt 3.5.2
-------------------------------------------------------------------------------------------------

void ScintillaEditBase::inputMethodEvent(QInputMethodEvent *event)
{
    // Clear the current selection.
    sqt->ClearSelection();
    if (preeditPos >= 0)
        sqt->SetSelection(preeditPos, preeditPos);

    // Insert the commit string.
    if (!event->commitString().isEmpty() || event->replacementLength()) {
        // Select the text to be removed.
        int commitPos = send(SCI_GETCURRENTPOS);
        int start = sqt->pdoc->GetRelativePosition(commitPos, event->replacementStart());
        int end = sqt->pdoc->GetRelativePosition(start, event->replacementLength());
        sqt->SetSelection(start, end);

        // Replace the selection with the commit string.
        QByteArray commitBytes = sqt->BytesForDocument(event->commitString());
        char *commitData = commitBytes.data();
        sqt->AddCharUTF(commitData, static_cast<unsigned int>(strlen(commitData)));
    }

    // Select the previous preedit string.
    int pos = send(SCI_GETCURRENTPOS);
    int length = sqt->BytesForDocument(preeditString).length();
    sqt->SetSelection(pos, pos + length);

    // Replace the selection with the new preedit string.
    QByteArray bytes = sqt->BytesForDocument(event->preeditString());
    char *data = bytes.data();
    bool recording = sqt->recordingMacro;
    sqt->recordingMacro = false;
    send(SCI_SETUNDOCOLLECTION, false);
    sqt->AddCharUTF(data, static_cast<unsigned int>(strlen(data)));
    send(SCI_SETUNDOCOLLECTION, true);
    sqt->recordingMacro = recording;
    sqt->SetSelection(pos, pos);

    // Store the state of the current preedit string.
    preeditString = event->preeditString();
    preeditPos = !preeditString.isEmpty() ? send(SCI_GETCURRENTPOS) : -1;

    if (!preeditString.isEmpty()) {
        // Apply attributes to the preedit string.
        int indicNum = 0;
        sqt->ShowCaretAtCurrentPosition();
        foreach (QInputMethodEvent::Attribute a, event->attributes()) {
            QString prefix = preeditString.left(a.start);
            QByteArray prefixBytes = sqt->BytesForDocument(prefix);
            int prefixLength = prefixBytes.length();
            int caretPos = preeditPos + prefixLength;

            if (a.type == QInputMethodEvent::Cursor) {
                sqt->SetSelection(caretPos, caretPos);
                if (!a.length)
                    sqt->DropCaret();

            } else if (a.type == QInputMethodEvent::TextFormat) {
                Q_ASSERT(a.value.canConvert(QVariant::TextFormat));
                QTextFormat format = a.value.value<QTextFormat>();
                Q_ASSERT(format.isCharFormat());
                QTextCharFormat charFormat = format.toCharFormat();

                QString sub = preeditString.mid(a.start, a.length);
                QByteArray subBytes = sqt->BytesForDocument(sub);
                int subLength = subBytes.length();

                if (charFormat.underlineStyle() != QTextCharFormat::NoUnderline) {
                    // Set temporary indicator for underline style.
                    QColor uc = charFormat.underlineColor();
                    int style = INDIC_PLAIN;
                    if (charFormat.underlineStyle() == QTextCharFormat::DashUnderline)
                        style = INDIC_DASH;
                    send(SCI_INDICSETSTYLE, INDIC_INPUTMETHOD + indicNum, style);
                    send(SCI_INDICSETFORE, INDIC_INPUTMETHOD + indicNum, uc.rgb());
                    send(SCI_SETINDICATORCURRENT, INDIC_INPUTMETHOD + indicNum);
                    send(SCI_INDICATORFILLRANGE, caretPos, subLength);
                    indicNum++;

                }
            }
        }
    }
}

QVariant ScintillaEditBase::inputMethodQuery(Qt::InputMethodQuery query) const
{
    int pos = send(SCI_GETCURRENTPOS);
    int line = send(SCI_LINEFROMPOSITION, pos);

    int lineStart = sqt->pdoc->LineStart(line);
    int lineEnd = sqt->pdoc->LineEnd(line);
    int maxStart = sqt->pdoc->GetRelativePosition(pos, - MAXLENINPUTIME);
    int maxEnd = sqt->pdoc->GetRelativePosition(pos, + MAXLENINPUTIME);

    int feedStart = (maxStart > 0) ? std::max(lineStart, maxStart) : lineStart;
    int feedEnd = (maxEnd > 0) ? std::min(lineEnd, maxEnd) : lineEnd;

    switch (query) {
        case Qt::ImMicroFocus:

        {
            int startPos = (preeditPos >= 0) ? preeditPos : pos;
            Point pt = sqt->LocationFromPosition(startPos);
            int width = send(SCI_GETCARETWIDTH);
            int height = send(SCI_TEXTHEIGHT, line);
            return QRect(pt.x, pt.y, width, height);
        }

        case Qt::ImFont:
        {
            char fontName[64];
            int style = send(SCI_GETSTYLEAT, pos);
            int len = send(SCI_STYLEGETFONT, style, (sptr_t)fontName);
            int size = send(SCI_STYLEGETSIZE, style);
            bool italic = send(SCI_STYLEGETITALIC, style);
            int weight = send(SCI_STYLEGETBOLD, style) ? QFont::Bold : -1;
            return QFont(QString::fromUtf8(fontName, len), size, weight, italic);
        }

        case Qt::ImCursorPosition:
        {
            return QVariant(sqt->pdoc->CountUTF16(feedStart, pos));

        }

        case Qt::ImAnchorPosition:
        {
            int anchor = sqt->sel.MainAnchor();
            if ((anchor < feedStart) || (feedEnd < anchor))

                return QVariant(sqt->pdoc->CountUTF16(feedStart, pos));
            return QVariant(sqt->pdoc->CountUTF16(feedStart, anchor));

        }

        case Qt::ImSurroundingText:
        {
            if (sqt->pdoc->TentativeActive())
                return QVariant(QString());

            QVarLengthArray<char,1024> buffer(feedEnd - feedStart + 1);

            Sci_CharacterRange charRange;
            charRange.cpMin = feedStart;
            charRange.cpMax = feedEnd;

            Sci_TextRange textRange;
            textRange.chrg = charRange;
            textRange.lpstrText = buffer.data();

            send(SCI_GETTEXTRANGE, 0, (sptr_t)&textRange);

            return sqt->StringFromDocument(buffer.constData());
        }

        case Qt::ImCurrentSelection:
        {
            QVarLengthArray<char,1024> buffer(send(SCI_GETSELTEXT));
            send(SCI_GETSELTEXT, 0, (sptr_t)buffer.data());

            return sqt->StringFromDocument(buffer.constData());
        }

        default:
            return QVariant();
    }
}

[Neil Hodgson]

johnsonj:

> retrieve surroudding for the old scintillqt 3.5.2

Eh? You want to revert to the input query code from 3.5.2 and fix that?

I’m getting lost in the streams of patches.

Neil

[johnsonj]

Sorry! Do not get me wrong.
I submitted raw source for your reading.
it is not intended being commited.
My intention is proving my concept of surrounding feature with the existing code.

Demonstration surrounding feature of ScintillaQt 3.52 (Qt4.8.6 winXP)

https://www.youtube.com/watch?v=MprI70fi3O8

just for proof of concept!

[johnsonj]

reconversion scintillaQt3.52 (Qt4.8.6, fcitx, lubuntu15.10)

just proof of concept!
note changed:     if (event->preeditString().isEmpty() && event->preeditString().isEmpty()) {

https://youtu.be/wuM9TYHBxnM

------------------------------------------------------------------------------------------------------------------------------------------------

void ScintillaEditBase::inputMethodEvent(QInputMethodEvent *event)
{
    if (event->preeditString().isEmpty() && event->preeditString().isEmpty()) {

        int commitPos = send(SCI_GETCURRENTPOS);
        int start = sqt->pdoc->GetRelativePosition(commitPos, event->replacementStart());
        int end = sqt->pdoc->GetRelativePosition(start, event->replacementLength());

        sqt->pdoc->DeleteChars(start, end - start);

    }

    // Clear the current selection.
    sqt->ClearSelection();
    if (preeditPos >= 0)
        sqt->SetSelection(preeditPos, preeditPos);

    // Insert the commit string.
    if (!event->commitString().isEmpty() || event->replacementLength()) {
        // Select the text to be removed.

        // Replace the selection with the commit string.
        QByteArray commitBytes = sqt->BytesForDocument(event->commitString());
        char *commitData = commitBytes.data();
        sqt->AddCharUTF(commitData, static_cast<unsigned int>(strlen(commitData)));
    }

...

-----------------------------------------------------------------------------------------------------------------------------------------

[johnsonj]

I patched Qt ime for reconversion.
Demontration: Haven performing reconversion Qt5.6.0 on windows7.

https://youtu.be/rwMzhi8Agd8

[Neil Hodgson]

johnsonj:

> I patched Qt ime for reconversion.
> Demontration: Haven performing reconversion Qt5.6.0 on windows7.

So this requires Qt to incorporate your patch before it can work?

Neil

[johnsonj]

Yes, it is.

It will not likely be to be commited forever.
https://bugreports.qt.io/browse/QTBUG-50791

My interest is more on scintilla rather than Qt.

[johnsonj]

It appears Fitcx-Qt5 does not update ime context actively.
I find out Haven works well for Qt5.6.0 without updateMicroFocus() on windows7.

I reported this issue to fcitx-interest.

[johnsonj]

But it still needs updateMicroFocus().
Reconversion is triggered for not only reconversion key but also every key striked.

Demo: Thai language support for Qt5.6.0 on Windows7 with Haven.
https://youtu.be/1jqAN0aiANs

from: https://blog.du-a.org/2010/10/29/ibus-and-surrounding-text/

kesmanee engine seems depending heavily on reconversion feature.
SciteWin also seems to have worked well for kesmanee engine since reconversion feature has been introduced.

[johnsonj]

 Catch up:

[johnsonj]

Thank you for initialCompose commited!
https://sourceforge.net/p/scintilla/code/ci/225f39cfd93159c37966d50db15db7fdb894a503/

That relieved me a lot.

Here is my demonstration for how to set reconversion key for fcitx-anthy:
I always use shift+enter for reconversion.

https://www.youtube.com/watch?v=SNtV4qcPUVA

Please take note that the caret position is different between LTR selection and RTL one.
Ime takes consideration of surrounding around caret and shows relevant candidate list from it's dictionary.
So Candidate list may be different.

If you chose the wrong candidate (the candidate box are hidden) then you are already out of scintilla ime, therefore you have no chance to fix it.
You need retry reconversion.

If candidate list is abnormal, there may be no dictionary for reconversion.
But I think it can not happen since japanese ime for windows has default dictionary installed.

If you tell me your windows environment, I wll try it.

All test have been passed under XP(ime 2007) and Win7(may be ime 2010).
                          with fcitx on lubuntu 16.04 and fedora 22.

[Neil Hodgson]

johnsonj:

> Here is my demonstration for how to set reconversion key for fcitx-anthy:
> I always use shift+enter for reconversion.
>
> https://www.youtube.com/watch?v=SNtV4qcPUVA

OK. I can now perform reconversion on Lubuntu 16.10 using the Leafpad editor with fcitx and anthy. From “かた” (“kata”) to “形” which is the same as pressing space during initial entry.

I can not get reconversion to occur in Haven or SciTE, with or without the RcQt1212.patch and rcGTK1212.patch. Shift+Enter is being intercepted as no change is made and its normally equivalent to pressing Enter. Adding a trace to inputMethodEvent in Haven shows no calls when Shift+Enter is pressed.

> If you tell me your windows environment, I wll try it.

Windows 10 with Japanese Microsoft IME.

Neil

[johnsonj]

Neil:

""""   OK. I can now perform reconversion on Lubuntu 16.10 using the Leafpad editor with fcitx and anthy. From “かた” (“kata”) to “形” which is the same as pressing space during initial entry."""

It is good. You may find leafpad(gtktextview) reconversion does not work well for RTL selection.

"""   I can not get reconversion to occur in Haven or SciTE, with or without the RcQt1212.patch and rcGTK1212.patch. Shift+Enter is being intercepted as no change is made and its normally equivalent to pressing Enter. Adding a trace to inputMethodEvent in Haven shows no calls when Shift+Enter is pressed."""

I also find Shift+Enter does not work for Haven.
Tempraraly Ctrl+Shift+Enter works well for me.
You may try anothor key.
Note: I think fcitx for lubuntu 16.10 has bugs especially for configure environment.
      But I have not known yet the causes of bugs

When it comes to reconversion, I am sure SciTE for Gtk and Haven for Qt works better than Gedit for Gtk and Kate for Qt.

"""   Windows 10 with Japanese Microsoft IME."""

It seems to takes time for me to be ready for it.

[johnsonj]

"""On Win32, the rcQt1212.patch leads to some poor behaviour with Japanese. If the selection is backwards (from last to first), as would occur if you chose the wrong candidate then immediately Shift+Left to fix, then the selection is deleted and a candidate list shows two entries with spaces. If the selection is forwards then there is a candidate list with some single character entries but it is not a similar list to that which appeared after typing that text."""

I read your message again about reconversion for windows 10.
That result should happen for all windows versions (xp, 7, 8, 9, 10).
The culprit is QTextBoundaryFinder, which appears to know ascii only.

https://bugreports.qt.io/browse/QTBUG-50791

Yes, Reconversion on all versions of Qt for windows has never worked so far.

[johnsonj]

catch up:

[Neil Hodgson]

johnsonj:

> catch up:
> <rcQt0521.patch>

The changes here appear to be protection from an empty commit and a change for reporting failure for surrounding text retrieval from an empty string variant to an empty variant.

Is an empty commit indicating some other event? Is this a possible fix for bug #1913?
https://sourceforge.net/p/scintilla/bugs/1913/

Did these fix something else? On which operating system?

Neil

[johnsonj]

Neil:

I have never met https://sourceforge.net/p/scintilla/bugs/1913/ bug.
but I think it should be good to defend the abnormal case.

[johnsonj]

the codes moved at the end for efficency.

[Neil Hodgson]

Is reconversion working on Qt/Windows yet?

Neil

[johnsonj]

Qt/Windows is ready for reconversion but the reconversion is still buggy for all windows applications.
scintllaQt reconversion is ready for Linux but it is still buggy.

My scintllaQt reconversion is working on Linux.
The same codes should be working on Windows too.
So I have tried and proven it is working on Qt/Windows side.

I want scintillaQt reconversion to trigger Qt/windows have to answer the reconversion bug.
We have good reasons "scintillaQt first, Qt/windows second!"

[johnsonj]

catch up for 3.7.8 with small changed comment.

[johnsonj]

pardon me!
mis spelling 3.7.8 -> 3.7.5