[scintilla:feature-requests] #1564 Optimize CreateFoldMap

[Zufu Liu]

---

[feature-requests:#1564] Optimize CreateFoldMap

Status: open
Group: Initial
Labels: Scintilla encoding dbcs win32
Created: Thu Jul 03, 2025 02:05 PM UTC by Zufu Liu
Last Updated: Thu Jul 03, 2025 02:05 PM UTC
Owner: Neil Hodgson
Attachments:

• DBCSFoldMap.py (2.1 kB; text/x-python)

Some ideas to speed up CreateFoldMap() in ScintillaWin.cxx.
1. WideCharLenFromMultiByte + WideCharFromMultiByte can be change to a single call ::MultiByteToWideChar(codePage, MB_ERR_INVALID_CHARS, pair.chars, 2, codePoint, _countof(codePoint));, where codePoint has size 2 or 4 (each byte converted to two surrogate pairs).
2. MultiByteFromWideChar can be called with std::wstring_view(wFolded, charsConverted) to eliminate the temporary string.
3. CaseFolderDBCS::Fold() should output ch and ch2 when pair[0] is zero (unconverted, e.g. ch2 is not trail byte):

const char *pair = foldingMap.at(ind).chars;
if (pair[0]) {
    folded[lenOut++] = pair[0];
    folded[lenOut++] = pair[1];
} else {
    folded[lenOut++] = ch;
    folded[lenOut++] = ch2;
}

Sine DBCS encoding has only hundred (100 - 200) case sensitive characters (Latin, Greek, Cyrillic, etc.), further optimization could only handle blocks (lead bytes) that has case sensitive characters.
for lead byte generated with attached script:

std::initializer_list<unsigned char> leadBytes;
switch (codePage) {
case cp932:
    leadBytes = {0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, 0xFA};
    break;
case cp936:
    leadBytes = {0xA2, 0xA3, 0xA6, 0xA7};
    break;
case cp949:
    leadBytes = {0xA1, 0xA3, 0xA5, 0xA7, 0xA8, 0xA9, 0xAC};
    break;
case cp950:
    leadBytes = {0x88, 0xA2, 0xA3, 0xC7, 0xC8};
    break;
default:
    leadBytes = {0xD9, 0xDA, 0xDB, 0xDC, 0xDD, 0xDE};
    break;
}
for (const unsigned char byte1 : leadBytes) {

global CodePageToFoldMap cpToFoldMap; may has problem as std::map is not nothrow default constructable (Visual C++ has a warning for it). it could be replaced with std::vector as only has max 5 entries (most time only one - the system ANSI code page). when use above leadBytes code, CreateFoldMap() is very fast, it may not worth to cache the result globally.

---

Sent from sourceforge.net because scintill...@googlegroups.com is subscribed to https://sourceforge.net/p/scintilla/feature-requests/

To unsubscribe from further messages, a project admin can change settings at https://sourceforge.net/p/scintilla/admin/feature-requests/options. Or, if this is a mailing list, you can unsubscribe from the mailing list.

[Neil Hodgson]

Due to vacation, it's unlikely I will progress new issues before mid-September.

At around 4ms on an older machine, FoldMap creation time doesn't appear to be significant. Caching avoids memory cost of multiple allocations.

[Zufu Liu]

As here are only five DBCS code pages, for most application only one (the system DBCS code page for East Asian) is active used,
I think CodePageToFoldMap can be changed to std::vector<std::pair<int, FoldMap>>.
std::vector is noexcept default constructible but std::map is not, using std::vector may benefit load/unload Scintilla.dll (avoid running complex constructor/destructor?).

For Notepad4, I just put FoldMap into CaseFolderDBCS, I think 64KB extra memory per document is not a thing, open 1000 documents in DBCS is just 64MB.

The size for FoldMap could be reduced, as the script DBCSFoldMap.py shows, for each DBCS code page, only at most eight blocks (lead bytes) contains case sensitive characters (less than 200 in total).

struct FoldMap {
    // offset to blockList, indexed by lead byte, if the value is zero, then no case mapping.
    uint16_t offset[128];
    // each block has 256 DBCSPair, at most eight blocks.
    std::array<DBCSPair, 8*256> blockList;
    //std::vector<DBCSPair> blockList;
    // index = offset[lead & 0x7f];
    // mapping = index ? blockList[index - 1 + trail] : zero;
};

CreateFoldMap() could be optimized (cp932 is 2x faster than other code pages):
1. merge the two MultiByteToWideChar() calls, since the expected result length is 1, a buffer of 2 or 4 is enough to catch all failure.
2. write back to foldingMap[index] can be delayed only when here is case mapping, zero means no change.
3. change char chars[2]; to char chars[2]{}; ensure FoldMap is zero initialized, no read of uninitialized when ch2 = mixed[i]; isn't trail byte.

Attachments:

• DBCSFoldMap.py (1.9 kB; text/x-python)

---

[feature-requests:#1564] Optimize CreateFoldMap

Status: open
Group: Initial
Labels: Scintilla encoding dbcs win32
Created: Thu Jul 03, 2025 02:05 PM UTC by Zufu Liu

Last Updated: Thu Jul 03, 2025 10:46 PM UTC
Owner: Neil Hodgson
Attachments: