How should I encode passwords with v4 security handler if I am using Unicode strings?
v4 security handler of PDF requires the password string to be encoded to PDFDocEncoding. If the password string contains characters that cannot be encoded to PDFDocEncoding, then those characters must first be normalized to codepage encoding (meaning the current codepage the OS uses), then convert the codepage encoding to PDFDocEncoding.
In Windows, you may choose to compile your app in Unicode rather than MCS. The problem when compiling your app in Unicode is that all user input will be given to you by Windows in UTF-16 encoding. And as your experiments have proven, there is no ideal way to obtain the user's input locale. With MBCS, however, Windows will map the source codepage encoding to the current system encoding so no conversion is necessary. For example, with MBCS, when your user enters the characters "тест" (regardless of the source codepage), if your current system encoding is using CP1252, then you will get 0xF2E5F1F2 as the coded characters for the original input. You can then use this to pass to UString and specify winansi encoding.