PO files

[Alessandro Falappa]

Hello all,

  I worked some more on the PO escaping topic and started by understanding more thoroughly the current code, here are my findings.

• PO file to events extraction is performed by the POFilter, PO events to file merge is performed by GenericFilterWriter with a GenericSkeletonWriter and a POEncoder.
• A separate POWriter class is also used to write PO files from events and is used in Rainbowkit PO and Transifex package writers.
• POFilter uses the code finder and a set of default rules to intercept C printf format specifiers (e.g. %-6.3f), basic Java MessageFormat placeholders (e.g. {0}) and some escape combinations (e.g. \r, \n, \t, \f,\a, \b, \v and \r\n) but the user could disable the code finder or change the set of default rules.
• Code finder rules do not intercept escaped quotes \" and escaped backslash \\.
• POFilter does not perform any escaped chars decoding after stitching together the content of msgstr/msgid quoted lines, this probably was done to let the code finder operate on the raw content, thus escaped quotes and backslashes pass trough undecoded.
• POEncoder does not perform encoding of backslashes, quotes, newlines, tabs and other control characters. This mirrors the POFilter behavior and works for escape combinations intercepted by the default code finder rules.
• Since escaped backslashes and quotes are not intercepted nor decoded it is easy to break the format inserting an unescaped quotes or backslash character in the target for a language for example.
• From my runs the POEncoder is almost always called with a TEXT EncoderContext. The encode(String) method is called when there are no codes while encode(char) method is called when there are codes. In the latter case the encoder has no way to detect if a quotes/backslash character was escaped or not as it has no way to "look" at the previous character.

Given the previous findings I am unsure on how to proceed and therefore ask for suggestion/guidance.

I could:

1. Implement decoding of escaped combinations in the POFilter and conversely encode those combinations in the POEncoder
2. Add code finder rules to intercept escaped quotes and escaped backslashes

Escaping quotes and backslashes not already escaped in the POEncoder encode(String) method as I did in my previous PR is not enough to get correct behaviour in all cases.

What do you think?

Regards,

Alessandro Falappa

Senior Java Developer

[jim]

Thank you for digging into this.

My vote would be for #1. Before applying any codefinder rules the PO filter should have already decoded the content.

Jim 

--
You received this message because you are subscribed to the Google Groups "okapi-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to okapi-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/CAE7HRiLe%2BN1AX%2BA%2BsFbPw4qk_skLjP0hNG0sc3CRhLhE9XGm0A%40mail.gmail.com.

[Yves Savourel]

+1.

This would give a clean extracted string even when the CodeFinder is not used.

-ys

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/541305a8-e721-676a-5a6f-66085a0dc1b6%40gmail.com.

[Alessandro Falappa]

Created PR https://bitbucket.org/okapiframework/okapi/pull-requests/504 with the #1 approach.

Alessandro Falappa

Senior Java Developer

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/01bb01d72195%24d32797b0%247976c710%24%40gmail.com.

[jim]

Alex,

I'm getting a PO unit test failure with newllines - probably an OS difference (Linux in my case). I'll take a look at it and push a fix.

testEscapesAmongAlreadyEscaped

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/CAE7HRi%2Bjm%2BZBy5V1jqhvunmqUGQECuJ-2hgu4vQ2B%3D_gweiRmA%40mail.gmail.com.

[jim]

Alex,

I updated the POWriter to use the new POEncoder. I don't believe the embedded encoder did anything special.

We still get one test failure. But I'm not 100% sure the expected string is correct. However, this may be a bug with strings that have a mixture of escaped and unescaped chars. Can you take a look?

Escape/Unescape will make your head spin :-)

@Test @Ignore("Check expected string to make sure it is correct")
public void testEscapesAmongAlreadyEscaped () {
   String snippet = ""
      + "msgid \"' \\\\ \" \\\\\\\"\r"
      + "msgstr \"' \\\\ \" \\\\\\\"\r\r";
   String expected = ""
      + "msgid \"' \\\\ \\\" \\\\\\\\\"\r"
      + "msgstr \"' \\\\ \\\" \\\\\\\\\"\r\r";
   String result = rewrite(getEvents(snippet, locEN, locFR), locFR);
   assertEquals(header.replace('\n', '\r')+expected, result);
}

[Alessandro Falappa]

Hello Jim,

  I haven't touched POWriter and its test class in my PR as I thought it was already correctly handling quotes and their escaping.

I will update my fork and have a look. By the way, I am on Linux too.

Regards,

Alessandro Falappa

Senior Java Developer

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/f5b174f9-d12c-02a9-9aaa-211e6d2616a5%40gmail.com.

[Alessandro Falappa]

Hi all,

  looking at this test failure unveiled a "corner case" bug in the POFilter (it currently chops a trailing lone backslash after an escaped backslash, e.g. msgstr "\\\") and also highlights that maybe the filter should be stricter against not escaped quotes and backslashes in the input. I am currently revisiting the logic in the filter to throw an OkapiBadFilterInputException if single quotes or backslashes are found in the input. The filter could also be made a bit more tolerant and accept unescaped quotes and backslashes to let the POEncoder escape them on output, what would be the best behaviour? Are malformed PO files frequent in the wild?

With regard to the POWriter failing unit test replacing its logic with a POEncoder is not a backward compatible change. Previously the POWriter corrected unescaped quotes and backslashes to always produce a correct PO file while the POEncoder always escapes quotes and backslashes relying on the unescaping of the POFilter. In order to let the POWriter manage events coming from non-escaping-aware sources I would revert the use of the POEncoder in POWriter.

Regards,

Alessandro Falappa

Senior Java Developer

Il giorno ven 26 mar 2021 alle ore 18:48 jim <jhargr...@gmail.com> ha scritto:

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/f5b174f9-d12c-02a9-9aaa-211e6d2616a5%40gmail.com.

[jim]

>>he filter could also be made a bit more tolerant and accept unescaped quotes and backslashes to let the POEncoder escape them on output,

How do other tools and PO editors handle this issue? IMHO, any PO content not properly escaped should be flagged as malformed (OkapiBadFilterInputException) and fixed. But if other popular tools are lenient - then we have to do the same. Our policy has normally been to be lenient on parsing, but strict on writing.

>>With regard to the POWriter failing unit test replacing its logic with a POEncoder is not a backward compatible change.

I'd like to have a single Encoder if possible to avoid confusion. Maybe we could enhance the logic of the POEncoder. We do have the ability to send in parameters if needed.

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/CAE7HRiLUhopahMhCmu3LpbFw15wPX%3D0wzYQPimZRamgn5gpdvw%40mail.gmail.com.

[Alessandro Falappa]

Hello Jim,

  I opened the following PR https://bitbucket.org/okapiframework/okapi/pull-requests/505/tweaked-po-escaped-characters-management.

It is lenient on reading and strict on writing as you indicate. The POWriter test should now pass as well.

Note however that escaping is not performed when writing the msgid parts as those go into the skeleton and don't get passed to the POEncoder (see added unit test UnescapedRewrite).

Let me know what you think.

Regards,

Alessandro Falappa

Senior Java Developer

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/8a1bf738-e96f-beba-59ed-3e38eb8b7b7a%40gmail.com.

[jim]

Thanks!

Do you guys on the CC list have any thoughts on PO format and escaping? Do most tools accept "malformed" or unescaped content? Should the POFilter do the same?

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/CAE7HRiJ%3DghyxFSufhdbP5mj4PQt2hPKX6P8O1yGuCLBFfRZADg%40mail.gmail.com.