Step for inline codes

[Alessandro Falappa]

Hello all,

  I was wondering why the inline code finder logic is always woven inside the filters (not all filters have it) rather than factoring it out to an Okapi Step that can then added to any pipeline.

Is there a technical reason or was it an explicit design choice?

I am asking because we find ourselves in situations where more and more we would have the ability to look for different placeholder styles in different formats, having a step would allow us to mix and match inline code finding with filters.

Thank you in advance.

Best Regards,

Alessandro Falappa

Integrations Team Leader

[Yves Savourel]

Hi Alessandro,

 

I think we did it as part of the filters initially because we tended to put all “filter-related” action in the filter.

But there is nothing that prevent to do this in a step after extraction.

It seems the tendency nowadays is to have several steps to perform the “full” extraction/filtering. It’s more flexible and powerful.

 

Cheers,

-ys

--
You received this message because you are subscribed to the Google Groups "okapi-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to okapi-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/CAE7HRiL1wY0RQDRdC%3DTivZZ-QKCw3x96Yw2jz2%3DgGCsNK2mUtg%40mail.gmail.com.

[jim]

Yes, I was going to say the same thing. In the early days steps were "post-processing".

Sergei's "PreprocessingFilter" filter is interesting in that it creates a single filter from a main filter and any number of steps. Code finder could be one of those steps.

Some day we will clean this up and remove CodeFinder from the filters.

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/000d01d78dbe%24a868ebb0%24f93ac310%24%40gmail.com.

[jim]

One of the limitations in some filters is that you can't use code finder and a subfilter at the same time. Moving code finder and (sub-)filters to a step would cleanly solve this issue.

I'm hoping in the near future we can address this redesign.

Jim

[Mihai Nita]

Although "philosophically" the codes are part of the filter in done cases.

For example `%s` world be part of the filter for .po files, and Java properties, and Android strings xml. But not for json, csv, yaml.

Maybe implementation-wise we would have the functionality in a library, called by the filter by default when it's part of the file format definition, and as a step otherwise.

Mihai

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/c2aed6a3-1e49-a7ab-4d61-de2b107d67c5%40gmail.com.

[Alessandro Falappa]

Hi Mihai,

  yes I agree that some formats explicitly define a base placeholder syntax, but we are seeing more and more users sending us documents in generic formats like JSON, CSV and YAML (and even XLSX!) with placeholders in several syntaxes.

Base placeholder syntax could, and maybe should, be a filter concern but having a Step to put in a pipeline with other filters allows for more flexibility. Of course inline code finding within a filter should be controllable (but I think already is).

Regards

Alessandro Falappa

Integrations Team Leader

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/CAK69zbnt%2BbBZmmuYCw6vKc%3D8RVEC9_Gz3OFqMnrHOy66ShyADw%40mail.gmail.com.

[jim]

We are seeing the same. That's why I'd like to get the subfilter as step code working so that we can add as many subfilters as needed. inline code finder could be just another subfilter step.

We will need to come up with a nice way to configure all this.

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/CAE7HRiL5_gHPWKJd18%2BBthNqiFrVvAedQTLDHc6u_t56ZFjsoQ%40mail.gmail.com.

[krsk...@gmail.com]

I wasn't sure what this discussion is really about but if we are talking about making a step that does subfiltering, let me remind you that this was attempted and failed twice.

https://groups.google.com/g/okapi-users/c/k1tMTo1VQLg/m/3XcodfdIAQAJ

I thought I wrote difficulties I faced with examples at least once, maybe twice, but I couldn't find that post :-(

I think the Subfiltering Step can work if we accept a restriction - each part of the TU between a pair of codes from the main filter is well-balanced from the subfilter's point of view. What does this mean? Pretending below is an OpenXML text document and we want to apply the HTML filter as a subfilter:

(1) This <em> can be </em> processed by Subfiltering Step.

(2) This <em> cannot be </em> processed by Subfiltering Step.

For (1), the main filter, OpenXML Filter in this case, will create a TU like:

[C1]This [/C1]<em> can be </em> [C2] processed by[/C2] by Subfiltering Step.

where [Cx] is a code representing the beginning of a run of  the same style and [/Cx] is the closing of the run.

If we inspect each part:

This: This doesn't have any HTML tag, so it is balanced.

<em> can be </em>: Balanced.

processed by: Balanced.

by Subfiltering Step.: Balanced

For (2), the main filter produces:

[C1] This <em> cannot[/C1] be </em> processed by Subfiltering Step.

Why? 

This <em> cannot: The ending em tag is missing.

be </em> processed by Subfiltering Step.: The starting em tag is missing.

Even if we accept this restriction, flagging violation of this restriction is difficult.

If a part includes "<p>This may be OK." Is this balanced? HTML does not require </p>, but what if </p> appears in the later part?

[Alessandro Falappa]

Hi Kuro,

  no I didn't mean a step that does sub-filtering but a step that does inline codes finding.

The idea presented by Jim is however intriguing.

Regards

Alessandro Falappa

Integrations Team Leader

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/dfce8e7e-f4ef-4b7c-87be-163e790b1b63n%40googlegroups.com.

[jim]

Kuro - that restriction is already in place with the current subfilter design - so I don't see a new problem. You expect the parent filter to produce a block that will be "wellformed" from the subfilter perspective. Whatever wellformed means for the sub-format.

  >>If a part includes "<p>This may be OK." Is this balanced? HTML does not require </p>, but what if </p> appears in the later part?

It is the responsibility of the parent filter to extract the self-contained block. But the only requirement should be that the block won't cause a parse error given the specific subfilter+configuration. In your specific example the HTML filter can parse "<p>This may be OK." just fine. If there are more tags then the parent filter should include those too.

Some formats may be too difficult to isolate blocks for subfiltering (but again that's not a new problem) - in that case the feature won't be supported. However, it is a good question of how that can be enforced in the pipeline.

One thing we do need to move forward with this is Mihai's stream modifications so we can expand and contract the events. Subfilter would create more events than the parent filter (one to many) etc..

Anyway, this is all a ways off in the future. Plenty of time to discuss later. I just wanted to make sure this is still on the radar.

Jim
 

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/dfce8e7e-f4ef-4b7c-87be-163e790b1b63n%40googlegroups.com.

[Mihai Nita]

Absolutely!

I agreed that we should have a step.

And configurable to understand most common "variants" (%s, %2.4d, %1$s, {0}, {foo}, ${foo}, {$foo}, whatever we can identify as widespread)

Mihai

To view this discussion on the web visit https://groups.google.com/d/msgid/okapi-devel/CAE7HRiL5_gHPWKJd18%2BBthNqiFrVvAedQTLDHc6u_t56ZFjsoQ%40mail.gmail.com.