Filters not applied when user is member of multiple groups

[Joe]

Greetings!

I'm having difficulty getting multiple group filters to apply to one user.  Here is my current setup:

Group 1: 

    Contains conditions to filter tuples based on a SQL join.  This is working correctly.  This condition is very specific (i.e. limited to a certain area)

Group 2:

    Contains field filters to exclude some fields that are often hidden.  This filter is something that would be applied to a lot of users/groups.

User 1:

    Is a member of Group 1 and Group 2

The intended behavior is that User 1 would have its search result rows filtered by the where clauses from Group 1 and not have access to the fields excluded by Group2.   However, the Group1 conditions are taking effect, but all fields are available (field filters from Group2 are not taking affect).

When I add the exclude filter directly on Group1, it works as expected.   Am I missing something here about how to configure filters and conditions to  allow them to work properly when split across multiple groups.

Thanks in advance for any help.

joe

[Mark Klein]

On Jan 9, 2013, at 9:17 PM, Joe wrote:

> Greetings!
>
> I'm having difficulty getting multiple group filters to apply to one
> user. Here is my current setup:
>
> Group 1:
> Contains conditions to filter tuples based on a SQL join. This
> is working correctly. This condition is very specific (i.e. limited
> to a certain area)
>
> Group 2:
> Contains field filters to exclude some fields that are often
> hidden. This filter is something that would be applied to a lot of
> users/groups.
>
>
> User 1:
> Is a member of Group 1 and Group 2
>
>
> The intended behavior is that User 1 would have its search result
> rows filtered by the where clauses from Group 1 and not have access
> to the fields excluded by Group2. However, the Group1 conditions
> are taking effect, but all fields are available (field filters from
> Group2 are not taking affect).

It's been a while since I've had my hands in that code ... I'm going
to have to go research this a bit. I can say this, internally variman
creates a single sql query ... I just don't know if the query is
additive without looking. Can you enable debug logging for SQL and
send me the logfile?

I take it these are both condition rules?

[Joe]

Mark,

They are not both condition rules.  All the condition rules appear to be working as expected.  It is the filter rules that aren't working correctly.  When a have a user who is a member of several groups with condition rules, everything works as expected.  However, if I add a group which has filter rules (e.g. to hide fields that should not accessible to members of said group) the fields are still visible.

I have enabled debug level logging and SQL logging.  I see no messages referring to the filter rules.  The SQL statements that are logged do, in fact, include those fields that should be excluded from the resultset.

Below is an example security-constraints section of the rets-config.xml that shows two group definitions NonDeleted Only correctly affects the resulting SQL query, but HidePrivateFields does not.

  <security-constraints>

    <group-rules group="NonDeleted Only">

      <condition-rule resource="Property" class="Residential">

        <sql-constraint>status &lt;&gt; 'D'</sql-constraint>

      </condition-rule>

      <condition-rule resource="Property" class="Commercial">

        <sql-constraint>status &lt;&gt; 'D'</sql-constraint>

      </condition-rule>

      <condition-rule resource="Property" class="Land">

        <sql-constraint>status &lt;&gt; 'D'</sql-constraint>

      </condition-rule>

    </group-rules>

    <group-rules group="HidePrivateFields">

      <exclude-rule resource="Property" class="Residential">

        <system-names>ExpirationDate AgentShowingInstructions</system-names>

      </exclude-rule>

      <exclude-rule resource="Property" class="Commercial">

        <system-names>ExpirationDate AgentShowingInstructions</system-names>

      </exclude-rule>

      <exclude-rule resource="Property" class="Land">

        <system-names>ExpirationDate AgentShowingInstructions</system-names>

      </exclude-rule>

    </group-rules>

  </security-constraints>

Thanks in advance for any help you can offer.

joe

[Mark Klein]

And you are sure that the HidePrivateFields rule is enabled for the user(s) where you want it applied?

[Joe]

Mark,

Yes.  I also replicated the issue with a completely new user this afternoon.  

If the user is a member of only the HidePrivateFields group, the excluded fields in the filter condition are correctly applied.

If the user is a member of only the HideStreetAddress group, the excluded field in the filter condition are correctly applied.

If the user is a member of both HidePrivateFields and HideStreetAddress (both of which only contain filter rules), neither of the filter rules are applied.

If the user is a member of either HidePrivateFields or HideStreetAddress and the NonDeleted Only group is added (which contains only SQL condition rules), the SQL condition rule is applied but the fields listed in the filter rules are not excluded.

If the user is a member of multiple groups which contain SQL condition rules, the condition rules are correctly merged and the conditions are added to the resulting SQL where clause.

Any ideas? 

Thanks again for your assistance!

[Mark Klein]

On Apr 8, 2013, at 3:22 PM, Joe wrote:

> If the user is a member of only the HidePrivateFields group, the
> excluded fields in the filter condition are correctly applied.
> If the user is a member of only the HideStreetAddress group, the
> excluded field in the filter condition are correctly applied.
> If the user is a member of both HidePrivateFields and
> HideStreetAddress (both of which only contain filter rules), neither
> of the filter rules are applied.
>
> If the user is a member of either HidePrivateFields or
> HideStreetAddress and the NonDeleted Only group is added (which
> contains only SQL condition rules), the SQL condition rule is
> applied but the fields listed in the filter rules are not excluded.
>
> If the user is a member of multiple groups which contain SQL
> condition rules, the condition rules are correctly merged and the
> conditions are added to the resulting SQL where clause.

Interesting.

> Any ideas?

No, unfortunately. Looks like there might be an issue with the union
of rules. I'm going to have to dig into the code to try to debug this
when I have some time. Unfortunately, that might be a week or two as
I'm heading out of town.

[Mark Klein]

On Apr 8, 2013, at 3:22 PM, Joe wrote:

> Any ideas?

Actually, my gut feel earlier was correct. I think the suspect is
TableGroupFilter at line 51. It does a union of the tables of all the
filter classes. So, even though the table doesn't exist in the
"exclude" rule, it exists in the other rule and since this is a union,
Viola!

Looks like this is a fundamental design issue with the way filters are
applied when there are multiple filters. I don't think it will be an
easy fix, but if you want to give it a try ... have at it! I'll look
at it more when I get back into town.

[Joe]

Mark,

My guy told me bug as well. It's been many years since I've worked with Java, but I'll dig in and see what I can come up with. Thank you for giving me a place to start looking. I'll post patches for any possible fixes I come up with.

Have a good trip and thanks for you help!

Joe

[Joe]

Mark,

I believe that I've tracked down the code that will need to be changed using your suggestion as a jumping off point. Can you confirm for me that the intended behavior for multiple filter rules is the following:

1. If only INCLUDE rules are found, only the union of the set of fields specified MUST BE in the resultant field list for the given resource and class.
2. If only EXCLUDE rules are found, all fields not in the specifically excluded fields MUST BE in the resultant field list for the given resource and class.
3. If both INCLUDE and EXCLUDE rules are found, the resultant list of fields shall be determined in the following manner.
1. Determine the set of fields explicitly INCLUDEd (F_I)
2. Determine the set of fields explicitly EXCLUDEd (F_E)
3. Resultant set of fields is (in typical deny trumps allow security fashion) the set operation F_I - F_E 

Condition (SQL) rules are ALWAYS additive, and it's up to the user to make sure they do not contradict each other.

I'm working on a patch based on the above behavior, and will post it when I've tested it.  

joe

[Joe]

I need to do a little more testing and update the unit tests, but so far the attached patch appears to fix the problem.  Let me know what you think. 

joe

[Mark Klein]

Still out of town for the wedding and can't get into this in much detail, but this looks OK to me. But ... Others that use the filtering mechanism, speak up now, or forever hold your peace!

Regards,

M.

--

To quote my Bro: "Please forgive spelling and grammar mistakes, they're the fault of [his] iPhone."

[Mark Klein]

Again, very cursory review, but what happens if you only have include rules? If you have one set with Col1 and Col2; another with Col3 and Col4, what happens with the intersection?

I think you may need a union of all the include rules, a union of all the exclude rules, and then an intersection of the two results:

(Incl1 + Incl2 + Incl<n>) * (Excl1 + Excl2 + Excl<n>)

Regards,

M.

--

To quote my Bro: "Please forgive spelling and grammar mistakes, they're the fault of [his] iPhone."

On Apr 15, 2013, at 2:24 AM, Joe <jke...@gmail.com> wrote:

I need to do a little more testing and update the unit tests, but so far the attached patch appears to fix the problem.  Let me know what you think. 

joe

--
You received this message because you are subscribed to the Google Groups "Variman RETS Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to variman-discu...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

<FixTableGroupFilterWithMultipleGroups.diff>

[Joe]

That is an excellent point! You are correct.  I had some code to deal with this, but it seems I removed it at some point.  

I'll make changes to account for that situation and post an updated diff later today.

joe

[Joe]

Attached is an updated patch which I believe deals with all cases.  I have tested it with both include only, exclude only, and both include and exclude rules.  Here is an overview of what I did:

1. TableGroupFilter.addRules now retains the MTable that correspond to rule entries in either a include or exclude Map
2. TableGroupFilter.findTables collapses the include and exclude maps to a single Set of each based on the groups of interest
1. Starting with all MTable entries for the given resource/class
2. If there are include rules, perform a set intersection to eliminate fields not explicitly included
3. If there are exclude rules, remove all fields in the excluded fields set.

Let me know what you think.  I am working on updating the test suite for the hibernate project to fix failing tests and to test these new scenarios.  

I appreciate your help and feedback.

joe

[Joe]

Forgot to attach the patch.  

[Mark Klein]

Joe:

Looks good but I only have my iPad with me, so I can't test until I get home. It may take a couple of more days before I get to this, but looks like a good patch just from review. BTW: thanks for adding the JavaDocs ... I do that too when I touch the code and the docs are lacking in that area. 

Anyone else (Danny?) want to comment on the patch?

Regards,

M.

--

To quote my Bro: "Please forgive spelling and grammar mistakes, they're the fault of [his] iPhone."

<FixTableGroupFilterWithMultipleGroups_20130415.diff>

[Joe]

I've actually got other javadoc updates that I can send if you want them all together.  For this issue, I tried to send only the core changes to make it easier for you to review.  

I'm familiarizing myself with the codebase, and would like to spend some time documenting and updating it a little (e.g. consistent use of generics).  I also have plans to add some additional functionality that I'd be glad to contribute back to the community; however, I'll open separate topics on them after we're certain this patch resolves the filter rule bug.

Regards,

joe

[Mark Klein]

On Apr 15, 2013, at 3:26 PM, Joe wrote:

Forgot to attach the patch.  

Looks like there is only a single file patched here, but I'm getting a compile error because FilterRule is missing containsSystemName. Also, please remember that the base Java to be used is 1.5, so things like String.isEmpty() don't exist (I've address that in TableGroupFilter already).

On Monday, April 15, 2013 5:26:00 PM UTC-5, Joe wrote:

Attached is an updated patch which I believe deals with all cases.  I have tested it with both include only, exclude only, and both include and exclude rules.  Here is an overview of what I did:

1. TableGroupFilter.addRules now retains the MTable that correspond to rule entries in either a include or exclude Map
2. TableGroupFilter.findTables collapses the include and exclude maps to a single Set of each based on the groups of interest
1. Starting with all MTable entries for the given resource/class
2. If there are include rules, perform a set intersection to eliminate fields not explicitly included
3. If there are exclude rules, remove all fields in the excluded fields set.

Let me know what you think.  I am working on updating the test suite for the hibernate project to fix failing tests and to test these new

Did you get anywhere on the test suite? I don't want to commit this stuff until everything is functional.

[Joe]

My apologies for the missed 1.5 issues.  I configured my IDE to target 1.5, but wasn't aware that this was only binary compatibility.  I have installed a 1.5 JDK to make sure everything builds and runs correctly against the 1.5 JRE.

Attached is a patch that contains everything including the updated test class for the TableGroupFilter with updated and additional tests.  I don't currently see test cases for the DefaultSearchSqlBuilder which I think would be the place to test whether ConditiionRules and TableGroupFilter rules are resolved into a correct SQL statement.  Since there were not tests present there, I haven't added any.  

Everything builds and tests correctly under JDK 1.5; and testing on a dedicated server yields the expected result for my group sets.

Please let me know if you notice anything else I need to address before you can commit.  I'm eager to move on to some other updates and feature additions. 

Thanks for your help,

joe