filter proposal

[Morteza Ansari]

Hi folks,

 

Here is the text I am proposing for the filter section.  Please send your comments/suggestions.

 

BTW, unfortunately I won’t be able to make our next call either (shame on me, this is 3 weeks in a row!).  I will roll in any comments and get it integrated into the API spec by early next week.

 

 

Cheers,

Morteza

 

 

Filtering

Filter parameter is used to limit the request results to resources that match the given criteria. The subset is determined by selecting only the Entries that satisfy the filter expression.

 

The expression language that is used in filter parameter supports references to attributes and literals. The literal values can be strings enclosed in single quotes, numbers and Boolean values (true or false).

 

The attribute name and attribute operator are case insensitive. For example the following two expressions will evaluate to the same logical value:

 

                Filter=userName Eq ‘john’

                filter=Username eq ‘john’

 

Filter MUST contain at least one valid Boolean expression. Each expression MUST contain an attribute name followed by an attribute operator and optional value. Multiple expressions MAY be combined using the two logical operators. Furthermore expressions can be grouped together using “()”.

 

The operators supported in the expression are listed in the following table.

 

Attribute Operators
eq	equal	The attribute and operator values must be identical for a match.
co	contains	The entire operator value must be a substring of the attribute value for a match.
sw	starts with	The entire operator value must be a substring of the attribute value, starting at the beginning of the attribute value. This criterion is satisfied if the two strings are identical.
pr	present (has value)	If the attribute has a non-empty value, or if it contains a non-empty node for complex attributes there is a match.
gt	greater than	If the attribute value is greater than operator value, there is a match. The actual comparison is dependent on the attribute type. For string attribute types, this is a lexicographical comparison and for DateTime types, it is a chronological comparison.
ge	greater than or equal	If the attribute value is greater than or equal to the operator value, there is a match. The actual comparison is dependent on the attribute type. For string attribute types, this is a lexicographical comparison and for DateTime types, it is a chronological comparison.
lt	less than	If the attribute value is less than operator value, there is a match. The actual comparison is dependent on the attribute type. For string attribute types, this is a lexicographical comparison and for DateTime types, it is a chronological comparison.
le	less than or equal	If the attribute value is less than or equal to the operator value, there is a match. The actual comparison is dependent on the attribute type. For string attribute types, this is a lexicographical comparison and for DateTime types, it is a chronological comparison.

 

Logical Operators
and	Logical and	The filter is only a match if both expressions evaluate to true.
or	Logical or	The filter is a match if either expression evaluates to true.

 

Grouping Operators
( )	Precedence grouping	Filter expressions are evaluated from left to right unless the expression is enclosed in brackets.

 

 

If the specified attribute in a filter expression is a Plural attribute, the Resource MUST match if any of the instances of the given attribute match the specified criterion (e.g. if a User has multiple emails values, only one has to match for the entire User to match). For complex attributes, the fully qualified Sub-Attribute must be specified in the expression using the '.' character as separator between the attribute name and the Sub-Attribute name. For example, to filter by userName the parameter value is userName and to filter by first name, the parameter value is name.givenName.

 

Providers MAY support additional filter operations if they choose. Providers MUST decline to filter results if the specified filter operation is not recognized.

 

String type attributes are case insensitive by default unless the attribute type is defined as a caseExact string.

Attribute operators ‘eq’, ‘co’, and ‘sw’ MUST perform caseIgnore matching for all string attributes unless the attribute is defined as caseExact. By default all string attributes are caseIgnore.

 

Examples:

filter=userName eq 'jsmith'

filter=userName co ‘smith’

filter=userName sw ‘J’

filter=title pr

filter=meta.lastModified gt ‘2011-05-13T04:42:34Z’

filter=meta.lastModified ge ‘2011-05-13T04:42:34Z’

filter=meta.lastModified lt ‘2011-05-13T04:42:34Z’

filter=meta.lastModified le ‘2011-05-13T04:42:34Z’

filter=title pr and userType eq ‘Employee’

filter=title pr or userType eq ‘Intern’

filter=userType eq ‘Employee’ and (email co ‘foo.com’ or email co ‘bar.com)

 

 

[Samuel Erdtman]

Good work,

One very small thing, should we capitalize must in the following sentence 

"...the fully qualified Sub-Attribute must be specified in the expression..." changing it to 

"...the fully qualified Sub-Attribute MUST be specified in the expression..."

Cheers

//Samuel

[Morteza Ansari]

I agree,  will correct it.  Thanks.

 

 

Cheers,

Morteza

[Erik Wahlström]

Looks great.

I just have one small worry regarding the line "Providers MUST decline to filter results if the specified filter operation is not recognized." Just to make it clear that decline means an http response error code, we should define the error code (400?) in the text.

Best regards

Erik Wahlström

[Morteza Ansari]

Good catch. Trey, what do you think?  I carried this from the previous filter text.


Cheers,
Morteza

-----Original Message-----
From: cloud-d...@googlegroups.com on behalf of Erik Wahlström
Sent: Sun 8/7/2011 3:11 AM
To: cloud-d...@googlegroups.com
Subject: Re: filter proposal

[Trey Drake]

Yes, a 400 is the right choice with the obvious note that a human parseable message should also be included; e.g., "Query operator 'foo' is not supported".

I'm due to update the API spec and will include the filter work in this round for review on Tuesday.  

Thanks,

Trey

[Kelly Grizzle]

Great stuff, Morteza! I have a few comments.

> The expression language that is used in filter parameter supports
> references to attributes and literals. The literal values can be strings
> enclosed in single quotes, numbers and Boolean values (true or false).

We probably need to escape strings that contain single-quotes. Of
course, with this we also need to allow escaping the escape
character. Also, I would prefer double-quotes around string literals
since this is more consistent with the JSON representation. Granted,
these are two completely different things, but the consistency is
nice. We could just go with JSON string escaping since there are
already rules defined around this.

> starts with
>
> The entire operator value must be a substring of the attribute value,
> starting at the beginning of the attribute value. This criterion is
> satisfied if the two strings are identical.

The last sentence here seems like a copy-and-paste error.

> Precedence grouping
>
> Filter expressions are evaluated from left to right unless the
> expression is enclosed in brackets.

I think typically AND takes precedence over OR in boolean
expressions. For example, the following:

foo OR bar AND baz

could also be written as:

foo OR (bar AND baz)

Should we use this instead of left-to-right evaluation?

--Kelly

[Trey Drake]

On Aug 9, 2011, at 10:21 AM, Kelly Grizzle wrote:

> Great stuff, Morteza! I have a few comments.
>
>> The expression language that is used in filter parameter supports
>> references to attributes and literals. The literal values can be strings
>> enclosed in single quotes, numbers and Boolean values (true or false).
>
> We probably need to escape strings that contain single-quotes. Of
> course, with this we also need to allow escaping the escape
> character. Also, I would prefer double-quotes around string literals
> since this is more consistent with the JSON representation. Granted,
> these are two completely different things, but the consistency is
> nice. We could just go with JSON string escaping since there are
> already rules defined around this.
>

Since the filter string is in the request URL vs the payload, section 1.2 (percent encoding) should apply here vs JSON escaping.

>> starts with
>>
>> The entire operator value must be a substring of the attribute value,
>> starting at the beginning of the attribute value. This criterion is
>> satisfied if the two strings are identical.
>
> The last sentence here seems like a copy-and-paste error.
>
>> Precedence grouping
>>
>> Filter expressions are evaluated from left to right unless the
>> expression is enclosed in brackets.
>
> I think typically AND takes precedence over OR in boolean
> expressions. For example, the following:
>

+1 AND precedes OR

[Kelly Grizzle]

> >> The expression language that is used in filter parameter supports
> >> references to attributes and literals. The literal values can be strings
> >> enclosed in single quotes, numbers and Boolean values (true or false).
>
> > We probably need to escape strings that contain single-quotes.  Of
> > course, with this we also need to allow escaping the escape
> > character.  Also, I would prefer double-quotes around string literals
> > since this is more consistent with the JSON representation.  Granted,
> > these are two completely different things, but the consistency is
> > nice.  We could just go with JSON string escaping since there are
> > already rules defined around this.
>
> Since the filter string is in the request URL vs the payload, section 1.2 (percent encoding) should apply here vs JSON escaping.

Good point - I agree with you about percent encoding. It feels like
the escaping is still necessary, though.

[Kelly Grizzle]

Sorry to resurrect an old thread, but I have been reviewing the API doc and noticed that these suggestions never made it back into the spec.  It would be nice to get this into the doc before next week if we choose to make these changes.

Morteza, did you have any thoughts about logical operator precedence or moving to double-quotes with JSON escaping for string literals (http://json.org/)?  As Trey pointed out, this would be subject to URL percent encoding as well.  I'm not sure if this needs to be spelled out in the spec, though, since this is a standard HTTP requirement.

This would allow filters like this:

filter=familyName eq "O'Malley"

filter=title sw "\"hello world\""

Thoughts?

--Kelly

[Kelly Grizzle]

I made these changes to the REST API. Please review the changes in
revision 88 (http://code.google.com/p/scim/source/detail?r=88) and let
me know of anything that should be changed or clarified.

On an unrelated note, I also made a slight change to the core schema
doc to indicate that the "groups" attribute is read-only. See
revision 87 - http://code.google.com/p/scim/source/detail?r=87.

--Kelly

On Oct 12, 1:11 pm, Kelly Grizzle <kpgriz...@gmail.com> wrote:
> Sorry to resurrect an old thread, but I have been reviewing the API doc and
> noticed that these suggestions never made it back into the spec.  It would
> be nice to get this into the doc before next week if we choose to make these
> changes.
>
> Morteza, did you have any thoughts about logical operator precedence or
> moving to double-quotes with JSON escaping for string literals (http://json.org/)? As Trey pointed out, this would be subject to URL
> percent encoding as well.  I'm not sure if this needs to be spelled out in
> the spec, though, since this is a standard HTTP requirement.
>
> This would allow filters like this:
>
> filter=familyName eq "O'Malley"
>
> filter=title sw "\"hello world\""
>
> Thoughts?
>