UserProfile configuration JSON format

1,895 views
Skip to first unread message

Vlastimil Elias

unread,
Apr 29, 2021, 4:31:01 AM4/29/21
to keyclo...@googlegroups.com

Hi,

as part of the work on UserProfile SPI we discussed configuration format for few last weeks. Finally we agreed that we will use original proposal available at https://github.com/keycloak/keycloak-community/blob/master/design/user-profile.md#default-provider as it was agreed before by more parties. But that proposal has some strange constructs which are even not valid JSON, and functionality is not fully clear for some aspects also.

So I refined it to be valid JSON and contain what we need now:

  • requirement section polished and made valid JSON easily parsable from Java code

  • validations section - converted to array so more validations can be used, context removed (we agreed it is not necessary as these validations are mainly attribute value format validations which should be same on all places, and important business validations dependant on contexts are added by UserProfileProvider code and not configured here. We can add it back later if there will be real need from real deployments), config section polished to be valid parsable JSON consistent with Validation SPI proposal

  • converter section - removed for now, can be added back later once it will be fully clear where and how to use it

  • annotations section - unchanged and probably unused for now, but will be necessary later (for dynamic forms) so kept

Config example showing all possibilities at one attribute (so they may be nonsense from functionality point of view):

{

  "attributes": [

      {

        "name": "department",

        "permissions": {

            "view": ["admin", "user"], 

            "edit": ["admin"],

        },

        "requirements": {

            "always": true,

            "roles" : ["user", "admin"],

            "scopes" : ["phone-1", "phone-2"]

        },

        "validations": [

        {

              "validator": "length",

              "configuration": { "min": 3, "max": 255 }

          },

          {

            "validator": "emailFormat"

          },

          {

            "validator": "emailDomainDenyList"

          }

        ],

        "annotations":  {

            "key": "value",  

            "gui-order": "1",

            "type": "dropdown"

        }

      },{

        "name": "phone",

        .... 

      }

  ]

}

Format description with functionality details:

Some configuration sections use these pseudo-roles:

  • user - any place where user is involved - activities during authentication flow, Account console and related User Account REST API

  • admin - activities from Admin REST API (covers Admin console). It is expected that this part will be evolved in the future to use Roles defined for realm in the admin console to allow fine grained control.


Config sections/fields description:

  • attributes - defines user profile attributes available in the realm. Array of attribute definition objects with fields:

    • name - name of the attribute, required, must be unique. Can contain only lowercase and uppercase letters, numbers and special characters ‘.’, ‘-’, ‘_’

    • permissions - mandatory structure to define permissions for the attribute:

      • view - array of strings containing pseudo-roles who can view attribute value, nobody can view it if no any role is named here. username and email fields are always visible to the user even if not configured here. Username is always visible to admin even if not configured here.

      • edit - array of strings containing pseudo-roles who can edit attribute value, nobody can edit it if no any role is named here. username and email fields may or may not be editable based on other realm settings, configuration here may not important for them if realm business rules do not allow them to be edited

    • requirements - optional rules defining when is attribute value required (it means value is a non-blank string). Attribute is NOT required if this section is not defined at all or if all rules in this structure keep attribute optional:

      • always - optional boolean, default false. If true then attribute is always required and other settings of this section have no effect 

      • roles - array of strings containing pseudo-roles which has attribute required - allows case when admin provisions account without some attribute, but user have to fill it in during first login

      • scopes - array of strings containing names of auth flow scopes for which is attribute required. Default client scopes are taken into account also, so required attributes can be defined per client. This section has no effect when the roles section contains “user” as attribute is always mandatory then for all auth flow scopes.

      • Note: username and email fields can be required based on the important realm setting (mainly “email as username”) - they will be required automatically by UserProfileProvider java code when necessary. If these settings keep email as not required, then it can be configured here as required

    • validations - array of objects representing validations applied to the attribute. In general these are mainly value format/business validations. “required” validation is not here as it is driven by requirements rules. username and email fields can have some validations dependent on the realm setting (“email as username”, “email unique” etc) and they are not configured here to keep config simple (they will be added automatically by UserProfileProvider java code when necessary, needs to be documented in the doc!). Validation object contains:

      • validator - name/id of the Validator defined/implemented in Validation SPI (mandatory). If type inherits from parent type this is a key.

      • configuration - configuration of the Validator for this validation (optional). Key value pairs, values can be text, number, boolean, array of values etc (will depend on Validator SPI). Each Validator defines its own possible configuration options.

    • annotations - optional JSON object containing unstructured key value pairs which are populated to the frontend frameworks (freemarker context, JavaScript, returned over user profile metadata REST API) and can be used as controls for UI/theme rendering etc.


It is expected that user profile editor will be implemented in the Admin Console later (config JSON file will be read from Admin REST API and persisted back this way). I'm not expect on JS frontend framework used there, so one of questions is whether this format is good to be manipulated inside of JS when editor will be implemented. Main question for me is that attributes array where attribute name is inside of the object representing attribute. I can imagine use of JS object instead, where attribute name will be key in that object, something like:

"attributes" : {

    "department": {"permissions": ...},

   "phone": {...}

}

This way attributes can be more easily accessed by dot notation by name in JS I think, but not sure if it is important for the Admin Console editor. It also more naturally expresses that attribute name must be unique, but we can validate this even for array structure. Drawback of this format is that schema is not fully defined as list of allowed fields in the "attributes" object is unknown.

Thanks in advance for any comments or advices, I already started implementation of the UserProfile provider based on this configuration (together with Pedro) as I don't expect any major functionality requirement changes for now, some minor changes in JSON format may be incorporated still. I plan to do PR against keycloak-community to update user-profile.md later once we discuss refined proposal here ;-)

Vlastimil

-- 
Vlastimil Elias
He / Him / His
Principal Software Engineer, DXP Application Development
Red Hat

Pedro Igor Craveiro e Silva

unread,
Apr 29, 2021, 9:59:49 AM4/29/21
to Vlastimil Elias, keyclo...@googlegroups.com
Looks like we could use `required` instead of `requirements` as per your definition we are defining whether the attribute is required.

"required": {}

An empty object here means the attribute is required, following the semantics for `always`.

The other settings we could have:

"required": {
     "when": {
         "user": ["user", "admin"]
     }
}

"required": {
     "when": {
         "scope": ["foo", "bar"]
     }
}

The `when` is more verbose, so we could also have `whenUser` and `whenScope`.
 
    • validations - array of objects representing validations applied to the attribute. In general these are mainly value format/business validations. “required” validation is not here as it is driven by requirements rules. username and email fields can have some validations dependent on the realm setting (“email as username”, “email unique” etc) and they are not configured here to keep config simple (they will be added automatically by UserProfileProvider java code when necessary, needs to be documented in the doc!). Validation object contains:

      • validator - name/id of the Validator defined/implemented in Validation SPI (mandatory). If type inherits from parent type this is a key.

      • configuration - configuration of the Validator for this validation (optional). Key value pairs, values can be text, number, boolean, array of values etc (will depend on Validator SPI). Each Validator defines its own possible configuration options.

    • annotations - optional JSON object containing unstructured key value pairs which are populated to the frontend frameworks (freemarker context, JavaScript, returned over user profile metadata REST API) and can be used as controls for UI/theme rendering etc.


It is expected that user profile editor will be implemented in the Admin Console later (config JSON file will be read from Admin REST API and persisted back this way). I'm not expect on JS frontend framework used there, so one of questions is whether this format is good to be manipulated inside of JS when editor will be implemented. Main question for me is that attributes array where attribute name is inside of the object representing attribute. I can imagine use of JS object instead, where attribute name will be key in that object, something like:

"attributes" : {

    "department": {"permissions": ...},

   "phone": {...}

}


This way attributes can be more easily accessed by dot notation by name in JS I think, but not sure if it is important for the Admin Console editor. It also more naturally expresses that attribute name must be unique, but we can validate this even for array structure. Drawback of this format is that schema is not fully defined as list of allowed fields in the "attributes" object is unknown.


I like this format too. I'm not sure about what you mean by "not fully defined as a list of allowed fields" though. The `attributes` object will be a map where the key set is the list of allowed fields.
 

Thanks in advance for any comments or advices, I already started implementation of the UserProfile provider based on this configuration (together with Pedro) as I don't expect any major functionality requirement changes for now, some minor changes in JSON format may be incorporated still. I plan to do PR against keycloak-community to update user-profile.md later once we discuss refined proposal here ;-)

Vlastimil

-- 
Vlastimil Elias
He / Him / His
Principal Software Engineer, DXP Application Development
Red Hat

--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/9d5c9545-360f-8fde-1a9c-8f5a5993b0bf%40redhat.com.

Marek Posolda

unread,
Apr 29, 2021, 10:16:09 AM4/29/21
to Vlastimil Elias, keyclo...@googlegroups.com
On 29. 04. 21 10:30, Vlastimil Elias wrote:

Hi,

as part of the work on UserProfile SPI we discussed configuration format for few last weeks. Finally we agreed that we will use original proposal available at https://github.com/keycloak/keycloak-community/blob/master/design/user-profile.md#default-provider as it was agreed before by more parties. But that proposal has some strange constructs which are even not valid JSON, and functionality is not fully clear for some aspects also.

I see. Had similar issue with the "Client policies" specification with invalid JSON :) That specs is here just FYI https://github.com/keycloak/keycloak-community/blob/master/design/client-policies.md

Probably will be nice to make sure that JSON configuration of client policies and user profile are a bit consistent with each other? Few days ago, I've started discussion regarding client policies https://groups.google.com/g/keycloak-dev/c/MfoTQbNPljE .

- For example agree on whether we go with camel cases like "emailDomanyDenyList" or hyphen formats like "email-domain-deny-list" . Client policies now use the hyphens for configuration options as well as for provider names. I see that you using combination (EG. gui-order VS emailDomanyDenyList). IMO the hyphen format is better, so I would align with that. It is not a blocker as the configuration options in Keycloak as well as provider names are not consistent in this area, but maybe now is good to time to start with some consistency here? :)

- Regarding provider configurations, the client policies will probably use the configuration option wrapped at same level as "validator" instead of dedicated "configuration" element. See option 2 from the mentioned client policies thread. Hence for example something like:

[
    {
        "validator": "length",

        "min": 3,
        "max": 255
    },
...

]

Format description with functionality details:

Some configuration sections use these pseudo-roles:

  • user - any place where user is involved - activities during authentication flow, Account console and related User Account REST API

  • admin - activities from Admin REST API (covers Admin console). It is expected that this part will be evolved in the future to use Roles defined for realm in the admin console to allow fine grained control.


Config sections/fields description:

  • attributes - defines user profile attributes available in the realm. Array of attribute definition objects with fields:

    • name - name of the attribute, required, must be unique. Can contain only lowercase and uppercase letters, numbers and special characters ‘.’, ‘-’, ‘_’

    • permissions - mandatory structure to define permissions for the attribute:

      • view - array of strings containing pseudo-roles who can view attribute value, nobody can view it if no any role is named here. username and email fields are always visible to the user even if not configured here. Username is always visible to admin even if not configured here.

      • edit - array of strings containing pseudo-roles who can edit attribute value, nobody can edit it if no any role is named here. username and email fields may or may not be editable based on other realm settings, configuration here may not important for them if realm business rules do not allow them to be edited

    • requirements - optional rules defining when is attribute value required (it means value is a non-blank string). Attribute is NOT required if this section is not defined at all or if all rules in this structure keep attribute optional:

      • always - optional boolean, default false. If true then attribute is always required and other settings of this section have no effect 

      • roles - array of strings containing pseudo-roles which has attribute required - allows case when admin provisions account without some attribute, but user have to fill it in during first login

      • scopes - array of strings containing names of auth flow scopes for which is attribute required. Default client scopes are taken into account also, so required attributes can be defined per client. This section has no effect when the roles section contains “user” as attribute is always mandatory then for all auth flow scopes.

      • Note: username and email fields can be required based on the important realm setting (mainly “email as username”) - they will be required automatically by UserProfileProvider java code when necessary. If these settings keep email as not required, then it can be configured here as required

    • validations - array of objects representing validations applied to the attribute. In general these are mainly value format/business validations. “required” validation is not here as it is driven by requirements rules. username and email fields can have some validations dependent on the realm setting (“email as username”, “email unique” etc) and they are not configured here to keep config simple (they will be added automatically by UserProfileProvider java code when necessary, needs to be documented in the doc!). Validation object contains:

      • validator - name/id of the Validator defined/implemented in Validation SPI (mandatory). If type inherits from parent type this is a key.

      • configuration - configuration of the Validator for this validation (optional). Key value pairs, values can be text, number, boolean, array of values etc (will depend on Validator SPI). Each Validator defines its own possible configuration options.

    • annotations - optional JSON object containing unstructured key value pairs which are populated to the frontend frameworks (freemarker context, JavaScript, returned over user profile metadata REST API) and can be used as controls for UI/theme rendering etc.


It is expected that user profile editor will be implemented in the Admin Console later (config JSON file will be read from Admin REST API and persisted back this way). I'm not expect on JS frontend framework used there, so one of questions is whether this format is good to be manipulated inside of JS when editor will be implemented. Main question for me is that attributes array where attribute name is inside of the object representing attribute. I can imagine use of JS object instead, where attribute name will be key in that object, something like:

"attributes" : {

    "department": {"permissions": ...},

   "phone": {...}

}

This way attributes can be more easily accessed by dot notation by name in JS I think, but not sure if it is important for the Admin Console editor. It also more naturally expresses that attribute name must be unique, but we can validate this even for array structure. Drawback of this format is that schema is not fully defined as list of allowed fields in the "attributes" object is unknown.

I am also not sure about this one. The array is useful for ordering, which IMO you can't easily achieve with attribute name as key. But not sure if ordering is a requirement for attributes? If you don't need ordering and at the same time, you want to achieve uniqueness, then the attribute name as key is maybe better option than array.

I don't have any experience with  JSON UI console editor. In the PR draft for client policies https://github.com/keycloak/keycloak/pull/7969 , it is ATM done just as a plain textarea where administrator needs to manually edit JSON as string. For new admin console, we probably need something more fancy, but in my PR, I focus just on the String for now.

Another things, which we may want to align on is:

- Versioning of the JSON configuration. Just replied to client policies thread in relation to this https://groups.google.com/g/keycloak-dev/c/MfoTQbNPljE

- Admin REST endpoints . See my PR and the discussion on client policies thread about this.

- Are we going with single JSON file with the default (built-in) user profile and client policies? Or have separate files for client policies and separate for user profile? ATM I even have separate files for "client profiles" and "client policies" in my PR https://github.com/keycloak/keycloak/tree/946e79bd6aa0bce805d5d7d191e2f4f3fc1bc3fe/services/src/main/resources . Use separate file for User Profile works for me, but maybe will be good to stick with same directory inside keycloak-services module?

In general, I am not 100% sure how much we need to make sure that user profile and client policies are consistent as we are close to deadline and trying to achieve 100% consistency can block as a bit. Also both admin console and admin REST API is going to change in the near future. However maybe we can do some low-hanging fruits and try to achieve some consistency on some basic things (EG. hyphen VS camelCase, Skip "configuration" element , versioning etc) .

Marek

Thanks in advance for any comments or advices, I already started implementation of the UserProfile provider based on this configuration (together with Pedro) as I don't expect any major functionality requirement changes for now, some minor changes in JSON format may be incorporated still. I plan to do PR against keycloak-community to update user-profile.md later once we discuss refined proposal here ;-)

Vlastimil

-- 
Vlastimil Elias
He / Him / His
Principal Software Engineer, DXP Application Development
Red Hat
--

Pedro Igor Craveiro e Silva

unread,
Apr 29, 2021, 10:43:49 AM4/29/21
to Marek Posolda, Vlastimil Elias, keyclo...@googlegroups.com
If you consider our endpoints and their payloads,  we don't use hyphens. For me, consistency is to not use hyphens.

If we want to change that later in favor of consistency, we should consider reviewing all of our all JSON representations.

Also, if you consider that we might want to expose the user profile config through a CRD in K8S, it should be a lot easier to map the JSON format to the YAML representation. Even though k8s supports JSON, YAML seems to be the preferred format.

- Regarding provider configurations, the client policies will probably use the configuration option wrapped at same level as "validator" instead of dedicated "configuration" element. See option 2 from the mentioned client policies thread. Hence for example something like:

[
    {
        "validator": "length",
        "min": 3,
        "max": 255
    },
...

]

It works for me too. But still think that: 

[
    "length": {"min": 3, "max": 255}
]

Reads a lot better. As per my comment in the client policies thread. Also making it easier to map to YAML.
I agree we should not try to make them consistent with each other. But some key bits that make sense to improve readability and writing.

Versioning, for instance. I would not take versioning into account right now because that has also an impact on our REST APIs. We started to define how versioning should look like in our REST API guidelines, but we did not yet implement anything around it. I would prefer to do it later once we decide to think about a more general solution for versioning (which also applies to any document, even payloads for Admin API)
 

Marek

Thanks in advance for any comments or advices, I already started implementation of the UserProfile provider based on this configuration (together with Pedro) as I don't expect any major functionality requirement changes for now, some minor changes in JSON format may be incorporated still. I plan to do PR against keycloak-community to update user-profile.md later once we discuss refined proposal here ;-)

Vlastimil

-- 
Vlastimil Elias
He / Him / His
Principal Software Engineer, DXP Application Development
Red Hat
--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/9d5c9545-360f-8fde-1a9c-8f5a5993b0bf%40redhat.com.


--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.

Vlastimil Elias

unread,
Apr 30, 2021, 2:45:25 AM4/30/21
to Pedro Igor Craveiro e Silva, Marek Posolda, keyclo...@googlegroups.com

Hi

On 29. 04. 21 16:43, Pedro Igor Craveiro e Silva wrote:
On Thu, Apr 29, 2021 at 11:16 AM Marek Posolda <mpos...@redhat.com> wrote:
On 29. 04. 21 10:30, Vlastimil Elias wrote:

Hi,

as part of the work on UserProfile SPI we discussed configuration format for few last weeks. Finally we agreed that we will use original proposal available at https://github.com/keycloak/keycloak-community/blob/master/design/user-profile.md#default-provider as it was agreed before by more parties. But that proposal has some strange constructs which are even not valid JSON, and functionality is not fully clear for some aspects also.

I see. Had similar issue with the "Client policies" specification with invalid JSON :) That specs is here just FYI https://github.com/keycloak/keycloak-community/blob/master/design/client-policies.md
I know, this is why I posted it here, to achieve some level of consistency.

I personally also like camelCase more than hyphens, as hyphens only add more characters ;-) Here stuff like "gui-order" and "emailFormat" is not part of the configuration format itself, it is free stuff in the first case, and name of validator from Validation SPI in second.

- Regarding provider configurations, the client policies will probably use the configuration option wrapped at same level as "validator" instead of dedicated "configuration" element. See option 2 from the mentioned client policies thread. Hence for example something like:

[
    {
        "validator": "length",
        "min": 3,
        "max": 255
    },
...

]

It works for me too. But still think that: 

[
    "length": {"min": 3, "max": 255}
]

Reads a lot better. As per my comment in the client policies thread. Also making it easier to map to YAML.

Right, this makes config shorter, and also forces that one validator is used only once per attribute (as keys in JSON objects have to be unique).

To be valid JSON, it have to be object, not array, but it is minor stuff.

Also for validators without configuration it will require empty object to be used, which looks a bit strange to me, I probably never saw JSON with empty objects ;-) But not a big deal I think:

"validations" : {
    "length": {"min": 3, "max": 255},

    "emailFormat": {},

    "emailDomainDenyList" : {}

Ordering is not important there, for dynamic GUI rendering it is expected to use extra annotation like "guiOrder"

I don't have any experience with  JSON UI console editor. In the PR draft for client policies https://github.com/keycloak/keycloak/pull/7969 , it is ATM done just as a plain textarea where administrator needs to manually edit JSON as string. For new admin console, we probably need something more fancy, but in my PR, I focus just on the String for now.

Another things, which we may want to align on is:

- Versioning of the JSON configuration. Just replied to client policies thread in relation to this https://groups.google.com/g/keycloak-dev/c/MfoTQbNPljE

- Admin REST endpoints . See my PR and the discussion on client policies thread about this.

We follow similar approach. JSON config file as string with simple textarea in Admin Console for now. I'll look at your PR to see yours REST API operations to have them similar

- Are we going with single JSON file with the default (built-in) user profile and client policies? Or have separate files for client policies and separate for user profile? ATM I even have separate files for "client profiles" and "client policies" in my PR https://github.com/keycloak/keycloak/tree/946e79bd6aa0bce805d5d7d191e2f4f3fc1bc3fe/services/src/main/resources . Use separate file for User Profile works for me, but maybe will be good to stick with same directory inside keycloak-services module?

I'd keep them separate to allow separate evolution of the format.

Vlastimil Elias

unread,
Apr 30, 2021, 2:54:47 AM4/30/21
to Pedro Igor Craveiro e Silva, keyclo...@googlegroups.com

Hi

So making attribute optional would mean to omit "required" item at al in the config, right?

An empty object here means the attribute is required, following the semantics for `always`.

The other settings we could have:

"required": {
     "when": {
         "user": ["user", "admin"]
     }
}

"required": {
     "when": {
         "scope": ["foo", "bar"]
     }
}

The `when` is more verbose, so we could also have `whenUser` and `whenScope`.

I do not like too much nesting here as we will never add other item beside "when", so I vote for flatter structure.

"whenUser" is also a bit confusing to me when one of values for it is "user". I'd keep "whenRole" or "whenUserRole" to allow future evolution to fine grained roles.


 
    • validations - array of objects representing validations applied to the attribute. In general these are mainly value format/business validations. “required” validation is not here as it is driven by requirements rules. username and email fields can have some validations dependent on the realm setting (“email as username”, “email unique” etc) and they are not configured here to keep config simple (they will be added automatically by UserProfileProvider java code when necessary, needs to be documented in the doc!). Validation object contains:

      • validator - name/id of the Validator defined/implemented in Validation SPI (mandatory). If type inherits from parent type this is a key.

      • configuration - configuration of the Validator for this validation (optional). Key value pairs, values can be text, number, boolean, array of values etc (will depend on Validator SPI). Each Validator defines its own possible configuration options.

    • annotations - optional JSON object containing unstructured key value pairs which are populated to the frontend frameworks (freemarker context, JavaScript, returned over user profile metadata REST API) and can be used as controls for UI/theme rendering etc.


It is expected that user profile editor will be implemented in the Admin Console later (config JSON file will be read from Admin REST API and persisted back this way). I'm not expect on JS frontend framework used there, so one of questions is whether this format is good to be manipulated inside of JS when editor will be implemented. Main question for me is that attributes array where attribute name is inside of the object representing attribute. I can imagine use of JS object instead, where attribute name will be key in that object, something like:

"attributes" : {

    "department": {"permissions": ...},

   "phone": {...}

}


This way attributes can be more easily accessed by dot notation by name in JS I think, but not sure if it is important for the Admin Console editor. It also more naturally expresses that attribute name must be unique, but we can validate this even for array structure. Drawback of this format is that schema is not fully defined as list of allowed fields in the "attributes" object is unknown.


I like this format too. I'm not sure about what you mean by "not fully defined as a list of allowed fields" though. The `attributes` object will be a map where the key set is the list of allowed fields.
No worry, I only meant that it will be free map, which was somehow problematic in era of XML validations against XML DTD's and XML schemas. But it is over hopefully ;-)

Marek Posolda

unread,
Apr 30, 2021, 5:25:19 AM4/30/21
to Vlastimil Elias, Pedro Igor Craveiro e Silva, keyclo...@googlegroups.com
Yes, ok. I take it that maybe we don't care about consistency for camelCase VS hyphens as Keycloak is still not very consistent on many places? :-)

- Regarding provider configurations, the client policies will probably use the configuration option wrapped at same level as "validator" instead of dedicated "configuration" element. See option 2 from the mentioned client policies thread. Hence for example something like:

[
    {
        "validator": "length",
        "min": 3,
        "max": 255
    },
...

]

It works for me too. But still think that: 

[
    "length": {"min": 3, "max": 255}
]

Reads a lot better. As per my comment in the client policies thread. Also making it easier to map to YAML.

Right, this makes config shorter, and also forces that one validator is used only once per attribute (as keys in JSON objects have to be unique).

To be valid JSON, it have to be object, not array, but it is minor stuff.

Also for validators without configuration it will require empty object to be used, which looks a bit strange to me, I probably never saw JSON with empty objects ;-) But not a big deal I think:

"validations" : {
    "length": {"min": 3, "max": 255},

    "emailFormat": {},

    "emailDomainDenyList" : {}

}


So question is if it is really accurate that there is one validator used only once per attribute? For example it may happen that you want to validate against more regexes. So you may need something like this:

"validations" : {
    "regex": { ... some regex1 config ...},
    "regex" : { ... some regex2 config}
}

which is not valid JSON as you can't have more "regex" keys within same JSON object. Of course, you can make sure that "regex" validator implementation will accept array of requested regexes instead of single regex. But maybe have array instead of single value can't be always achieved for all validator implementations...

One general concept, which I am not sure whether it is considered, is some kind of "conditional validations" . For example to address use-cases like:

- if user's country is "us", then make sure that attribute "zipCode" is in format like 5-digit number . Otherwise if country is "french guyana", then make sure that zipCode starts with "973" followed by 2 numbers etc.

Not sure if example is too advanced, but maybe there is use-case to validate that "zipCode" (or some other similar attributes) need to be validated based on some condition (like country of user). So assume we would have "condition" validator (which will wrap some other validator implementation), I wonder that the fact that count of conditions is limited to 1 is good... Construct some complex with single condition containing arrays can be a challenge or impossible at all...

Not sure if the examples above makes much sense, but in general, I wonder if having single validator per attribute is an advantage or rather a limitation...

Sure. Just a note that REST API for client policies/profiles is not completely ready yet. There are still some ongoing discussions in the client policies thread...

- Are we going with single JSON file with the default (built-in) user profile and client policies? Or have separate files for client policies and separate for user profile? ATM I even have separate files for "client profiles" and "client policies" in my PR https://github.com/keycloak/keycloak/tree/946e79bd6aa0bce805d5d7d191e2f4f3fc1bc3fe/services/src/main/resources . Use separate file for User Profile works for me, but maybe will be good to stick with same directory inside keycloak-services module?

I'd keep them separate to allow separate evolution of the format.
+1

In general, I am not 100% sure how much we need to make sure that user profile and client policies are consistent as we are close to deadline and trying to achieve 100% consistency can block as a bit. Also both admin console and admin REST API is going to change in the near future. However maybe we can do some low-hanging fruits and try to achieve some consistency on some basic things (EG. hyphen VS camelCase, Skip "configuration" element , versioning etc) .

I agree we should not try to make them consistent with each other. But some key bits that make sense to improve readability and writing.

Versioning, for instance. I would not take versioning into account right now because that has also an impact on our REST APIs. We started to define how versioning should look like in our REST API guidelines, but we did not yet implement anything around it. I would prefer to do it later once we decide to think about a more general solution for versioning (which also applies to any document, even payloads for Admin API)

Hmm... regarding versioning, I see the difference between versiononing of REST API and versioning of JSON format. IMO it will be good to support versioning of JSON format from day 1. It will simplify migration IMO as when Keycloak reads the user profile (or client policy) JSON from realm attribute, it can detect if it is correct version and properly support migrating it by going through some "JSON migrator chain" .

IMO we made the mistake that we did not support versioning on our JSON representation from day 1 in Keycloak. This means we still need to maintain some clunky constructs with many if conditions and keep the historic fields as deprecated fields on representation objects (EG. all the deprecated fields on RealmRepresentation object like "applications", "socialProviders" etc).

Marek

Vlastimil Elias

unread,
Apr 30, 2021, 8:39:49 AM4/30/21
to Marek Posolda, Pedro Igor Craveiro e Silva, keyclo...@googlegroups.com

Hi

Right, you can probably combine everything into one nice oneliner regexp ;-)

I though about this before a bit, I'm really not sure if two same validators will be ever necessary, common "one function" validators shouldn't, only this kind of scriptable ones is questionable a bit.

One general concept, which I am not sure whether it is considered, is some kind of "conditional validations" . For example to address use-cases like:

- if user's country is "us", then make sure that attribute "zipCode" is in format like 5-digit number . Otherwise if country is "french guyana", then make sure that zipCode starts with "973" followed by 2 numbers etc.

Not sure if example is too advanced, but maybe there is use-case to validate that "zipCode" (or some other similar attributes) need to be validated based on some condition (like country of user). So assume we would have "condition" validator (which will wrap some other validator implementation), I wonder that the fact that count of conditions is limited to 1 is good... Construct some complex with single condition containing arrays can be a challenge or impossible at all...

Not sure if the examples above makes much sense, but in general, I wonder if having single validator per attribute is an advantage or rather a limitation...

This is why we have Validator SPI proposal and separate PR for it. It allows to plug in custom validators, IMO they are better suited to create more complex validators than trying to express conditions in json configuration. One of my requirements for this SPI and its integration with UserProfile is that Validator must see other attributes from UserProfile also to allow cross/conditional validations.

To be fully correct, it is enough to add versioning in 2nd version of the format ;-) Just kidding, rest of team is better suited to define this, I can add it without problem into UserProfile config format, we have a room on the first level of it.

Have a nice weekend

Vl.

Pedro Igor Craveiro e Silva

unread,
Apr 30, 2021, 8:42:50 AM4/30/21
to Vlastimil Elias, Marek Posolda, keyclo...@googlegroups.com
Me neither :) But only when some API is not configured (on purpose or not) to not send empty arrays.

In terms of syntax, I think it keeps the document simple (if YAML, you just need `length:`). In terms of semantics, it means "a length validator with no config but defaults".
IIRC, Jackson should respect the order if using a linked hash map? 

Pedro Igor Craveiro e Silva

unread,
Apr 30, 2021, 8:55:11 AM4/30/21
to Marek Posolda, Vlastimil Elias, keyclo...@googlegroups.com
I can't argue much because I don't have a concrete use case either. However, advanced validations (I guess policies too) should be possible through SPI. 
IMO, they are quite related because it might happen that a version of a document implies a different behavior.

It is perfectly fine to have versioning like what you are proposing, but my feeling is that it is not the long-term solution. Which should be followed by a well-defined versioning of APIs too. For instance:

* Having a single endpoint (/foo) accepting different versions would become hard to maintain
* How do we properly communicate that a version is no longer supported or deprecated? Having /{version}/foo where a document set to v2 can not be used with API v1 is better than a single endpoint accepting everything. So we can make sure people migrated and properly remove support for v1. As well as possible return to clients the expected format for a version for their respective APIs.

From an API perspective, we have some of these concerns addressed in the rest guidelines.

So my take on this is that we should think more carefully about it.

Marek Posolda

unread,
May 3, 2021, 3:24:33 AM5/3/21
to Pedro Igor Craveiro e Silva, Vlastimil Elias, keyclo...@googlegroups.com

I see. You guys are probably right that more complex stuff will be better with the custom validator implementations.

Still not 100% sure if have one validator (or executor, or condition) per attribute is a good thing or rather limitation... :)

Ah, I see. You are concerned for admin REST API versioning.

For client policies, I was not thinking about any versioning for the admin REST API itself. I was assuming that for admin REST API, you will always need to use the PUT request with the JSON, which corresponds to current Keycloak version implementation. Otherwise error is thrown. +1 to your points as it seems that versioning for REST API should be better defined in the guidelines and hence it should be likely relevant for the "next admin REST API" generation, not for the current one IMO.

My concern for versioning was more related to the migration and the fact, that we save Client policies (or user profile as well AFAIK?) as attribute in the realm. Hence more the use-case like this:

1) Administrator creates some user profile (or client policy) in Keycloak 14
2) Now Keycloak is being migrated to version 16
3) In version 16, the JSON with the user profile attribute is read.
4) Now assuming that there was some change in the JSON representation format (EG. attribute "gui-order" was renamed to "guiOrder"). If you renamed the particular field in the Java representation object, then Keycloak will either throw the error or ignore the field.

Hence my concern was, that between step 3 and step 4, there will be some "JSON migration chain" to trigger the migration of JSON format from version 14 to version 16. Hence support versioning for migration between Keycloak versions not for admin REST API itself. And for this, just adding something like "keycloakVersion" field to the root of JSON should be fine for now IMO.

Marek

Reply all
Reply to author
Forward
0 new messages