Quarkus Extension approach proposal for integration products like Keycloak
Dominik Guhr
Hey Quarkus team! Hope you are doing well :) We from the Keycloak team have a proposal created and would love to hear your thoughts and feedback on it. So without making this post even longer, here it comes:
Problem statement
A good chunk of Keycloaks value comes from its extensibility and its ability to integrate well in various environments, from on-prem to k8s, from AWS to OpenShift, from Oracle to Postgres.
Currently, Keycloak is very opinionated about the extensions available out of the box in its distribution for various reasons listed below. This is needed, but makes life hard for developers and users of Keycloak when they want to use additional extensions that are already available in the Quarkus universe.
Keycloak itself is a quarkus extension under the covers, and uses the mutable-jar approach to provide great flexibility when addressing the different environments it runs in.
Keycloak relies on the re-augmentation step to provide an optimized runtime with fast startup and low memory footprint. Mutable-jar applications do not allow extensions other than those available at compile time to be installed when re-augmenting the application.
Keycloak as a layered product can not include Quarkus extensions OOTB, other than those that are strictly necessary to support its core capabilities. Otherwise, this can cause:
too big distribution size
High maintenance cost for the core team
Bad UX / Hard to choose what extensions should be available by default as they are environment / use case specific (e.g.: you might want or not a JAX-RS Client, centralized logging, …)
Higher risk of e.g. CVEs -> more dependencies are included and available from the distribution even when the environment Keycloak runs in only needs a subset of, for example, database drivers.
Use cases / stories
This section should give you a general idea of what we have in mind mid- to long term, trying not to go into technical solution details but exploring the problem space. It is not considered complete and likely to be extended/changed in the future. See “Action points” for more specific points we'd love to hear your thoughts about.
As a Keycloak developer I want to be able to leverage Quarkus extensions and use them when implementing my custom providers so that I can take advantage of the available extensions from the Quarkus ecosystem.
For a better developer experience, developers should be able to:
Choose an extension from Quarkus ecosystem
Write their custom providers using the extensions
Package their custom providers and deploy to the server
Run re-augmentation to create a new “closed world assumption” based on the chosen extensions and providers
Have their custom providers and Quarkus extensions installed and available at runtime
As a Keycloak developer I want to be able to leverage Quarkus extensions and use them when implementing my custom providers ( == "custom keycloak extensions") so that I can take advantage of the available extensions from the Quarkus ecosystem.
As a platform team member of a “customer”, I want to tailor our platforms Keycloak component specific to my operational environment, with the least complexity possible, so that the risk of being affected by e.g. CVEs from dependencies not used in my environment is lowered and compliance is fullfilled (some historical context: https://github.com/keycloak/keycloak-community/pull/328/files#r735664241.. )
I want one reliable, secure “marketplace” to go to to tailor Keycloak to my specific needs
I want to know what is “supported upstream by the core project” and what is not.
Example: The marketplace offers extensions for different databases, where some are community supported, meaning that there is no direct CI run to test Keycloaks behaviour when using these databases, some are “core extensions” (see story below).
Example 2: The marketplace offers extensions for being able to integrate Keycloak with different observability solutions.
Example 3: The marketplace offers extensions for being able to connect Keycloak to eventing systems.
Example 4: The marketplace offers extensions for being able to integrate Keycloak with different centralized logging solutions.
…
I want to download a packaged “distribution” containing the tailored Keycloak+extensions package. (Maybe even pre-optimized as a fast-jar instead of a mutable-jar)
I want to download/create a git repository of a developer boilerplate ready for extending Keycloak with my internal extensions.
(Later:) I want to create the tailored Keycloak distribution as a docker image directly in e.g. my orgs quay.io registry account
(Even later: ) I want my tailored “closed world” Keycloak to be downloaded / uploaded to a container registry from the marketplace as a native image.
As a maintainer of Keycloak core, I want to strip away some features of the current monolithic distribution into available, supported “core extensions”. The “software formerly known as distribution” will still be there, but will consist of a lightweight “core Keycloak” codebase, handling AuthN and AuthZ (our core domain), and a set of supported core extensions packaged into it. Our pipelines will still test this full package, thus making it “upstream supported”, but we will have the flexibility to remove stuff from “core” that is not strictly necessary there, to reduce maintenance cost and increase focus on our core capabilities.
E.g. some social identity providers like Twitter, ...
E.g. support for LDAP connections (to not necessarily have JNDI enabled for our users, that is needed to communicate with LDAP and enabled by default atm)
…
As a platform team member of a “customer”, I want to be able to leverage the huge quarkiverse ecosystem by installing my needed extensions, to tailor Keycloak specific to my platform's Keycloak component needs.
E.g. by invoking kc extensions install <quarkiverse-extension/MAR-with-metadata-for-compatibility> followed by a kc build to activate the extension
As a platform team member of a “customer”, I want to be able to search the huge Quarkiverse ecosystem, to find the available extensions together with their support status for the current Keycloak version.
E.g. by invoking kc extensions search <query>
The current state
People are actually able to use some Quarkiverse extensions in Keycloak, but in an experimental, non-convenient and unsupported way:
Download the “<extension>-deployment.jar” and the “<extension>.jar” and all the transient jars the extensions use
Move them to Keycloaks “providers” folder
Run a “kc.sh build” (this command invokes a re-augmentation of the mutable jar)
Start Keycloak. The extension is registered, a new closed world assumption created. Or not, when anything goes wrong (e.g. quarkiverse.logging vs quarkus.logging)
Drawbacks:
There is no ruleset/metadata to check if a (set of) extensions are compatible with each other and with the extensions Keycloak uses OOTB (e.g. quarkiverse-logging-json and quarkus-logging-json can not be used together).
The whole process is experimental / non-idiomatic, the augmentation guide states “ Augmentation can only use extensions that were present during compile time.”
Potential Solution
One possible solution to make it possible to add additional extensions to a mutable application like Keycloak may be to introduce the concept of a MAR (Mutable Application Archive) that relies on the JAR format and adds a well-defined, structured “Mutable Application extension” contract, that describes the extensions that should be installed during re-augmentation and the user code using these extensions.
Action points:
In order to achieve this vision, we would love to talk with you about:
Your general feedback of the viability of this idea, basically it’s kind of a “layered framework approach”.
The MAR contract for the necessary metadata to integrate quarkus extensions into Keycloak / layered products
Figure out how to dynamically enable or disable extensions with mutable jars
…? We want your feedback :)
Alexey Loubyansky
--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAChHV7BfxDXcTokFT5g0tEaJ2-ZMQWyjLqFQc4tJR9G8y3tppA%40mail.gmail.com.
Max Rydahl Andersen
> Great to hear this kind of ideas. I think you could consider building a
> Keycloak Quarkus platform. The concept of a platform is described in [1].
> You can include your own metadata in the platform descriptor and even in
> each individual extension.
> Quarkus dev tools by default target io.quarkus.platform. There is also
> com.redhat.quarkus.platform and we also heard of other companies building
> their own platforms.
>
> About the (re)assembly part, it sounds like we’ll need to be able to
> resolve dependencies of the selected extensions. As a result, it’s not
> about re-augmentation anymore, afaiu, but an ordinary Quarkus build
> process. It doesn’t mean however we’ll need a Maven/Gradle project for it,
> Quarkus bootstrap could be used directly to build an app.
>>
>> too big distribution size
>> -
>> High maintenance cost for the core team
>> Bad UX / Hard to choose what extensions should be available by default
>> as they are environment / use case specific (e.g.: you might want or not a
>> JAX-RS Client, centralized logging, …)
>> Higher risk of e.g. CVEs -> more dependencies are included and
>> available from the distribution even when the environment Keycloak runs in
>> only needs a subset of, for example, database drivers.
>>
>> Use cases / stories
>>
>> This section should give you a general idea of what we have in mind mid-
>> to long term, trying not to go into technical solution details but
>> exploring the problem space. It is not considered complete and likely to be
>> extended/changed in the future. See “Action points” for more specific
>> points we'd love to hear your thoughts about.
>>
>> As a Keycloak developer I want to be able to leverage Quarkus extensions
>> and use them when implementing my custom providers so that I can take
>> advantage of the available extensions from the Quarkus ecosystem.
>>
>> For a better developer experience, developers should be able to:
>>
>>
>> Choose an extension from Quarkus ecosystem
>> Write their custom providers using the extensions
>> Package their custom providers and deploy to the server
>> Run re-augmentation to create a new “closed world assumption” based on
>> the chosen extensions and providers
>> Have their custom providers and Quarkus extensions installed and
>> available at runtime
>>
>>
>> As a Keycloak developer I want to be able to leverage Quarkus extensions
>> and use them when implementing my custom providers ( == "custom keycloak
>> extensions") so that I can take advantage of the available extensions from
>> the Quarkus ecosystem.
>>
>> As a platform team member of a “customer”, I want to tailor our platforms
>> Keycloak component specific to my operational environment, with the least
>> complexity possible, so that the risk of being affected by e.g. CVEs from
>> dependencies not used in my environment is lowered and compliance is
>> fullfilled (some historical context:
>> https://github.com/keycloak/keycloak-community/pull/328/files#r735664241..
>> )
>>
>> I want one reliable, secure “marketplace” to go to to tailor Keycloak
>> to my specific needs
>> I want to know what is “supported upstream by the core project” and
>> what is not.
>> Example: The marketplace offers extensions for different databases,
>> where some are community supported, meaning that there is no direct CI run
>> to test Keycloaks behaviour when using these databases, some are “core
>> extensions” (see story below).
>> Example 2: The marketplace offers extensions for being able to
>> integrate Keycloak with different observability solutions.
>> Example 3: The marketplace offers extensions for being able to
>> connect Keycloak to eventing systems.
>> Example 4: The marketplace offers extensions for being able to
>> integrate Keycloak with different centralized logging solutions.
>>
>> …
>> -
>> I want to download a packaged “distribution” containing the tailored
>> Keycloak+extensions package. (Maybe even pre-optimized as a fast-jar
>> instead of a mutable-jar)
>> I want to download/create a git repository of a developer boilerplate
>> ready for extending Keycloak with my internal extensions.
>> (Later:) I want to create the tailored Keycloak distribution as a
>> docker image directly in e.g. my orgs quay.io registry account
>> (Even later: ) I want my tailored “closed world” Keycloak to be
>> downloaded / uploaded to a container registry from the marketplace as a
>> native image.
>>
>>
>> As a maintainer of Keycloak core, I want to strip away some features of
>> the current monolithic distribution into available, supported “core
>> extensions”. The “software formerly known as distribution” will still be
>> there, but will consist of a lightweight “core Keycloak” codebase, handling
>> AuthN and AuthZ (our core domain), and a set of supported core extensions
>> packaged into it. Our pipelines will still test this full package, thus
>> making it “upstream supported”, but we will have the flexibility to remove
>> stuff from “core” that is not strictly necessary there, to reduce
>> maintenance cost and increase focus on our core capabilities.
>>
>> E.g. some social identity providers like Twitter, ...
>> E.g. support for LDAP connections (to not necessarily have JNDI
>> enabled for our users, that is needed to communicate with LDAP and enabled
>> by default atm)
>>
>> …
>>
>> As a platform team member of a “customer”, I want to be able to leverage
>> the huge quarkiverse ecosystem by installing my needed extensions, to
>> tailor Keycloak specific to my platform's Keycloak component needs.
>>
>> E.g. by invoking kc extensions install
>> <quarkiverse-extension/MAR-with-metadata-for-compatibility> followed
>> by a kc build to activate the extension
>>
>>
>> As a platform team member of a “customer”, I want to be able to search
>> the huge Quarkiverse ecosystem, to find the available extensions together
>> with their support status for the current Keycloak version.
>>
>>
>> E.g. by invoking kc extensions search <query>
>>
>> The current state
>>
>> People are actually able to use some Quarkiverse extensions in Keycloak,
>> but in an experimental, non-convenient and unsupported way:
>>
>> Download the “<extension>-deployment.jar” and the “<extension>.jar”
>> and all the transient jars the extensions use
>> Move them to Keycloaks “providers” folder
>> Run a “kc.sh build” (this command invokes a re-augmentation of the
>> mutable jar)
>> Start Keycloak. The extension is registered, a new closed world
>> assumption created. Or not, when anything goes wrong (e.g.
>> quarkiverse.logging vs quarkus.logging)
>>
>>
>> Drawbacks:
>>
>> There is no ruleset/metadata to check if a (set of) extensions are
>> compatible with each other and with the extensions Keycloak uses OOTB (e.g.
>> quarkiverse-logging-json and quarkus-logging-json can not be used
>> together).
>> The whole process is experimental / non-idiomatic, the augmentation
>> guide states “ Augmentation can only use extensions that were present
>> during compile time.”
>>
>>
>> Potential Solution
>>
>> One possible solution to make it possible to add additional extensions to
>> a mutable application like Keycloak may be to introduce the concept of a
>> MAR (Mutable Application Archive) that relies on the JAR format and adds
>> a well-defined, structured “Mutable Application extension” contract, that
>> describes the extensions that should be installed during re-augmentation
>> and the user code using these extensions.
>>
>> Action points:
>>
>> In order to achieve this vision, we would love to talk with you about:
>>
>> Your general feedback of the viability of this idea, basically it’s
>> kind of a “layered framework approach”.
>> The MAR contract for the necessary metadata to integrate quarkus
>> extensions into Keycloak / layered products
>> Figure out how to dynamically enable or disable extensions with
>> mutable jars
>> …? We want your feedback :)
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Quarkus Development mailing list" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to quarkus-dev...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/quarkus-dev/CAChHV7BfxDXcTokFT5g0tEaJ2-ZMQWyjLqFQc4tJR9G8y3tppA%40mail.gmail.com
>> .
>
> --
> You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
Max Rydahl Andersen
just a strawman suggestion here ... would it not be more interesting keycloak
does not require custom loading of jars (wether quarkus or not) and instead
make extensions viable by running several processes that communicate over a
a grpc/rest api?
Then we would avoid this whole class of issues?
/max
--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAChHV7BfxDXcTokFT5g0tEaJ2-ZMQWyjLqFQc4tJR9G8y3tppA%40mail.gmail.com.
/max
https://xam.dk/about
Alexey Loubyansky
On 17 Jul 2022, at 16:02, Alexey Loubyansky wrote:
> Great to hear this kind of ideas. I think you could consider building a
> Keycloak Quarkus platform. The concept of a platform is described in [1].
> You can include your own metadata in the platform descriptor and even in
> each individual extension.
> Quarkus dev tools by default target io.quarkus.platform. There is also
> com.redhat.quarkus.platform and we also heard of other companies building
> their own platforms.
>
> About the (re)assembly part, it sounds like we’ll need to be able to
> resolve dependencies of the selected extensions. As a result, it’s not
> about re-augmentation anymore, afaiu, but an ordinary Quarkus build
> process. It doesn’t mean however we’ll need a Maven/Gradle project for it,
> Quarkus bootstrap could be used directly to build an app.
as in compile java code or just about packaging the jars ?
Stuart Douglas
Hey Quarkus team! Hope you are doing well :) We from the Keycloak team have a proposal created and would love to hear your thoughts and feedback on it. So without making this post even longer, here it comes:
Problem statement
A good chunk of Keycloaks value comes from its extensibility and its ability to integrate well in various environments, from on-prem to k8s, from AWS to OpenShift, from Oracle to Postgres.
Currently, Keycloak is very opinionated about the extensions available out of the box in its distribution for various reasons listed below. This is needed, but makes life hard for developers and users of Keycloak when they want to use additional extensions that are already available in the Quarkus universe.
Keycloak itself is a quarkus extension under the covers, and uses the mutable-jar approach to provide great flexibility when addressing the different environments it runs in.
Keycloak relies on the re-augmentation step to provide an optimized runtime with fast startup and low memory footprint. Mutable-jar applications do not allow extensions other than those available at compile time to be installed when re-augmenting the application.
Keycloak as a layered product can not include Quarkus extensions OOTB, other than those that are strictly necessary to support its core capabilities. Otherwise, this can cause:
too big distribution size
High maintenance cost for the core team
Bad UX / Hard to choose what extensions should be available by default as they are environment / use case specific (e.g.: you might want or not a JAX-RS Client, centralized logging, …)
Higher risk of e.g. CVEs -> more dependencies are included and available from the distribution even when the environment Keycloak runs in only needs a subset of, for example, database drivers.
Use cases / stories
This section should give you a general idea of what we have in mind mid- to long term, trying not to go into technical solution details but exploring the problem space. It is not considered complete and likely to be extended/changed in the future. See “Action points” for more specific points we'd love to hear your thoughts about.
As a Keycloak developer I want to be able to leverage Quarkus extensions and use them when implementing my custom providers so that I can take advantage of the available extensions from the Quarkus ecosystem.
For a better developer experience, developers should be able to:
Choose an extension from Quarkus ecosystem
Write their custom providers using the extensions
Package their custom providers and deploy to the server
Run re-augmentation to create a new “closed world assumption” based on the chosen extensions and providers
Have their custom providers and Quarkus extensions installed and available at runtime
As a Keycloak developer I want to be able to leverage Quarkus extensions and use them when implementing my custom providers ( == "custom keycloak extensions") so that I can take advantage of the available extensions from the Quarkus ecosystem.
As a platform team member of a “customer”, I want to tailor our platforms Keycloak component specific to my operational environment, with the least complexity possible, so that the risk of being affected by e.g. CVEs from dependencies not used in my environment is lowered and compliance is fullfilled (some historical context: https://github.com/keycloak/keycloak-community/pull/328/files#r735664241.. )
I want one reliable, secure “marketplace” to go to to tailor Keycloak to my specific needs
I want to know what is “supported upstream by the core project” and what is not.
Example: The marketplace offers extensions for different databases, where some are community supported, meaning that there is no direct CI run to test Keycloaks behaviour when using these databases, some are “core extensions” (see story below).
Example 2: The marketplace offers extensions for being able to integrate Keycloak with different observability solutions.
Example 3: The marketplace offers extensions for being able to connect Keycloak to eventing systems.
Example 4: The marketplace offers extensions for being able to integrate Keycloak with different centralized logging solutions.
…
I want to download a packaged “distribution” containing the tailored Keycloak+extensions package. (Maybe even pre-optimized as a fast-jar instead of a mutable-jar)
I want to download/create a git repository of a developer boilerplate ready for extending Keycloak with my internal extensions.
(Later:) I want to create the tailored Keycloak distribution as a docker image directly in e.g. my orgs quay.io registry account
(Even later: ) I want my tailored “closed world” Keycloak to be downloaded / uploaded to a container registry from the marketplace as a native image.
As a maintainer of Keycloak core, I want to strip away some features of the current monolithic distribution into available, supported “core extensions”. The “software formerly known as distribution” will still be there, but will consist of a lightweight “core Keycloak” codebase, handling AuthN and AuthZ (our core domain), and a set of supported core extensions packaged into it. Our pipelines will still test this full package, thus making it “upstream supported”, but we will have the flexibility to remove stuff from “core” that is not strictly necessary there, to reduce maintenance cost and increase focus on our core capabilities.
E.g. some social identity providers like Twitter, ...
E.g. support for LDAP connections (to not necessarily have JNDI enabled for our users, that is needed to communicate with LDAP and enabled by default atm)
…
As a platform team member of a “customer”, I want to be able to leverage the huge quarkiverse ecosystem by installing my needed extensions, to tailor Keycloak specific to my platform's Keycloak component needs.
E.g. by invoking kc extensions install <quarkiverse-extension/MAR-with-metadata-for-compatibility> followed by a kc build to activate the extension
As a platform team member of a “customer”, I want to be able to search the huge Quarkiverse ecosystem, to find the available extensions together with their support status for the current Keycloak version.
E.g. by invoking kc extensions search <query>
The current state
People are actually able to use some Quarkiverse extensions in Keycloak, but in an experimental, non-convenient and unsupported way:
Download the “<extension>-deployment.jar” and the “<extension>.jar” and all the transient jars the extensions use
Move them to Keycloaks “providers” folder
Run a “kc.sh build” (this command invokes a re-augmentation of the mutable jar)
Start Keycloak. The extension is registered, a new closed world assumption created. Or not, when anything goes wrong (e.g. quarkiverse.logging vs quarkus.logging)
Drawbacks:
There is no ruleset/metadata to check if a (set of) extensions are compatible with each other and with the extensions Keycloak uses OOTB (e.g. quarkiverse-logging-json and quarkus-logging-json can not be used together).
The whole process is experimental / non-idiomatic, the augmentation guide states “ Augmentation can only use extensions that were present during compile time.”
Potential Solution
One possible solution to make it possible to add additional extensions to a mutable application like Keycloak may be to introduce the concept of a MAR (Mutable Application Archive) that relies on the JAR format and adds a well-defined, structured “Mutable Application extension” contract, that describes the extensions that should be installed during re-augmentation and the user code using these extensions.
Action points:
In order to achieve this vision, we would love to talk with you about:
Your general feedback of the viability of this idea, basically it’s kind of a “layered framework approach”.
The MAR contract for the necessary metadata to integrate quarkus extensions into Keycloak / layered products
Figure out how to dynamically enable or disable extensions with mutable jars
…? We want your feedback :)
--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAChHV7BfxDXcTokFT5g0tEaJ2-ZMQWyjLqFQc4tJR9G8y3tppA%40mail.gmail.com.