Architecture discussion: serving MintMaker log files

[Mario Fernandes]

TLDR: the MintMaker team would like the community's input regarding an architectural decision

Hello, Konflux community!

We are achitecting MintMaker's integration with Konflux UI, and we are to make one important decision in the near future.

I would like to request your input, as it would be very valuable.

Problem

We are aiming to serve log files in the UI, so users can see how MintMaker runs went.

The challenge is how to do so while still having fine grained control over permissions: currently, the pipelineruns that execute our jobs are in MintMaker's namespace, and standard RBAC rules would only allow us to either allow access to all users outside our namespace or to none.

We would like to serve the log files following the same permissions as the component namespace. That is: users with permission in component A namespace should be able to see the logs for component A, but not for component B.

Solution 1: extra SAR verification step

In this solution we leave the log files in MintMaker's namespace, and perform an extra SAR check to verify if a certain user has access in a certain component namespace.

This extra step could be either in Konflux UI's side, before sending the request for the log file to MintMaker's backend, or in MintMaker's backend side, after receiving that request.

Solution 2: CRDs for log files + copying to component namespace

In this solution, once MintMaker's pipelineruns are finished, we copy their log files to the corresponding component namespace.

After this, standard RBAC rules will take care of the permissions as usual.

We could also create a CRD for MintMaker log files, to increase granularity in the access control.

Proposal

We are more propense to adopt solution 2, as it seem more natural and requires less infra to be maintained from MintMaker's side.

Please give us your input on this, it will be very invaluable for us to reach the best decision.

Thank you!

[Emil Natan]

Option 2 won't scale if you want to keep those logs for more than a couple of hours and it will have operational overload since you'll need to lock the pipelineruns until you store them and that'll interfere with other controllers which also store and prune the objects (including logs) from the cluster. And there are already at least two other systems storing those logs, so you do not want to duplicate that further.

Option 1 makes more sense to me in that regard. Another option is to reengineer MintMaker and make it run the pipelienrun rebuilding the image in the tenant namespace, unless you have some security considerations for running the MintMaker jobs in the MintMaker namespace.

I do not see how kubearchive (mentioned below) will solve the problem, at least not out of the box, as this is permissions issue. And I'll assume it is already used to store those pipelinerun logs anyway.

Emil

--
You received this message because you are subscribed to the Google Groups "Konflux CI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to konflux+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/konflux/64878b5f-1db5-4f93-ab1d-cb29ced462bdn%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Emil Natan

[Ralph Bean]

Have you thought about evolving mintmaker to *execute* the pipelineruns in the users' tenant namespaces?

This way, normal tekton-results *and* kubearchive rules would apply in terms of both storage and user rbac access. From the UI's point of view, it wouldn't need to know anything special. From a `kubectl` point of view, _users_ wouldn't need to know anything special.

Note, an interesting side effect of this is on the usage of secrets by mintmaker pipelineruns. The mintmaker pipeline runs would run in a context that has access to secrets provided by the user. For instance, if the user has a base image that they're pulling from quay.io/my-private-location, then today mintmaker cannot help by providing updates to that base image. The mintmaker namespace doesn't contain a pull secret for it. But, if the user has a base image in quay.io/my-private-location then they already have to provide a pull secret for their builds to work. The mintmaker pipeline could benefit from the presence of that very same secret in order to perform its renovate activities _for that tenant_.

The CRD for storing log files is definitely out. That would bring the cluster to its knees.

--

[Andrew McNamara]

@Gal Ben Haim and I were agreeing on that in a separate thread -- running in the tenant namespaces. It would require a change for MintMaker as mentioned. This feels architecturally consistent with Konflux. If a Renovate pipeline needs a subscription to access anything (images, RPMs, etc), those should already exist in tenant namespaces as they are needed for builds.

I feel like I chatted with @Adam Ormandy in the past about enabling functionality for users to request on-demand MintMaker runs in their namespace (essentially the ask above?) and then after that, being able to actually hand over control of the scheduling to users if we can have the Renovate pipelines run using something like an IntegrationTestScenario cron job. I don't always treat ITS's like they were intended -- I see them as a handy tool for managing eventing of PipelineRuns. I have suggested to users in the past that they can use these to run a cron in order to help maintain repositories including switching to a pull model of nudging and re-pinning to latest artifact.lock files for generic fetching.

MintMaker can also create a lot of load on deployments when there are many Components due to the sheer number of PipelineRuns and Events. This large concentrated workload likely makes many performance-related assumptions invalid. :)

Andrew McNamara

He / Him / His

To view this discussion visit https://groups.google.com/d/msgid/konflux/CAGq%2BFfzF3c9ZuQ3gL4_TrRbqW4GmGNN%3D-9oRhqpG_XNt0U59Uw%40mail.gmail.com.

[Ralph Bean]

> MintMaker can also create a lot of load on deployments when there are many Components due to the sheer number of PipelineRuns and Events. This large concentrated workload likely makes many performance-related assumptions invalid. :)

A nice thing about moving to a model where mintmaker always runs its renovate pipelines in the user tenant namespace is that those runs will be subject to the ResourceQuotas _of that tenant_. "large tier" users can have more while "extra small tier" users can have less.

> if we can have the Renovate pipelines run using something like an IntegrationTestScenario cron job. I don't always treat ITS's like they were intended -- I see them as a handy tool for managing eventing of PipelineRuns.

Heh, this feels like a hack to me. A necessary hack today b/c users don't have the direct control they might otherwise need. I don't think we should guide users to use ITS's like this.

But, imagine, if the users could create their own CronJob that created a DependencyUpdateCheck resource in their namespace that made mintmaker launch renovate pipelines for only their components in their namespace. The UI and kubectl have natural access to the user's namespace to surface all of that information. The cron schedule, the DUC, the pipelines, all of it.

[Adam Kaplan]

A hearty +1 to moving the Renovate PipelineRuns to the tenant namespace.

On the subject of credentials - we already have multiple vectors where Renovate would need access to user-provided information with container images and RPMs. This will only increase as we add more "supported" package manager ecosystems to Konflux, such as Maven repositories (Java) and Python. I can see a need for npm in light of recent supply chain attacks.

To view this discussion visit https://groups.google.com/d/msgid/konflux/CAGq%2BFfymgc_tGpaEMJq%3Dqh%3DHRavSMzUSsCtCHv%2BH91q4QJAZfA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam Kaplan

He/Him

Senior Principal Software Engineer

Red Hat

100 E. Davie Street

adam....@redhat.com    

[Julen Landa Alustiza]

I love the idea of being able to create the DependencyUpdateCheck on our tenant to trigger renovate jobs. There are cases where we know exactly when we want to run renovate, however we need to wait for the next loop and would be a huge help to be able to trigger renovate on a given component whenever we know it needs an update.

A tag following git submodule for upstream-downstream syncing is a pretty good example for those teams owning the upstream and the downstream repositories. We could set a webhook on the tag event upstream to create a DependencyUpdateCheck to trigger renovate for the downstream component.

I like this better than just being able to cronjob PipelineRuns, either with ITSs or with another mechanism. At the end you can always have a PipelineRun that render the DependencyUpdateCheck .

Regarding the logs, would not moving the pods to the user namespace have less performance impact than copying the logs over, and would not suit better to the already in place capabilities of the UI to read the tekton logs from kubearchive if needed/desired? Moving the PipelineRun inside the tenant namespace sounds like less headache long term than adding new CRDs and copying the logs, although we would need to revisit the quotas due to the load balancing of those PipelineRuns from the MintMaker namespace to the user's namespace

Julen,

To view this discussion visit https://groups.google.com/d/msgid/konflux/CAGq%2BFfymgc_tGpaEMJq%3Dqh%3DHRavSMzUSsCtCHv%2BH91q4QJAZfA%40mail.gmail.com.

[Mario Fernandes]

Hello everyone,

I first wanted to thank you all for the inputs. I really appreciate all the attention you gave so far.

Changing MintMaker to run its pipelines in user namespaces could bring important changes to the project, and they need to be investigated first.
For example, a few technical questions we would need to figure in this new context would be:

• how to set up the caching mechanism
• possible increase in the cardinality of observability metrics (Prometheus), which was already a sensitive point during implementation
• support from the MintMaker dev team might get jeopardized, in case we lack permission for the logs (we don't have access to the namespaces by default)
• MintMaker usage analysis could also become jeopardized for the same reason (we currently use the logs)
• we would have additional concerns regarding the management of secrets, as cross-namespace flow poses additional threats and challenges
• how to group Renovate runs, since we do not group by components; this could lead to a higher number of runs overall
• there would be one more reason for uncertainty in MintMaker runs (e.g., if a user is running out of resources/quotas)

Besides those, there are further questions that are more related to the experience that we are providing to the users:

• if we use user's quota and resources, they will probably want to configure this (and should have the right to do so), and it will be one more thing for them to maintain
• currently, users already face challenges having only to maintain a renovate.json file, we are unsure if it would be desirable to give users one more responsibility
• this change would probably make support effort increase

Overall, this change steps away from the current vision for the dependency management service, which is to be a more managed and abstracted experience.

I hope we managed to make it more concrete what the change would entail, and that it might not be so simple. This would be an architectural change, after all.

Now, investigating all those points and coming up with the best solutions will take some time and effort from the MintMaker team (first the spikes, then implementation, testing, etc).
However, we do have a pressing matter to solve: improve user experience with MintMaker and Konflux in general, and lower our cost and effort with user support.
The original proposal we made is to enable one important feature in the integration of MintMaker with Konflux UI.
And currently, having this integration ready soon is our priority.

All that said, we do not oppose making this architectural change (running pipelines in user namespaces) in the future. However, we see it as out of the scope of the current feature that we are planning.

[Greg Allen]

On 12/2/25 15:59, 'Mario Fernandes' via Konflux CI wrote:

Hello everyone,

I first wanted to thank you all for the inputs. I really appreciate all the attention you gave so far.

Changing MintMaker to run its pipelines in user namespaces could bring important changes to the project, and they need to be investigated first.
For example, a few technical questions we would need to figure in this new context would be:

• how to set up the caching mechanism
• possible increase in the cardinality of observability metrics (Prometheus), which was already a sensitive point during implementation
• support from the MintMaker dev team might get jeopardized, in case we lack permission for the logs (we don't have access to the namespaces by default)
• MintMaker usage analysis could also become jeopardized for the same reason (we currently use the logs)
• we would have additional concerns regarding the management of secrets, as cross-namespace flow poses additional threats and challenges
• how to group Renovate runs, since we do not group by components; this could lead to a higher number of runs overall
• there would be one more reason for uncertainty in MintMaker runs (e.g., if a user is running out of resources/quotas)

Besides those, there are further questions that are more related to the experience that we are providing to the users:

• if we use user's quota and resources, they will probably want to configure this (and should have the right to do so), and it will be one more thing for them to maintain
• currently, users already face challenges having only to maintain a renovate.json file, we are unsure if it would be desirable to give users one more responsibility
• this change would probably make support effort increase

Overall, this change steps away from the current vision for the dependency management service, which is to be a more managed and abstracted experience.

I hope we managed to make it more concrete what the change would entail, and that it might not be so simple. This would be an architectural change, after all.

Now, investigating all those points and coming up with the best solutions will take some time and effort from the MintMaker team (first the spikes, then implementation, testing, etc).
However, we do have a pressing matter to solve: improve user experience with MintMaker and Konflux in general, and lower our cost and effort with user support.
The original proposal we made is to enable one important feature in the integration of MintMaker with Konflux UI.
And currently, having this integration ready soon is our priority.

All that said, we do not oppose making this architectural change (running pipelines in user namespaces) in the future. However, we see it as out of the scope of the current feature that we are planning.

So I assume this goes back to the original proposal then?

If that's true, then you were proposing solution #2 which was pretty much deemed a non-starter.

Does that mean you are back to #1, or do you have another solution?

If you do go with solution #1, you might want to reconsider as that just becomes throw-away work if you eventually end up running in the tenant namespace as has been proposed.

To view this discussion visit https://groups.google.com/d/msgid/konflux/48e424db-30c1-4461-9452-a8d54725b8adn%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

-- 
-- Greg Allen
gal...@redhat.com

[Gal Ben Haim]

On Tue, Dec 2, 2025 at 10:59 PM 'Mario Fernandes' via Konflux CI <kon...@googlegroups.com> wrote:

Hello everyone,

I first wanted to thank you all for the inputs. I really appreciate all the attention you gave so far.

Changing MintMaker to run its pipelines in user namespaces could bring important changes to the project, and they need to be investigated first.
For example, a few technical questions we would need to figure in this new context would be:

• how to set up the caching mechanism
• possible increase in the cardinality of observability metrics (Prometheus), which was already a sensitive point during implementation
• support from the MintMaker dev team might get jeopardized, in case we lack permission for the logs (we don't have access to the namespaces by default)
• MintMaker usage analysis could also become jeopardized for the same reason (we currently use the logs)
• we would have additional concerns regarding the management of secrets, as cross-namespace flow poses additional threats and challenges
• how to group Renovate runs, since we do not group by components; this could lead to a higher number of runs overall
• there would be one more reason for uncertainty in MintMaker runs (e.g., if a user is running out of resources/quotas)

Besides those, there are further questions that are more related to the experience that we are providing to the users:

• if we use user's quota and resources, they will probably want to configure this (and should have the right to do so), and it will be one more thing for them to maintain
• currently, users already face challenges having only to maintain a renovate.json file, we are unsure if it would be desirable to give users one more responsibility
• this change would probably make support effort increase

I suggest starting a document with all those concerns / requirements so we can plan to address them. I don't see anything on that least which isn't feasible to achieve.

I would love to provide my assistance in architecting it.

Overall, this change steps away from the current vision for the dependency management service, which is to be a more managed and abstracted experience.

I hope we managed to make it more concrete what the change would entail, and that it might not be so simple. This would be an architectural change, after all.

Now, investigating all those points and coming up with the best solutions will take some time and effort from the MintMaker team (first the spikes, then implementation, testing, etc).
However, we do have a pressing matter to solve: improve user experience with MintMaker and Konflux in general, and lower our cost and effort with user support.
The original proposal we made is to enable one important feature in the integration of MintMaker with Konflux UI.
And currently, having this integration ready soon is our priority.

I'm sorry but I can't accept this argument. When Mintmaker was developed 1-2 years ago I made the same suggestion about the need to run the pipelines in the user's

namespace, since it provides the visibility you are now seeking. At that time my request was refused with the same argument (priorities, pressing matter to solve, etc).

I think we've reached the point where we need to focus on this core architecture change, as suggested by many in this thread, and run the dependency update pipelines in the user's namespace.

To view this discussion visit https://groups.google.com/d/msgid/konflux/48e424db-30c1-4461-9452-a8d54725b8adn%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

GAL bEN HAIM

KoNFLUX ARCHITECTURE

[Julen Landa Alustiza]

On Tue, Dec 2, 2025 at 9:59 PM 'Mario Fernandes' via Konflux CI <kon...@googlegroups.com> wrote:

Hello everyone,

we do have a pressing matter to solve: improve user experience with MintMaker and Konflux in general, and lower our cost and effort with user support.

If the main driver is not doing it right but reducing the pressure, why don't just link the splunk logs of each component's repository/branch discuss/architect the long term solution independently?

[Greg Allen]

You're not alone. I also made this suggestion in that timeframe, and got the same response.

To view this discussion visit https://groups.google.com/d/msgid/konflux/CADpdQO4h4s41c3VBYMebfLgeXqE%2B3EFj_n%3D-JT0XFPQ892fUog%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

-- 
-- Greg Allen
gal...@redhat.com

[John Casey]

I've added responses inline, but I want to address something up top here first:

We need to make sure we're maintaining a focus on overall user experience here, and not focusing on a single feature that might actually make clusters less stable and more vulnerable to load spikes. If the clusters are less stable, that undermines the whole point of this work.

On Tue, Dec 2, 2025 at 2:59 PM 'Mario Fernandes' via Konflux CI <kon...@googlegroups.com> wrote:

Hello everyone,

I first wanted to thank you all for the inputs. I really appreciate all the attention you gave so far.

Changing MintMaker to run its pipelines in user namespaces could bring important changes to the project, and they need to be investigated first.
For example, a few technical questions we would need to figure in this new context would be:

• how to set up the caching mechanism

i want to highlight this section, since it's the closest to where I've been working recently: 

• possible increase in the cardinality of observability metrics (Prometheus), which was already a sensitive point during implementation
• support from the MintMaker dev team might get jeopardized, in case we lack permission for the logs (we don't have access to the namespaces by default)
• MintMaker usage analysis could also become jeopardized for the same reason (we currently use the logs)

I believe we should still be able to review MintMaker logs via the logging aggregator, even if the pod logs aren't available in OpenShift. I might be wrong there, and if so I'd like to learn about it. Having stable, documented queries for things like this in a logging aggregation system would be preferable for SRE as well. But also, we should be working hard to reduce how often we have to go to logs to diagnose problems.

Which brings up metrics and monitoring. I believe we already expose some metrics with tenant labels intact, but if not we can always define RecordingRules to drop those labels, which would remove some important context...but it's context that we currently live without anyway. 

 

• we would have additional concerns regarding the management of secrets, as cross-namespace flow poses additional threats and challenges
• how to group Renovate runs, since we do not group by components; this could lead to a higher number of runs overall
• there would be one more reason for uncertainty in MintMaker runs (e.g., if a user is running out of resources/quotas)

I would think this would make the total quota landscape for a tenant more clear, not less so. Yes, MintMaker might become a source of transient spikes that threaten other things in the namespace. But on the other side, that load is happening in an out-of-sight part of the cluster today (from the tenant's perspective) and still has (other) performance implications that hit things like the API server.

 

To view this discussion visit https://groups.google.com/d/msgid/konflux/48e424db-30c1-4461-9452-a8d54725b8adn%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

John Casey

https://pronoun.is/he

Senior Principal Software Engineer

Technical Lead, SP Resilience Engineering

PnT Software Production

--

Your work-life balance is important to me. There is no need to respond to this outside of your normal work hours.

[Adam Ormandy]

Hello everyone
The reason why I asked Mario and the MM team to look into more granular access management, as I do not like how we solved access to the releases in rhtap-releng namespace, i.e. I can see pipelines of things I have no connection to.

That said, we can still do what we do for the release pipelines and that is, giving users read access to the pipelineRuns in the MM namespace and filter out the runs that belongs to the users content in the UI. Taking into account that the preferred long-term solution is to run MM pipelines in user namespaces, this would provide desired value without creating throw-away work and we already have a template we can follow. Because of that, I am inclined to move forward with this solution.

Adam