Quarkus Tekton Extension + PVCs + Native Mode?

[KimJohn Quinn]

So, I don't even know where to begin or 100% the root cause yet but I will share what we just wrestled with because it literally drove us to drinking…

We are integrating Tekton into our Quarkus platform and first wrestled with getting the Tekton Operator (v0.67.0) installed and configured how we like into EKS.  Finally, after weeks, we got it running the way we wanted.

Once we got Tekton running we built two pipelines - one using a PVC and another not. Applying these directly to Tekton, either in MiniKube or EKS, worked as expected.

Finally, we moved into Quarkus and pulled in the Quarkus Tekton extension (originally v1.0.0 then v1.0.1).  We built out a service that created the equivalent pipeline run we applied manually and then used our service to kick off Tekton.

This is where it gets interesting...

1. We could apply the Tekton pipeline runs, etc. directly into the cluster or minikube and they worked.
   
   

2. We could invoke our Tekton service locally, as a JVM instance, pointing to either minikube or the EKS cluster and both pipelines worked.
   
   

3. When we deployed the Tekton service as a native instance the pipeline not using the PVC worked but the one using a PVC failed with this error below. We also validated it was specific to the PVC by changing the other pipeline to use a PVC and got the same error. In all cases, we never even got to Tekton because of this.

"message": "Failure executing: POST at: https://172.20.0.1:443/apis/tekton.dev/v1beta1/namespaces/pipeline/pipelineruns. Message: admission webhook \"webhook.pipeline.tekton.dev\" denied the request: mutation failed: cannot decode incoming new object: json: unknown field \"claims\". Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=admission webhook \"webhook.pipeline.tekton.dev\" denied the request: mutation failed: cannot decode incoming new object: json: unknown field \"claims\", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=BadRequest, status=Failure, additionalProperties={})."

What we went through:

1. We opened up all of our ClusterRoles (temporarily) thinking maybe we configured them wrong - did not work.
   
   

2. We downgraded the Tekton Operator to v0.66 - did not work.
   
   

3. We switched from the Tekton Operator to regular Pipelines - did not work
   
   

4. We thought since the error was referring to either a missing/added property that it might be the Kubernetes client or the Tekton extension in the client, and its models, so we overrode all the dependencies across the board (it did change from the ArrayOrString param value to ParamValue) - did not work.
   
   

5. We dumped out the spec being sent to Tekton and found the “claims” property which in the spec was being sent over was an empty array [] and we nulled it out - this worked! Again, this worked as a JVM instance only natively did it fail.

We hacked the code and essentially went from the commented section to the below section:

/*.item(() -> tektonClient.v1beta1().pipelineRuns()

                    .inNamespace(tektonConfig.namespace())

                    .resource(r).create())

                .invoke(pr -> log.debug("Applied pipeline run")))

                .runSubscriptionOn(Infrastructure.getDefaultWorkerPool())*/

                .item(() ->

                {

                    for (WorkspaceBinding b: r.getSpec().getWorkspaces())

                    {

                        if (b.getVolumeClaimTemplate() != null)

                        {

                            b.getVolumeClaimTemplate().getSpec().getResources().setClaims(null);

                        }

                    }

                    String pipelineRun = r.toString();

                    return tektonClient.v1beta1().pipelineRuns()

                        .inNamespace(tektonConfig.namespace())

                        .resource(r).create();

                })

Hypothesis:

Since we could apply the pipelines directly we felt the pipelines and Tekton were not a problem, running locally or against the cluster  in JVM mode worked so the code did not seem to be a problem,  running natively using the same service only the PVC pipeline failed - so, it was either something cluster related or specific to running natively. 

The fact we had to manually change that specific property to null instead of [] makes us think it is something specific to the native instance, similar to when we need to apply RegisterForReflection in specific cases.

Why this was difficult to debug:

This was extremely difficult to debug initially because there were a lot of moving parts.  Even now we do not know for sure if that is the actual problem.  We had to contend with reconciling the versions of the extension, the Kubernetes client and its extensions (i.e. Kubernetes Client + Tekton extension + models), Tekton itself which has been challenging to get working outside of Quarkus and so we thought maybe it was something there related to the CRDs or differences between the operator and proper install.

It would be really nice if some extensions listed what versions they reference (ie Kubernetes Client vX.y.z) because they would have saved us a lot of pain digging into all the pieces especially when it deals with CRDs or moving systems, like Tekton.

[Max Rydahl Andersen]

Hi KimJohn,

Does not sound fun.

I'm struggling to fully grok what/where you actually changed code and where you think things should change?

I grok its some diff in behaviour from java mode and native - we should track that down because that shouldn't happen.

Could you open issue with some more details; or even PR with the code change that I think is in one of the Quarkus or Quarkiverse extension(s)?

/max
https://xam.dk/about

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/fc11352f-ed0f-41c9-9dda-3ffc7a00d230n%40googlegroups.com.

[KimJohn Quinn]

Hello Max.

That was just a write-up before we lost the details.  We can't say for sure if it is native or JVM (but native is the only outlier and where it does not work with PVCs, all other cases do work as expected when not native).

The change we made was in our code where the Tekton client is being used to create the PipelineRun.  We manually "nulled" out the "claims" property.  Because the same code and extensions work in every other scenario we don't "think" it is the extension itself or even what it has pulled in for the Kubernetes/Tekton extension.  Also, sounds terrible, but this was a complete shot in the dark.

The ask from our side, was in some cases, the Tekton Quarkus Extension for example, it would be nice to have a quick way to know what it uses rather than digging through the POMs.  Because we thought originally it was either a Kubernetes issue or the Fabric8 client itself (maybe the models were out of whack and there were newer releases) we spent a lot of time reconciling the versions and trying to grab the latest and override the extension thinking it was that.

We are still doing more investigation but wanted to share that experience...

 

[Ioannis Canellos]

Just to make sure I am getting things right.

You are building a Quarkus application that uses the `quarkus-tekton-client` from quarkiverse to talk to tekton.

When you build this app as native and run pipelines with pvcs you are hitting issues, due to `claims` containing an empty array.

Am I close?

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAF1TLv4KkEhPTHE8oy4Rs4hzTf3T6WSr6oLRw8oLqHuo%3DY4BWQ%40mail.gmail.com.

--

Ioannis Canellos

[KimJohn Quinn]

Yes, you are correct.

Here is what we think/ know:

• Applying the Tekton pipelines directly to the cluster work.  So, the operator and the pipelines seem to be correct as well as any cluster related stuff like security, etc.
  
  
• Using the JVM instance of our service against the cluster works, both the PVC and non-PVC.
  
• 
  
• Using the Native instance of our service the non-PVC pipeline works.
  
  
• Using the Native instance of our service the PVC pipeline fails with that "claims" error.
  
  
• At least for us, it seems specific to the PVC.  If we change the working pipeline to use a PVC it fails with the same error.
  
  
• Nulling out the "claim" before submitting it to Tekton seems to resolve the issue.  Where before we never even made it to Tekton because it was invalid we are now running pipelines.
  

Since everything works before we go native we dont think it is necessarily the Tekton extension itself, the version of the Fabric8 client/extension/models (this is what we originally thought and that led us down a rabbit hole). Our assumption is that the spec being sent over is the same...

[Ioannis Canellos]

Who is responsible for creating the Pipeline/Run used by the Quarkus application?

Are they read from files and somehow the deserialization processes create the weird `claims` fied?

--

Ioannis Canellos

[KimJohn Quinn]

First we load a template from a file, which has almost nothing in it, then using the DSL of the Tekton client we enrich that with parameters and taints.

Also, I hope we are not coming off as upset over this - we are not ;)  If this is the biggest issue we have faced in 3 years of continual updates, multiple native releases per day, and most other issues, especially if dealing with Quarkus pinpointable in a few hours tops, I'll take it - it was a little more challenging to "think" we found a solution because there were a lot of moving parts.

[KimJohn Quinn]

One more thing, if i use the Tekton client and output the YAML of the run it looks exactly like what we send over normally (short of the extra parameters).

[Ioannis Canellos]

So, this is not a deserialization issue, right?

You create the claim template yourselves? If this is the case would you be able to provide a reproducer, so that I can try it out, or guide me through the exact steps?

--

Ioannis Canellos

[KimJohn Quinn]

I will try to outline the exact steps and/or get you a reproducer.  You gave me an idea yesterday, what if it is something with loading the run then enriching it vs. creating it completely I would like to try.  It will probably be some time next week though at the earliest I can get  you something...

Thanks.

[Ioannis Canellos]

FWIW, in dekorate we do have a collection of dekorators related to tekton, that they can be used directly with tekton client builders via the `.accept()` method.

They may or may not be of use for use case, but they will certainly give you more ideas on how to manipulate tekton resources.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/21e8d3f1-b629-4cfa-9d8c-2b87ea061a2dn%40googlegroups.com.

--

Ioannis Canellos

[Charles Moulliard]

Hi KimJohn 

Can you share please the PipelineRun generated as I would like to understand why the Tekton webhook when it validates the incoming request complains about the unknown field "cannot decode incoming new object: json: unknown field claims" ?

AFAIK, the volumeClaimTemplate (see Tekton api doc) corresponds to this standard k8s API object: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/

Cheers

Charles