Data Driven Conformance for e2e testing K8s POD APIs

[Hippie Hacker]

The CNCF is contracting help to write missing e2e tests for stable apis, specifically ones that make sense to promote to conformance tests.

POD API endpoints are extremely feature rich, have multiple implementations, and are used by everybody. This combination makes them of particular interest as we collect user journey data to prioritize which tests to write next. [SIG-Architecture May 10th discussion] 

The APISnoop team would like provide a list of popular POD K8s API endpoints and additionally the parameters/responses used by real world API consumers at the next sig-node meeting on June 13th.

To do so, we need to collect audit-logs while driving applications that use the POD API to it's fullest.

I'm asking for help in identifying which applications we should focus on, in addition to help driving those applications to give us meaningful POD endpoint/verb parameters/responses.

Projects with existing e2e tests would be best, but we are willing to manually drive them if doing so can produce actionable data (based on real user journeys) to drive our conformance efforts.

Please respond to this thread, or directly to https://github.com/cncf/apisnoop/issues/16

Cheers,

Hippie Hacker

[SIG-Architecture May 10th discussion] https://www.youtube.com/watch?v=gCTLdzsqzaY&feature=youtu.be&list=PL69nYSiGNLP2m6198LaLN6YahX7EEac5g&t=3398

[Zhen Wang]

When you say POD API, are you talking about the rest API operations (since you mentioned "endpoints")? Is pod spec in scope for this discussion?

Best

-Zhen

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-node+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-node@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/06cebba2-b5a7-4417-81cd-0a17a0bc5214%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Hippie Hacker]

The current scope is for the e2e tests that can  be promoted to confomance tests.

Our current data collection occurs via audit logging via api server.

I'd definitely like to hear options for collecting audit style data for pod spec.

On Fri, Jun 1, 2018, 5:17 AM Zhen Wang <zh...@google.com> wrote:

When you say POD API, are you talking about the rest API operations (since you mentioned "endpoints")? Is pod spec in scope for this discussion?

Best

-Zhen

On Wed, May 30, 2018 at 9:51 PM, Hippie Hacker <h...@ii.coop> wrote:

The CNCF is contracting help to write missing e2e tests for stable apis, specifically ones that make sense to promote to conformance tests.

POD API endpoints are extremely feature rich, have multiple implementations, and are used by everybody. This combination makes them of particular interest as we collect user journey data to prioritize which tests to write next. [SIG-Architecture May 10th discussion] 

The APISnoop team would like provide a list of popular POD K8s API endpoints and additionally the parameters/responses used by real world API consumers at the next sig-node meeting on June 13th.

To do so, we need to collect audit-logs while driving applications that use the POD API to it's fullest.

I'm asking for help in identifying which applications we should focus on, in addition to help driving those applications to give us meaningful POD endpoint/verb parameters/responses.

Projects with existing e2e tests would be best, but we are willing to manually drive them if doing so can produce actionable data (based on real user journeys) to drive our conformance efforts.

Please respond to this thread, or directly to https://github.com/cncf/apisnoop/issues/16

Cheers,

Hippie Hacker

[SIG-Architecture May 10th discussion] https://www.youtube.com/watch?v=gCTLdzsqzaY&feature=youtu.be&list=PL69nYSiGNLP2m6198LaLN6YahX7EEac5g&t=3398

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.

[Hippie Hacker]

> On Fri, Jun 1, 2018, 5:17 AM Zhen Wang <zh...@google.com> wrote:
>>
>> When you say POD API, are you talking about the rest API operations (since
>> you mentioned "endpoints")? Is pod spec in scope for this discussion?

I see there is an open issue to add support for Node audit logging[1]
that I think would be particularly relevant.

[1] https://github.com/kubernetes/kubernetes/issues/39441

[Zhen Wang]

That one is for system journald logs, not API audit logging.

Best

-Zhen

[Zhen Wang]

Hi Chris and Rohan,

Aish (aishs@) is currently driving the effort of increasing conformance test coverage for POD APIs. Here is the POD API end points that she is going to start looking into. The reason is mostly because those are the ones supported in client-go library. I just chatted with her and she will follow up with you shortly.

As for the POD APIs, you can refer to kubernetes reference doc. For POD APIs, there are 2 parts:

• The REST API end points, like write operations, that I think APISnoop is currently checking.
• The specs, like PodSpec, that I think APISnoop currently do not check. As an example, you can think about this as the body of the HTTP request when posting to /api/v1/namespaces/{namespace}/pods. I think potentially APISnoop may be extended to check those as next steps. And checking the specs are actually more meaningful when talking about pod behaviors, because kubelet will do things according to the specs.

Best

-Zhen

[Rohan Fletcher]

The REST API end points, like write operations, that I think APISnoop is currently checking.

Yes we compare endpoints and methods against those defined in the OpenAPI spec. 

 

The specs, like PodSpec, that I think APISnoop currently do not check. As an example, you can think about this as the body of the HTTP request when posting to /api/v1/namespaces/{namespace}/pods. I think potentially APISnoop may be extended to check those as next steps. And checking the specs are actually more meaningful when talking about pod behaviors, because kubelet will do things according to the specs.

We agree. We don't look at the request data currently, but it is something we'd like to implement. I have created a ticket to keep track of this here: https://github.com/cncf/apisnoop/issues/19.

Thanks for your interest.

Regards,

Rohan