Re: Sig-Network: We need your help with the Third-Party Security Audit (kube-proxy)

[Tim Hockin]

We do have a SIG meeting this week, but it is not clear what you're asking for?

On Mon, May 13, 2019 at 8:01 AM Aaron Small <aas...@google.com> wrote:
>
> Hello sig-network,
>
> I'm helping facilitate the Third-Party Security Audit, and we've come to the point where we need the assistance of other SIGs to help us flesh out our Threat Model and Rapid Risk Assessments. We have done a fair amount of independent research, and believe we can fill in the gaps quickly with your help.
>
> Can you make some subject matter experts available this week? We could also attend your weekly meeting if that's easier for you.
>
> Thanks,
>
> Aaron Small | Product Manager, Kubernetes and GKE | aas...@google.com | +1 206-712-1779

[Tim Hockin]

I just realized I may not be at sig-net this week.

Looking at the questions, this will be interesting. Kube-proxy is
fairly privileged and implements a very important abstraction. Would
it be bad to push this back behind KubeCon?

On Mon, May 13, 2019 at 8:34 AM Stefan Edwards
<stefan....@trailofbits.com> wrote:
>
> Absolutely Aaron, I’ve attached the kube-proxy RRA markdown doc to this email.
>
> Tim, in the interest of time, I’ve pre-filled some of the threat modeling information for kube-proxy (since it’s a public component and we can understand it’s architecture) in the form of a Mozilla-style Rapid Risk Assessment (RRA), that way you don’t have to answer simple questions like “what does the service do?” We’ll mostly focus on the data processed and where it’s stored, as well as any other connections, what “controls” are in place (like how does kube-proxy know which requests to trust) and so on.
>
> Does that make sense?
>
> Kind Regards,
> — S.
>
>
> On May 13, 2019, at 11:26 AM, Aaron Small <aas...@google.com> wrote:
>
> Stephan and I would like to join and ask a series of questions about the architecture of kube-proxy they will be security centric.
>
> Stephan, the existing RRAs aren't publicly viewable yet. Can you share some examples in this thread?

[Aaron Small]

[Stefan Edwards]

We also can do this asynchronously as well, via email or slack; I’ve run these assessments by post even (please don’t make me run this assessment by post hahahaha). The RRA should be focused to max one hour, but we can run it whenever or however, would that help? It needn’t be part of the SIG standup per se (though I will certainly share the result with the larger SIG in case anyone else has input they’d like to share).

Kind Regards,
— S.

[Aaron Small]

Stephan and I would like to join and ask a series of questions about the architecture of kube-proxy they will be security centric. 

Stephan, the existing RRAs aren't publicly viewable yet. Can you share some examples in this thread?

From: Tim Hockin <tho...@google.com>
Date: Mon, May 13, 2019, 8:21 AM
To: Aaron Small
Cc: kubernetes-sig-network, <stefan....@trailofbits.com>

[Stefan Edwards]

[Mike Spreitzer]

I am not the greatest expert on kube-proxy, but looking at that RRA provokes some reactions.

I think the "How does the service work" section is trying to describe the first two modes implemented, the userspace proxy and the iptables proxy.  There is a third alternative now, based on IPVS.

The RRA talks about controls on kube-proxy, but does not consider kube-proxy in its role as a control on other things.  That is, as part of what establishes network communications, kube-proxy itself acts as a control on communicating parties.  For example, kube-proxy gets its inputs (mainly Endpoints objects) asynchronously from kubernetes apiservers, which in turn are simply storage and communication depots and it is an asynchronous controller that updates the Endpoints objects, in turn asynchronously based on still other data.  Is there, for example, a concern about lag between changes in the original data --- which is what Pods have what labels and what IP addresses --- and their downstream effects in kube-proxy?

Also, there is a lot more in networking that is potentially relevant to a security audit than kube-proxy.  NetworkPolicy comes to mind, as well as the very basics of CNI and IPAM.  What's the status on that?

Regards,
Mike

"'Stefan Edwards' via kubernetes-sig-network" <kubernetes-...@googlegroups.com> wrote on 05/13/2019 08:42:42 PM:

> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-ne...@googlegroups.com.
> To post to this group, send email to kubernetes-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> To view this discussion on the web visit https://groups.google.com/
> d/msgid/kubernetes-sig-network/EF2FAECA-6048-41BB-
> A033-015283FDC718%40trailofbits.com.
> For more options, visit https://groups.google.com/d/optout.

[Stefan Edwards]

Good morrow Mike,

That’s correct; I didn’t want to include IPVS because it was marked as experimental, and we are tied to the 1.13.4 branch of k8s for this assessment. It seemed unfair to assess kube-proxy in light of an experimental mode for the version we are working on. I certainly can include it here if you feel it’s accurately represented in that version of Kubernetes, that’s no problem. 

Additionally, controls within or on kube-proxy itself are all in scope; we certainly wouldn’t want to limit what kube-proxy has implemented, but our focus is on kube-proxy itself, rather than other portions of the system. So for your example, there’s absolutely a concern about a TOCTOU between an update to a kube-apiserver and the application to kube-proxy. 

We have undertaken a technical assessment of Kubernetes from the perspectives of code & a running dynamic system, and the edges surrounding CNI were in scope, whilst actual CNI implementations themselves were out of scope. So here, we’re most interested in what an attacker with the various positions listed in the RRA could do with access to kube-proxy (for example, hopefully an external, unauthenticated user never has direct access to kube-proxy itself, but what would happen if they did? What could a local malicious user do to manipulate kube-proxy that they may not be able to do with simple host access? &c.) 

Does that distinction make sense? Also, I’m happy to clarify anything within the RRAs, and this sort of discussion is extremely helpful.

Kind Regards,

 — S.

P.S. I’ve added Jay, a member of the Security Working Group, to this email; he’s working with Aaron & I on the threat modeling processes. 

[Tim Hockin]

IPVS is GA now.

In looking at the calendar for tomorrow, I don't think I can make sig-network, so I'm going to ask that we push this until after KubeCon (next week) and we can maybe convene a small group of sig-net volunteers to try to answer the questions.

[Stefan Edwards]

I’m roughly ok with that; Aaron, are you ok holding off threat modeling reporting one week so we can convene with the sig-net team?

[Aaron Small]

I'm fine with that. 

[Stefan Edwards]

Hey Tim and Co,

Do we have some time this week to discuss kube-proxy wrt the RRA?

Thanks!

 — S.

[Tim Hockin]

Shiel is trying to find a time.  This week is the 5th birthday, so there are a lot of promotional things happening.

[Stefan Edwards]

Thank you, that’s much appreciated! I have a full day (like 10-11 hours) of meetings scheduled, but I should be good for most of the rest of the week.

Kind Regards,

 — S.

[Stefan Edwards]

Good Evening sig-network,

We in the Security Working Group are *finally* winding down the threat modeling, and have updated the Rapid Risk Assessment (RRA) documents and the data flow diagrams. In order to allow everyone to participate and view, I have copied the RRA responses to individual gists; these include:

- The RRA template everyone saw, with updates based on the meetings we had.

- The Original data flow diagram

- The updated (rough draft) data flow diagram

Please feel free to comment on the gists as you see fit. Also, many thanks again for everyone’s time, it was greatly appreciated. The feedback will be incorporated into the threat modeling document, and published publicly at the end of the engagement.

Many thanks again, and kind regards,

 — S.

Kube-proxy: https://gist.github.com/lojikil/836900ca6dd226beb4f79eac2bc7ccbf