Enabling Egress NetworkPolicy

[Casey Davenport]

One of the things I've been thinking about is that we have to solve how egress NetworkPolicy is enabled.

In GA NetworkPolicy, ingress is currently enabled via selection of pods.  So, the behavior today is that when a Pod gets selected by a NetworkPolicy, it gets a default deny on its ingress traffic (but egress is unaffected).

When we add egress, we can't say "Whenever a pod is selected, it gets a default deny on its ingress AND its egress traffic" since that would be a breaking change to the API's behavior. 

There might be a few options here, but wanted to throw it out now so folks could chew on it if they hadn't spotted this already.

Any thoughts on how we might do this?

[Thomas Graf]

We have currently implemented it this way:

If pod is selected and has at least one egress policy attached, egress
is limited to the sum of all allowed CIDRs. I understand that this is
not entirely consistent. While at ingress, an empty selector means
allow-all, at egress, an empty []toCIDR means allow to none. Couldn't
come up with anything better so far.

// EgressRule contains all rule types which can be applied at egress, i.e.
// network traffic that originates inside the endpoint and exits the endpoint
// selected by the endpointSelector.
//
// - All members of this structure are optional. If omitted or empty, the
// member will have no effect on the rule.
type EgressRule struct {
// ToPorts is a list of destination ports identified by port
number and
// protocol which the endpoint subject to the rule is allowed to
// connect to.
//
// Example:
// Any endpoint with the label "role=frontend" is allowed to initiate
// connections to destination port 8080/tcp
//
// +optional
ToPorts []PortRule `json:"toPorts,omitempty"`

// ToCIDR is a list of IP blocks which the endpoint subject
to the rule
// is allowed to initiate connections to in addition to connections
// which are allowed via FromEndpoints. This will match on the
// destination IP address of outgoing connections.
//
// Example:
// Any endpoint with the label "app=database-proxy" is allowed to
// initiate connections to 10.2.3.0/24
//
// +optional
ToCIDR []CIDR `json:"toCIDR,omitempty"`
}

[Dan Winship]

On 07/13/2017 07:47 PM, Casey Davenport wrote:
> One of the things I've been thinking about is that we have to solve how
> egress NetworkPolicy is enabled.
>

> In GA NetworkPolicy, ingress is currently enabled via /selection/ of

> pods. So, the behavior today is that when a Pod gets selected by a
> NetworkPolicy, it gets a default deny on its ingress traffic (but egress
> is unaffected).
>
> When we add egress, we can't say "Whenever a pod is selected, it gets a
> default deny on its ingress AND its egress traffic" since that would be
> a breaking change to the API's behavior.
>
> There might be a few options here, but wanted to throw it out now so
> folks could chew on it if they hadn't spotted this already.
>
> Any thoughts on how we might do this?

We could add a "policyTypes" field indicating what kinds of traffic the
policy affects. And if it's unspecified, it would get defaulted to:

- [ "ingress", "egress" ] if the policy has both ingress: and egress:
sections

- [ "egress" ] if the policy has an egress: section but no ingress:
section

- [ "ingress"] if the policy has an ingress: section but no egress:
section, or, for backward compatibility, if it has neither
ingress: nor egress:.

And then update the semantics to say that a pod is isolated for ingress
if any NetworkPolicy *with "ingress" in its policyTypes* selects it, and
it's isolated for egress if any NetworkPolicy with "egress" in its
policyTypes selects it.

The only time you'd actually have to explicitly specify policyTypes is
to write a default-deny egress policy.

-- Dan

[cmluciano]

Thanks for bring this up, it was keeping me awake last night :). 

Is there any sort of validation that we should do on the k8s side to make sure that the egress rules are not colliding with the ingress rules or vice-versa? I think a log entry might be useful, but I'm not sure if we should take that a step further and reject collisions.

[Tim Hockin]

I am on mobile, sorry if it is terse.  Ingress is already a block. In the ingress block is provided, this policy applies to ingress traffic.  If the new egress block is provided, this policy applies to egress traffic.  Any given policy can be one, the other, or both.

I am probably missing something, but I can not easily xref code.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

[Casey Davenport]

Tim - I think the issue comes from here: https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/networking/types.go#L52-L55

Specifically it says "If this field [Ingress] is empty then this NetworkPolicy does not allow any traffic".  I think the behavior you're describing would be "If this field is empty, then this NetworkPolicy does not impact ingress traffic".

This corresponds with the docs, which provide an example for a "default-deny" policy what has no ingress section: https://kubernetes.io/docs/concepts/services-networking/network-policies/#default-policies 

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.

[Tim Hockin]

Shoot.  

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.

[Casey Davenport]

Yeah, as in "in the foot" :)

I've been thinking over this a bit and I think the suggestion made above by Dan earlier in the thread is a good one (and I haven't been able to think of anything better yet).

Can we go to beta with that and see what feedback we get?

Shoot.  

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.

[Tim Hockin]

SGTM. It sounds reasonable to me.

[Tim Hockin]

Looking at some of the docs on this, I can't believe we botched this.
This should serve as a lesson in last-minute API changes. No blame
(or we all share it equally, or maybe it's my damned fault at the end
:)

Sigh. If not for that example, we might have had a loophole. Alas.

[cmluciano]

Please take a look at https://github.com/kubernetes/kubernetes/pull/51351 for relevant discussion on the defaulting behavior. @dwinship and I could use more input on naming for a new NetworkPolicyPeer option.

>>>>>> an email to kubernetes-sig-network+unsub...@googlegroups.com.

>>>>>> To post to this group, send email to
>>>>>> kubernetes-...@googlegroups.com.
>>>>>> Visit this group at
>>>>>> https://groups.google.com/group/kubernetes-sig-network.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "kubernetes-sig-network" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an

>>>> email to kubernetes-sig-network+unsub...@googlegroups.com.

[Tim Hockin]

replied on the PR

>> >>>>>> an email to kubernetes-sig-ne...@googlegroups.com.

>> >>>>>> To post to this group, send email to
>> >>>>>> kubernetes-...@googlegroups.com.
>> >>>>>> Visit this group at
>> >>>>>> https://groups.google.com/group/kubernetes-sig-network.
>> >>>>>> For more options, visit https://groups.google.com/d/optout.
>> >>>>
>> >>>> --
>> >>>> You received this message because you are subscribed to the Google
>> >>>> Groups
>> >>>> "kubernetes-sig-network" group.
>> >>>> To unsubscribe from this group and stop receiving emails from it,
>> >>>> send an

>> >>>> email to kubernetes-sig-ne...@googlegroups.com.

>> >>>> To post to this group, send email to
>> >>>> kubernetes-...@googlegroups.com.
>> >>>> Visit this group at
>> >>>> https://groups.google.com/group/kubernetes-sig-network.
>> >>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-sig-ne...@googlegroups.com.

[Tim Hockin]

NetworkPolicy implementors - we need you to think about this, and we
need it ASAP.

Read this thread, from here forward
https://github.com/kubernetes/kubernetes/pull/51351#issuecomment-325798171

And think about if/how your implementation would behave.

I will start a new thread to try to get attention