egress network policy requirements

[cmluciano]

I wanted to resurrect the requirements discussion around egress policy requirements. We have this listed for beta support in 1.8 and I want to finalize the use-cases before I submit an updated proposal. 

The latest document summarizes many use-cases for egress filtering that seem to boil down to the following:

1. I want my pod to not be able to talk to something in the outside world". 
1. A user submitting a CIDR that will be allowed/blocked explicitly
1. There was some discussion about implications of using the CIDR in relation to the IPv6 roadmap as well
2. In prior proposals there was also the possibility of locking down certain ports between pod-pod communication. 
   

Do we have any general agreement on which things egress rules should handle? 

-cmluciano

[Dan Winship]

On 07/13/2017 02:33 PM, cmluciano wrote:
> I wanted to resurrect the requirements discussion around egress policy
> requirements. We have this listed for beta support in 1.8 and I want to
> finalize the use-cases before I submit an updated proposal.
>
> Thelatest document

> <https://docs.google.com/document/d/1-Cq_Y-yuyNklvdnaZF9Qngl3xe0NnArT7Xt_Wno9beg/edit>

> summarizes many use-cases for egress filtering that seem to boil down to
> the following:
>

> 1. I want my pod to not be able to talk to something in the outside
> world".
> 1. A user submitting a CIDR that will be allowed/blocked explicitly
> 1. There was some discussion about implications of using the

> CIDR in relation to the IPv6 roadmap as well

> 2. In prior proposals there was also the possibility of locking down

> certain ports between pod-pod communication.
>
> Do we have any general agreement on which things egress rules should
> handle?

FWIW, OpenShift has either implementations of, or customer requests for,
every use case in that document.

For purposes of this discussion, those use cases can probably be divided
into three groups:

- The "Self-censorship" and "Mandatory self-censorship" use cases
fit very cleanly into the NetworkPolicy model; the project admin
creates NetworkPolicies indicating what their pods are allowed
to access, and the network plugin implements those policies.

This is also more-or-less what is proposed by Dan Williams's
recently-closed-due-to-inactivity PR
(https://github.com/kubernetes/kubernetes/pull/26720) (although
that includes some complications with CIDR-based rules that
aren't needed for these use cases).

(The doc doesn't explicitly mention "locking down certain ports
between pod-pod communication", but the proposal in #26720 does
implement that; if we're implementing the feature as an extension
of NetworkPolicy, it's pretty obvious that being able to specify
specific ports would be part of it.)

- The two "firewall" use cases imply a permissions system with
separate "cluster admins" and "project admins" which didn't match
Kubernetes's permissions model in the past (although I think parts
of OpenShift's auth/permissions system have now started being
absorbed into Kubernetes proper?). Because of the "cluster admin
overrides project admin"-ness, it also doesn't fit as cleanly into
NetworkPolicy, as NetworkPolicy is currently defined. The PR above
tried to figure out some way to handle this, and I think there was
general agreement that it failed.

Also, FWIW, while we have implemented this feature in OpenShift, it
has turned out to be less popular than expected, because even though
people want this functionality, they don't want it *in OpenShift*;
they want to use their existing firewall, but have it be able to
accept/reject traffic based on what kubernetes Namespace the
traffic comes from. So that leads to...

- The "recognizable source IP" use case. This is vastly different in
terms of kubernetes functionality than the other two, although it
ends up being used primarily for restricting outgoing traffic, so
it's similar in intent. We originally had implemented this as a
sort of one-off thing, but now there are users who basically want to
completely throw away the normal pod-to-external traffic flow
(via NAT to the node IP) and have all pod-to-external traffic end up
with an IP that reflects its source namespace, so they can then
set up external firewall rules. We're still figuring out the best
approach here.

Anyway, for Kubernetes 1.8, I think it would make sense for us to tackle
the "self-censorship" cases, by adding stuff to NetworkPolicy along the
lines of #26720, but with simpler CIDR handling (and with updates to
reflect the v1 changes to NetworkPolicy). We may want to discuss the
other use cases but I'm not sure we want to try to tackle them for 1.8.

And also, if we're implementing CIDR-based egress rules, we should at
least talk about CIDR-based ingress rules as well (although that gets
more complicated because you need to coordinate with whatever sort of
load balancer/ingress mechanism is originally accepting the packet).

-- Dan

[Casey Davenport]

Yeah, I agree. I think we should be trying to solve the "70% use-case" that covers the majority of users, and I see that as the "self-censorship" case discussed in the doc.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

[cmluciano]

Thank you both! Before I start submitting PRs are we in agreement for egress being beta in 1.8? If so, egress policies would be submitted through annotations instead of fields like ingress policies. I'm not sure if we think this would be a severe headache for users or not.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.

[Casey Davenport]

Yeah, I think so.

We probably want this to be beta for v1.8, GA for v1.9, which IIUC means it needs to be an annotation for v1.8 (Tim can correct me if that's wrong!)

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.

To post to this group, send email to kubernetes-...@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.

[Tim Hockin]

As far as I know, we don't WANT to use annotations, but the machinery
to do otherwise is not in place yet.

Thanks Dan Winship for the nice writeup. I think I agree with all you
said. Specifically, recognizable source IP is highly desired by our
customers too, but we have no great way to offer it yet, especially at
L3. At L7 you can run HTTP proxies on N nodes, use the masquerade
technique (or other egress-enablement) and allow those N at the far
side. We may want to look at a pre-baked, kube native implementation
of that pattern for people to use.

In the mean time, the question is about self-censorship for egress.
On one hand, it feels different from ingress. Where ingress policy is
self-defense, this is self-censorship. It feels common enough to
ingress policy that I don't have a problem with it, but I wonder how
quickly we start to get feature requests for DNS names.

Now to refresh the spec.

Do we speak in terms of Services or Pods? Honestly, I still feel that
ingress should have been Service-oriented. I am more convinced than
ever that this was wrong.

Do we want to allow Namespaces, as we do with Ingress. I agree
symmetry is strong.

Do we want to allow ingress fromCIDR at the same time?

Who is driving?

[cmluciano]

I plan on driving the implementation with review from Dan Winship and Casey Davenport.

• Service vs Pods
  
• I agree that services is important. I know this Istio project would really benefit from this as well.
• Should we add ServiceSelector to the NetworkPolicyPeer as well? 
• Namespaces
• My vote is yes.
• Ex. If an org has certain namespaces scoped for dev, staging, etc. they may want to ensure that the cannot communicate with production.
• fromCIDR ingress
• In my WIP, I added it to the NetworkPolicyPeer so that it could be leveraged by ingress.

>>>> an email to kubernetes-sig-network+unsub...@googlegroups.com.

>>>>
>>>> To post to this group, send email to kubernetes-...@googlegroups.com.
>>>>
>>>>
>>>> Visit this group at
>>>> https://groups.google.com/group/kubernetes-sig-network.
>>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "kubernetes-sig-network" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to kubernetes-sig-network+unsub...@googlegroups.com.

>> To post to this group, send email to
>> kubernetes-...@googlegroups.com.
>> Visit this group at
>> https://groups.google.com/group/kubernetes-sig-network.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-sig-network+unsub...@googlegroups.com.

[Tim Hockin]

On Mon, Jul 17, 2017 at 9:18 AM, cmluciano <cmlu...@us.ibm.com> wrote:
> I plan on driving the implementation with review from Dan Winship and Casey
> Davenport.
>
> Service vs Pods
>
> I agree that services is important. I know this Istio project would really
> benefit from this as well.
> Should we add ServiceSelector to the NetworkPolicyPeer as well?

Here's why I think we made a mistake. I believe we will see
implementations (as I argued before) that deliver service traffic as a
VIP. I think we will find it more restrictive in implementation to
try to derive whether a VIP is allowed given a PodSelector than if we
had just used Services.

It means that the VIP implementations will not work with the existing
NP implementations.

If I we add ServiceName or ServiceSelector, we still have PodSelector.
We can't get rid of it. That means that some NPs won't work well or
at all on some NP implementations - unportable is bad.

The VIP implementations are not far enough along for me to know for
SURE that this is the case, but had we used Services I am pretty sure
it would be a non issue.

>> >>>> an email to kubernetes-sig-ne...@googlegroups.com.

>> >>>>
>> >>>> To post to this group, send email to kubernetes-...@googlegroups.com.
>> >>>>
>> >>>>
>> >>>> Visit this group at
>> >>>> https://groups.google.com/group/kubernetes-sig-network.
>> >>>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "kubernetes-sig-network" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an

>> >> email to kubernetes-sig-ne...@googlegroups.com.

>> >> To post to this group, send email to
>> >> kubernetes-...@googlegroups.com.
>> >> Visit this group at
>> >> https://groups.google.com/group/kubernetes-sig-network.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "kubernetes-sig-network" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to kubernetes-sig-ne...@googlegroups.com.

>> > To post to this group, send email to
>> > kubernetes-...@googlegroups.com.
>> > Visit this group at
>> > https://groups.google.com/group/kubernetes-sig-network.
>> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-sig-ne...@googlegroups.com.

[Casey Davenport]

> If I we add ServiceName or ServiceSelector, we still have PodSelector.

We can't get rid of it. 

Which is part of why I think we should stick to PodSelector and NamespaceSelector for egress in v1.8 at least.  Let's take bite-size pieces at a time (especially given the desire to focus on stability rather than new features).

It feels like adding ServiceSelector would likely be significant enough that it would warrant doing on its own (separate from egress), and it's not a requirement for solving the use-cases we're trying to tackle in v1.8.

[cmluciano]

Do we also want to add CIDRselector or just mirror current Ingress function to egress?

[cmluciano]

FYI there seems to be a precedent for beta features being fields https://github.com/kubernetes/kubernetes/pull/49061.

[Tim Hockin]

We opted to not do that again until we have better machinery.  Details escape me at the moment, but I can dig them up.

[cmluciano]

ack. Should egress start as v1beta2?

[Tim Hockin]

I don't think we need to review the whole object, but if we did it would be v2beta1.  That is a big hammer.  Our annotations maybe complex enough to warrant it, but I don't know ow if anyone else has gone there yet.

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.