status of NetworkPolicy egress/ipBlock implementation?

[Dan Winship]

We are trying to figure out where we are on the process of moving
NetworkPolicy egress (https://github.com/kubernetes/features/issues/366)
and ipBlock (https://github.com/kubernetes/features/issues/367) from
beta to GA.

Could maintainers of plugins respond and give some information about the
status of these features in your plugins? Eg:

- Have you implemented either or both features yet? If so, have they
gone out in a "stable" release of your plugin? If not, do you have a
timeline for implementing them?

- Are there any gaps in the functionality? Do you handle ipBlock
ingress for non-cluster CIDRs (eg, via Ingress, LoadBalancer, or
NodePort)? Do you handle ipBlock egress for non-cluster CIDRs (eg,
via selector-less Services)

- Do you have any feedback (positive or negative) from users? If not,
do you at least know for sure that anyone is even using the feature?
(If the NetworkPolicy-based implementation is an extension/variation
of some "native" form of policy in your plugin, then information
about how/whether people are using the "native" form might also be
interesting, and especially, if there are people using the "native"
form who are not happy with the NetworkPolicy form.)

- Do you have any unit tests? If so, why haven't you submitted them
upstream to add to test/e2e/network/network_policy.go? :-)

[Dan Winship]

On 01/15/2018 12:34 PM, Dan Winship wrote:
> We are trying to figure out where we are on the process of moving
> NetworkPolicy egress (https://github.com/kubernetes/features/issues/366)
> and ipBlock (https://github.com/kubernetes/features/issues/367) from
> beta to GA.
>
> Could maintainers of plugins respond and give some information about the
> status of these features in your plugins? Eg:
>
> - Have you implemented either or both features yet? If so, have they
> gone out in a "stable" release of your plugin? If not, do you have a
> timeline for implementing them?

Answering for OpenShift: we haven't yet implemented either feature.

Egress support (at least with PodSelectors and NamespaceSelectors) is
currently "proposed" for OpenShift 3.10 (probably early summer).
"Proposed" is the lowest level of planning commitment / most likely to
get punted to the next release, but it doesn't seem like this is going
to be a lot of work so it probably isn't going to get bumped.

ipBlock support is currently not scheduled for any release. The planning
card has a comment about needing to understand the implications better.
(eg, if users are going to expect it to apply to ingress/loadbalancer
traffic, then that's a lot more work).

[Tim Hockin]

As a SIG we need to figure out if it is OK to GA an API with
implementations lagging, and if it is OK for some implementations to
have partial support for GA APIs. E.g. is OpenShift "conformant" to
NetworkPolicy (inasmuch as we don't have a compliance effort for it -
maybe we should?)

> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
> To post to this group, send email to kubernetes-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.

[Salvatore Orlando]

I am not involved with any open source integration at the moment so I'm not able to provide the requested feedback.

AFAICT work is still in progress on this feature for the OVN integration, but the folks that are not active there can provide a more detailed answer.

To Tim's point, I was wondering if there has there been any feedback from operators and users already?

I find the fact that some implementations might lack support for GA APIs - without any way of providing feedback to the user -  rather bad.

Perhaps a decent idea would be to keep these in beta for one more release, maybe documenting the status better in the API reference and in [1], and set a high priority goal for making them GA in the release after?

On this note, resuming the discussion on the NetworkPolicyStatus object might help imho.

Further, I have been struggling finding a notice about the beta status for the IpBlock object in the API reference. I have been looking in [2], and [3]. Is there any other place I should have looked at?

Regards,

Salvatore

[1] https://kubernetes.io/docs/concepts/services-networking/network-policies/#the-networkpolicy-resource

[2] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#ipblock-v1-networking

[3] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#networkpolicypeer-v1-networking

> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
> To post to this group, send email to kubernetes-sig-network@googlegroups.com.

> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.

[Christopher M Luciano]

I mistakenly started a new thread about the feature status. I also have not seen any notes from API consumers on changes that should be made prior to a GA feature designation.

I do agree that it's important to note what expectations there should be for implementors.  I am fine delaying for another release if we do not have many implementations, but I'm not sure what a reasonable period of time should be. Should we delay until we have at least 2 or 3 implementations?

@Salvatore-

I'm unsure where the field or type level designation is supposed to show up. I'm with you that it is confusing to determine what feature level these are at if it does not show up in the generated documentation. 

I reviewed the API changes documentation and added some notations in Egress code comments . I will reach out to API-Machinery to see if I did this properly.

> To post to this group, send email to kubernetes-...@googlegroups.com.

> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.

To post to this group, send email to kubernetes-...@googlegroups.com.

[Casey Davenport]

Copying my response from Christopher's thread:

We implemented both ipBlock and egress in Calico as of Calico v2.6 a few months ago. I've spoken with a handful of users that are using those features without much notable feedback either positive or negative. Calico has similar native capabilities so for many Calico users this might just be business as usual. 

I don't think that we had plans to make changes other than in response to feedback from users or implementations. I don't think we've received substantial feedback from either so far.

Given that it doesn't seem like there are any other implementations yet, and not much in the way of user feedback, I think keeping this in beta for another release sounds sensible enough.

> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.

> To post to this group, send email to kubernetes-...@googlegroups.com.

> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.

To post to this group, send email to kubernetes-...@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.

[Thomas Graf]

On 15 January 2018 at 09:34, Dan Winship <dwin...@redhat.com> wrote:
> We are trying to figure out where we are on the process of moving
> NetworkPolicy egress (https://github.com/kubernetes/features/issues/366)
> and ipBlock (https://github.com/kubernetes/features/issues/367) from
> beta to GA.
>
> Could maintainers of plugins respond and give some information about the
> status of these features in your plugins? Eg:
>
> - Have you implemented either or both features yet? If so, have they
> gone out in a "stable" release of your plugin? If not, do you have a
> timeline for implementing them?

We have partially implemented egress and ipBlock in Cilium. More details below.

>
> - Are there any gaps in the functionality?


> Do you handle ipBlock
> ingress for non-cluster CIDRs (eg, via Ingress, LoadBalancer, or
> NodePort)?

How can this work with arbitrary Ingress?

> Do you handle ipBlock egress for non-cluster CIDRs (eg,
> via selector-less Services)

Users have immediately asked us to be able to specify the name of a
headless service rather than duplicating non-cluster CIDRs which are
already stored as k8s endpoints. We have implemented this via a
"toService" field [0]

[0] http://docs.cilium.io/en/latest/policy/language/#services-based

Personally I find this to be a lot more user friendly as it isolates
service name to IP mapping into a single resource.

> - Do you have any feedback (positive or negative) from users? If not,
> do you at least know for sure that anyone is even using the feature?

The feature is definitely used extensively to restrict access to a
limited set of external services. The preferred way right now seems to
be via a simplified service construct instead of hardcoding IPs.

We have no indication that the egress feature is used to limit access
to other external pods. Ingress rules remain the sole instrument to
achieve in cluster isolation based on feedback.

Negative feedback includes:
- Confusion around requiring to whitelist all in-cluster pods just to
solve the use of of restricting *external* access of pod X to foo.com.
The desired behaviour here is that the action of restricting external
access does not require to duplicate all ingress rules at egress. We
ended up disabling label based egress policy for now because of this
confusion.
- Several question-marks around label based egress policy and
compatibility with identity based policy enforcement such as mTLS and
SPIFFE. While the new identity based systems continue the trend of
decoupling networking from policy, the label based egress is a step
backwards and ties policy to in-cluster addressing by requiring all
nodes to be aware of all pod IPs to strictly enforce the
specification.
- Questions around how the mutual egress and ingress policies should
be written when crossing namespaces.

> (If the NetworkPolicy-based implementation is an extension/variation
> of some "native" form of policy in your plugin, then information
> about how/whether people are using the "native" form might also be
> interesting, and especially, if there are people using the "native"
> form who are not happy with the NetworkPolicy form.)

Yes, because of a lack in being able to specify service names.

[Tim Hockin]

On Tue, Jan 16, 2018 at 9:10 PM, Thomas Graf <tg...@suug.ch> wrote:
> On 15 January 2018 at 09:34, Dan Winship <dwin...@redhat.com> wrote:
>> We are trying to figure out where we are on the process of moving
>> NetworkPolicy egress (https://github.com/kubernetes/features/issues/366)
>> and ipBlock (https://github.com/kubernetes/features/issues/367) from
>> beta to GA.
>>
>> Could maintainers of plugins respond and give some information about the
>> status of these features in your plugins? Eg:
>>
>> - Have you implemented either or both features yet? If so, have they
>> gone out in a "stable" release of your plugin? If not, do you have a
>> timeline for implementing them?
>
> We have partially implemented egress and ipBlock in Cilium. More details below.
>
>>
>> - Are there any gaps in the functionality?
>
>
>> Do you handle ipBlock
>> ingress for non-cluster CIDRs (eg, via Ingress, LoadBalancer, or
>> NodePort)?
>
> How can this work with arbitrary Ingress?
>
>> Do you handle ipBlock egress for non-cluster CIDRs (eg,
>> via selector-less Services)
>
> Users have immediately asked us to be able to specify the name of a
> headless service rather than duplicating non-cluster CIDRs which are
> already stored as k8s endpoints. We have implemented this via a
> "toService" field [0]

IMO podSelector should have been serviceName all over. We can (I
think) still add those. I would need to think hard about compat.

> [0] http://docs.cilium.io/en/latest/policy/language/#services-based
>
> Personally I find this to be a lot more user friendly as it isolates
> service name to IP mapping into a single resource.
>
>> - Do you have any feedback (positive or negative) from users? If not,
>> do you at least know for sure that anyone is even using the feature?
>
> The feature is definitely used extensively to restrict access to a
> limited set of external services. The preferred way right now seems to
> be via a simplified service construct instead of hardcoding IPs.
>
> We have no indication that the egress feature is used to limit access
> to other external pods. Ingress rules remain the sole instrument to
> achieve in cluster isolation based on feedback.
>
> Negative feedback includes:
> - Confusion around requiring to whitelist all in-cluster pods just to
> solve the use of of restricting *external* access of pod X to foo.com.
> The desired behaviour here is that the action of restricting external
> access does not require to duplicate all ingress rules at egress. We
> ended up disabling label based egress policy for now because of this
> confusion.

Not sure I understand. If you want a global deny-egress rule,
NetworkPolicy is a bad fit..

> - Several question-marks around label based egress policy and
> compatibility with identity based policy enforcement such as mTLS and
> SPIFFE. While the new identity based systems continue the trend of
> decoupling networking from policy, the label based egress is a step
> backwards and ties policy to in-cluster addressing by requiring all
> nodes to be aware of all pod IPs to strictly enforce the
> specification.

good questions

> - Questions around how the mutual egress and ingress policies should
> be written when crossing namespaces.
>
>> (If the NetworkPolicy-based implementation is an extension/variation
>> of some "native" form of policy in your plugin, then information
>> about how/whether people are using the "native" form might also be
>> interesting, and especially, if there are people using the "native"
>> form who are not happy with the NetworkPolicy form.)
>
> Yes, because of a lack in being able to specify service names.
>

[Thomas Graf]

Are you worried about whether pod to pod level policy should whitelist
the path via service?
Should service policy allow direct pod to pod if not redirected via service IP?

>
>> [0] http://docs.cilium.io/en/latest/policy/language/#services-based
>>
>> Personally I find this to be a lot more user friendly as it isolates
>> service name to IP mapping into a single resource.
>>
>>> - Do you have any feedback (positive or negative) from users? If not,
>>> do you at least know for sure that anyone is even using the feature?
>>
>> The feature is definitely used extensively to restrict access to a
>> limited set of external services. The preferred way right now seems to
>> be via a simplified service construct instead of hardcoding IPs.
>>
>> We have no indication that the egress feature is used to limit access
>> to other external pods. Ingress rules remain the sole instrument to
>> achieve in cluster isolation based on feedback.
>>
>> Negative feedback includes:
>> - Confusion around requiring to whitelist all in-cluster pods just to
>> solve the use of of restricting *external* access of pod X to foo.com.
>> The desired behaviour here is that the action of restricting external
>> access does not require to duplicate all ingress rules at egress. We
>> ended up disabling label based egress policy for now because of this
>> confusion.
>
> Not sure I understand. If you want a global deny-egress rule,
> NetworkPolicy is a bad fit..

I expressed this poorly. Several users returned this identical feedback:

Steps taken:
1. Put all pods in ingress defaultDeny. Isolate pod to pod
interactions. Define label based ingress rules.
2. Certain pods require outside world access, others need access to
certain external services, e.g. github.com. Define egress defaultDeny
policy + egress ipBlock for certain pods. Puts all pods in egress
defaultDeny. All pod to pod communication is blocked. Confusion.

User brings up the following questions:
- Am I expected to duplicate all rules at ingress and egress?
- I don't want to only enforce at egress because I can't trust
senders. How can I avoid duplication at ingress and egress?
- Why do I need to whitelist egress if I have whitelisted ingress if I
only want to restrict external egress access on top of existing
ingress rules?

[Dan Winship]

On 01/17/2018 12:10 AM, Thomas Graf wrote:
>> Do you handle ipBlock
>> ingress for non-cluster CIDRs (eg, via Ingress, LoadBalancer, or
>> NodePort)?
>
> How can this work with arbitrary Ingress?

Indeed, but I believe that's the expected semantic. Or at least, if
that's *not* the expected semantic, then ipBlock ingress seems
completely pointless, which is also a possibility. IIRC there was never
any proposed use case for it (other than the impossible-to-implement
one); it was just that people didn't like the asymmetry of having
ipBlock only be available for egress.

> Negative feedback includes:
> - Confusion around requiring to whitelist all in-cluster pods just to
> solve the use of of restricting *external* access of pod X to foo.com.
> The desired behaviour here is that the action of restricting external
> access does not require to duplicate all ingress rules at egress. We
> ended up disabling label based egress policy for now because of this
> confusion.

Hm... So FWIW, if you want to disable all external access but not affect
pod-to-pod access, then you can do:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: deny-egress-external-only
spec:
podSelector:
policyTypes:
- Egress
egress:
- to:
- namespaceSelector: {}

ie, "allow egress to all pods in all namespaces, but nowhere else". Not
really obvious, but it should work. (Unless I got it wrong, which is
totally possible. NetworkPolicies are really hard to write correctly.)
Maybe we should add this as an example to the docs.

(Random thought for possible "NetworkPolicy v2" planning: if we had some
sort of Namespace-wide switch like the old "DefaultDeny" annotation,
then we could implement this as a separate base state ({"egress":
{"isolation": "DenyExternal"}}) rather than forcing the user to figure
out the right set of rules for implementing it as "deny all but then
allow certain things". Though in either case, things get complicated if
the admin decides that actually they want one or two in-cluster deny
rules too.)

> - Several question-marks around label based egress policy and
> compatibility with identity based policy enforcement such as mTLS and
> SPIFFE.

Yeah, I hadn't heard about SPIFFE before KubeCon but I want to look into
that more. (Although more for OpenShift's "egress IP" use cases than for
NetworkPolicy. It seems more targeted at admins who want to implement
their policy outside of kubernetes, and just need kubernetes to be
playing along in a way that their external firewall can understand.)

-- Dan

[Ed Warnicke]

<snip> 

Users have immediately asked us to be able to specify the name of a
headless service rather than duplicating non-cluster CIDRs which are
already stored as k8s endpoints. We have implemented this via a
"toService" field [0]

[0] http://docs.cilium.io/en/latest/policy/language/#services-based

Personally I find this to be a lot more user friendly as it isolates
service name to IP mapping into a single resource.

Yeah, I was kind of surprised that NetworkPolicy chose to incorportae CIDRs directly into the NetworkPolicy rather than having a Resource encapsulating the CDIRs with labels to which selectors could be applied.

I sort of expected it to end up sort of like:

kind: IpBlock

metadata:

    name: finance-servers

    labels:

        owner=finance

spec:

    - ipBlock:

        cidr: 172.17.0.0/16

        except:

        - 172.17.1.0/24

kind: IpBlock

metadata:

    name: private-cloud

    labels:

        cloud=private

spec:

    - ipBlock:

        cidr: 10.0.0.0/24

kind: NetworkPolicy

metadata:

  name: test-network-policy

  namespace: default

spec:

  podSelector:

    matchLabels:

      role: db

  policyTypes:

  - Ingress

  - Egress

  ingress:

  - from:

    - ipBlockSelector:

        matchLabels:

            owner=finance

    - namespaceSelector:

        matchLabels:

          project: myproject

    - podSelector:

        matchLabels:

          role: frontend

    ports:

    - protocol: TCP

      port: 6379

  egress:

  - to:

    - ipBlockSelector:

        matchLabels:

            cloud=private

    ports:

    - protocol: TCP

      port: 5978

  

 

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.

[Tim Hockin]

There's a balance to be struck in usability vs composition (or
normalization). Over-normalizing leads to awkward and hard to use
interfaces. Denormalizing leads to redundant information. Service is
another example of this (though that one is poorly organized due to
organic growth).

In this model - is ipblock a namespace-local thing or a global thing?
If it is namespace local, we make it harder to use (2 create ops) in
simple cases and no easier to use in complex cases (unless we do
cross-namespace references). If it is a global, you have to think
carefully about curation and access control.

>> email to kubernetes-sig-ne...@googlegroups.com.

>> To post to this group, send email to

>> kubernetes-...@googlegroups.com.

>> Visit this group at
>> https://groups.google.com/group/kubernetes-sig-network.
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-sig-ne...@googlegroups.com.

> To post to this group, send email to

> kubernetes-...@googlegroups.com.

[Thomas Graf]

On 17 January 2018 at 06:29, Dan Winship <dwin...@redhat.com> wrote:
> On 01/17/2018 12:10 AM, Thomas Graf wrote:
>>> Do you handle ipBlock
>>> ingress for non-cluster CIDRs (eg, via Ingress, LoadBalancer, or
>>> NodePort)?
>>
>> How can this work with arbitrary Ingress?
>
> Indeed, but I believe that's the expected semantic. Or at least, if
> that's *not* the expected semantic, then ipBlock ingress seems
> completely pointless, which is also a possibility. IIRC there was never
> any proposed use case for it (other than the impossible-to-implement
> one); it was just that people didn't like the asymmetry of having
> ipBlock only be available for egress.

The high level use case that comes to mind is the following:
Expose a service to another cluster which can be identified by a source CIDR.

In order for this (and any other ingress CIDR use case) to be useful
at all, the rules must be applied while the original client IP is
still preserved.
kube-proxy and the majority (all?) of ingress controllers do not
provide this behaviour and will rewrite the IP address to either the
node IP or the IP of the ingress controller pod. We have looked into
enforcing the rules pre kube-proxy but this is not safe either as the
policy is specified against the backend pod and not the service.

In summary, ingress cidr is pretty much useless in its current form as
far as I can see. Putting it into GA would be a mistake in my opinion.

Instead, the user may want to define rules like this:
- service/ingress X can be accessed from cidr 10.10.0.0/16
- pod [x,y] can be accessed via service/ingress X

This allows to ensure that:
- pods can only be accessed through a certain service redirection to
prevent direct pod to pod access and to allow a pod to restrict who
can loadbalance to it.
- nodePort/ingress IPs can be protected, ingress can be restricted to
VPN/VPC ranges. This protection can be enforced decoupled from the
loadbalancing decision before the traffic hits the nodePort or the
ingress IP.

We are looking into a CNI+service proposal as mutual awareness is
required to make this work in a user friendly manner.

>> Negative feedback includes:
>> - Confusion around requiring to whitelist all in-cluster pods just to
>> solve the use of of restricting *external* access of pod X to foo.com.
>> The desired behaviour here is that the action of restricting external
>> access does not require to duplicate all ingress rules at egress. We
>> ended up disabling label based egress policy for now because of this
>> confusion.
>
> Hm... So FWIW, if you want to disable all external access but not affect
> pod-to-pod access, then you can do:
>

> [...]

>
> ie, "allow egress to all pods in all namespaces, but nowhere else". Not
> really obvious, but it should work. (Unless I got it wrong, which is
> totally possible. NetworkPolicies are really hard to write correctly.)
> Maybe we should add this as an example to the docs.

Neat. This should definitely be added to the docs as it answers a FAQ
for which we were not able to give a good answer so far.
From a user friendliness perspective, it is horrible though ;-)

We have been experimenting with something like this:

[...]
egress:
- toEntities:
- cluster
- world
- host

Users like this for its obvious meaning and clearly defined context
without requiring to understand any implementation details.

> (Random thought for possible "NetworkPolicy v2" planning: if we had some
> sort of Namespace-wide switch like the old "DefaultDeny" annotation,
> then we could implement this as a separate base state ({"egress":
> {"isolation": "DenyExternal"}}) rather than forcing the user to figure
> out the right set of rules for implementing it as "deny all but then
> allow certain things". Though in either case, things get complicated if
> the admin decides that actually they want one or two in-cluster deny
> rules too.)

Having more fine grained default deny default behaviour would be valuable:

{ ingress | egress }
- DefaultDenyAll
- DefaultDenyExternal
- DefaultDenyCluster

I think this could be worked into the existing policyTypes field. The
name is a bit confusing to me in general. Haven't fully thought this
through though.

>> - Several question-marks around label based egress policy and
>> compatibility with identity based policy enforcement such as mTLS and
>> SPIFFE.
>
> Yeah, I hadn't heard about SPIFFE before KubeCon but I want to look into
> that more. (Although more for OpenShift's "egress IP" use cases than for
> NetworkPolicy. It seems more targeted at admins who want to implement
> their policy outside of kubernetes, and just need kubernetes to be
> playing along in a way that their external firewall can understand.)

Right. I think it would be foolish not to consider these new
architecture that are on a steep adoption curve.

[Ed Warnicke]

I agree with you about the over-normalization, denormalization balance.  One point though about over-denormalization is that it can make updates very difficult and error prone.  I would expect in many cases that an ipBlock represents a semantically meaningful thing (the internet in total minus some blacklist, a different cluster or cloud, a list of various out-of-cluster servers, etc).   I would not expect those things to change rapidly, but by denormalizing them all the way to the policy the way we have, as bare CIDR blocks, it makes it difficult (and error-prone) to update them over time as they do, inevitably change.

As to namespace local vs global, I will confess to rough-sketching out quickly and not having thought that aspect of the suggestion all the way through.  I can see tradeoff either way, but am inclined towards Namespace scoped for the reasons you suggested wrt curation.  The trick, as you point out, is how to keep it easy to use given that.

Ed

>> email to kubernetes-sig-network+unsub...@googlegroups.com.

>> To post to this group, send email to

>> kubernetes-sig-network@googlegroups.com.

>> Visit this group at
>> https://groups.google.com/group/kubernetes-sig-network.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-sig-network+unsub...@googlegroups.com.

> To post to this group, send email to

> kubernetes-sig-network@googlegroups.com.

[Tim Hockin]

A pattern in Kube (which I don't like, but whatever) is to have 2
resources - namespaced and cluster, and reference them explicitly with
a Kind. Now the API gets more cumbersome but at least you can curate
"all corp IPs" distinct from "some denormalized thnig". Or if this is
really an important use case allow references to global ipblocks
objects (what happens if that does change?) and CIDR literals as
different rule types. Doesn't allow easy combining, though.

>> >> email to kubernetes-sig-ne...@googlegroups.com.

>> >> To post to this group, send email to

>> >> kubernetes-...@googlegroups.com.

>> >> Visit this group at
>> >> https://groups.google.com/group/kubernetes-sig-network.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "kubernetes-sig-network" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to kubernetes-sig-ne...@googlegroups.com.

>> > To post to this group, send email to

>> > kubernetes-...@googlegroups.com.

[Tim Hockin]

On Wed, Jan 17, 2018 at 9:23 AM, Thomas Graf <tg...@suug.ch> wrote:
> On 17 January 2018 at 06:29, Dan Winship <dwin...@redhat.com> wrote:
>> On 01/17/2018 12:10 AM, Thomas Graf wrote:
>>>> Do you handle ipBlock
>>>> ingress for non-cluster CIDRs (eg, via Ingress, LoadBalancer, or
>>>> NodePort)?
>>>
>>> How can this work with arbitrary Ingress?
>>
>> Indeed, but I believe that's the expected semantic. Or at least, if
>> that's *not* the expected semantic, then ipBlock ingress seems
>> completely pointless, which is also a possibility. IIRC there was never
>> any proposed use case for it (other than the impossible-to-implement
>> one); it was just that people didn't like the asymmetry of having
>> ipBlock only be available for egress.
>
> The high level use case that comes to mind is the following:
> Expose a service to another cluster which can be identified by a source CIDR.

CIDR is a poor primitive for this use case, but ACK that we don't have
a better one just yet. NetPolicy is a cluster-centric abstraction
(for now?) and istio is a better fit for this because it acknowledges
that the problem of networking is larger than a cluster.

> In order for this (and any other ingress CIDR use case) to be useful
> at all, the rules must be applied while the original client IP is
> still preserved.
> kube-proxy and the majority (all?) of ingress controllers do not

kube-proxy default mode (iptables) and new mode (ipvs) both preserve
source IP. Ingress doesn't because the presumption is that HTTP
doesn't need it (with XFF headers).

> provide this behaviour and will rewrite the IP address to either the
> node IP or the IP of the ingress controller pod. We have looked into
> enforcing the rules pre kube-proxy but this is not safe either as the
> policy is specified against the backend pod and not the service.

This last line is the biggest mistake NetPolicy made. Maybe we should
EOL NetPolicy and rebuild it as ServiceNetPolicy. Only half joking.

> In summary, ingress cidr is pretty much useless in its current form as
> far as I can see. Putting it into GA would be a mistake in my opinion.

I don't think "useless" is right - it's more designed for integration
with "legacy" systems that tend to be less dynamic than kubernetes.

> Instead, the user may want to define rules like this:
> - service/ingress X can be accessed from cidr 10.10.0.0/16
> - pod [x,y] can be accessed via service/ingress X

The problem is that Ingress might be implemented as Pods or might not.
Policy has to apply equally. Agree that having some global rules for
ingress controllers might be a good evolution. "All pods in all
namespaces can be accessed by pods in namespace nginx-ingress" for
example.

[Christopher M Luciano]

Based on this thread, I think we can agree that GA for ipBlock and Egress should not GA within 1.10. There are a few valid enhancements that I'm not sure we can handle within the v1 API (ideas like service and hierarchical labels. 

I wonder if we should instead concentrate on a list of use-cases that are not met by the current NetworkPolicy conventions. The use-case doc can then fuel a design doc for the v2 API. 

>> >> email to kubernetes-sig-network+unsub...@googlegroups.com.

>> >> To post to this group, send email to
>> >> kubernetes-...@googlegroups.com.
>> >> Visit this group at
>> >> https://groups.google.com/group/kubernetes-sig-network.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "kubernetes-sig-network" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to kubernetes-sig-network+unsub...@googlegroups.com.

[Tim Hockin]

I think it would be awesome to compile a purely use-case based list of gripes.

>> >> >> email to kubernetes-sig-ne...@googlegroups.com.

>> >> >> To post to this group, send email to
>> >> >> kubernetes-...@googlegroups.com.
>> >> >> Visit this group at
>> >> >> https://groups.google.com/group/kubernetes-sig-network.
>> >> >> For more options, visit https://groups.google.com/d/optout.
>> >> >
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "kubernetes-sig-network" group.
>> >> > To unsubscribe from this group and stop receiving emails from it,
>> >> > send
>> >> > an

>> >> > email to kubernetes-sig-ne...@googlegroups.com.

>> >> > To post to this group, send email to
>> >> > kubernetes-...@googlegroups.com.
>> >> > Visit this group at
>> >> > https://groups.google.com/group/kubernetes-sig-network.
>> >> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-sig-ne...@googlegroups.com.

[Thomas Graf]

On 17 January 2018 at 10:00, Tim Hockin <tho...@google.com> wrote:
> On Wed, Jan 17, 2018 at 9:23 AM, Thomas Graf <tg...@suug.ch> wrote:
>> The high level use case that comes to mind is the following:
>> Expose a service to another cluster which can be identified by a source CIDR.
>
> CIDR is a poor primitive for this use case, but ACK that we don't have
> a better one just yet. NetPolicy is a cluster-centric abstraction
> (for now?) and istio is a better fit for this because it acknowledges
> that the problem of networking is larger than a cluster.

It is worth having a discussion whether there is any value in trying
to extend or to just keep NetPolicy in its current scope and come up
with a new service based model that works across clusters and is
compatible with Istio and identity/mTLS based systems.

>> In order for this (and any other ingress CIDR use case) to be useful
>> at all, the rules must be applied while the original client IP is
>> still preserved.
>> kube-proxy and the majority (all?) of ingress controllers do not
>
> kube-proxy default mode (iptables) and new mode (ipvs) both preserve
> source IP. Ingress doesn't because the presumption is that HTTP
> doesn't need it (with XFF headers).

This is only guaranteed if the ClusterIP translation happens on either
the node of the source pod or on the node of the destination pod,
right? It will not reliably work for NodePort and ingress at all as
far as I can tell. My main point here is that it cannot be reliably
used to whitelist access from particular external source ranges based
on user feedback.

>> provide this behaviour and will rewrite the IP address to either the
>> node IP or the IP of the ingress controller pod. We have looked into
>> enforcing the rules pre kube-proxy but this is not safe either as the
>> policy is specified against the backend pod and not the service.
>
> This last line is the biggest mistake NetPolicy made. Maybe we should
> EOL NetPolicy and rebuild it as ServiceNetPolicy. Only half joking.

I would fully support this.

>> Instead, the user may want to define rules like this:
>> - service/ingress X can be accessed from cidr 10.10.0.0/16
>> - pod [x,y] can be accessed via service/ingress X
>
> The problem is that Ingress might be implemented as Pods or might not.
> Policy has to apply equally. Agree that having some global rules for
> ingress controllers might be a good evolution. "All pods in all
> namespaces can be accessed by pods in namespace nginx-ingress" for
> example.

Another option is to make enforcement of "ingress X can only be
accessed from 10.10.0.0/16" the responsibility of ingress itself. Most
proxies should already have this capability anyway. I'm not sure about
cloud provider LBs though. The same can be said for service,
kube-proxy or whatever implements services can protect itself.

Maybe we can treat ingress and service equally from a policy perspective.

[Tim Hockin]

On Wed, Jan 17, 2018 at 11:25 AM, Thomas Graf <tg...@suug.ch> wrote:
> On 17 January 2018 at 10:00, Tim Hockin <tho...@google.com> wrote:
>> On Wed, Jan 17, 2018 at 9:23 AM, Thomas Graf <tg...@suug.ch> wrote:
>>> The high level use case that comes to mind is the following:
>>> Expose a service to another cluster which can be identified by a source CIDR.
>>
>> CIDR is a poor primitive for this use case, but ACK that we don't have
>> a better one just yet. NetPolicy is a cluster-centric abstraction
>> (for now?) and istio is a better fit for this because it acknowledges
>> that the problem of networking is larger than a cluster.
>
> It is worth having a discussion whether there is any value in trying
> to extend or to just keep NetPolicy in its current scope and come up
> with a new service based model that works across clusters and is
> compatible with Istio and identity/mTLS based systems.

Honestly, widespread adoption of Istio (and Istio-ish things) may just
obsolete NetworkPolicy).

>>> In order for this (and any other ingress CIDR use case) to be useful
>>> at all, the rules must be applied while the original client IP is
>>> still preserved.
>>> kube-proxy and the majority (all?) of ingress controllers do not
>>
>> kube-proxy default mode (iptables) and new mode (ipvs) both preserve
>> source IP. Ingress doesn't because the presumption is that HTTP
>> doesn't need it (with XFF headers).
>
> This is only guaranteed if the ClusterIP translation happens on either
> the node of the source pod or on the node of the destination pod,
> right? It will not reliably work for NodePort and ingress at all as
> far as I can tell. My main point here is that it cannot be reliably
> used to whitelist access from particular external source ranges based
> on user feedback.

For *external* IPs (sorry, missed that) you have to use
`externalTrafficPolicy: Local` to avoid SNAT, yes.

>>> provide this behaviour and will rewrite the IP address to either the
>>> node IP or the IP of the ingress controller pod. We have looked into
>>> enforcing the rules pre kube-proxy but this is not safe either as the
>>> policy is specified against the backend pod and not the service.
>>
>> This last line is the biggest mistake NetPolicy made. Maybe we should
>> EOL NetPolicy and rebuild it as ServiceNetPolicy. Only half joking.
>
> I would fully support this.

Proposals welcome. There was pushback when I proposed it (admittedly
late in the lifecycle) that not all targets would be in a Service. I
argued that creating a service is low-cost. There was also some
concern about whether this only applies to the service or to the pods
behind it (direct pod-to-pod w/o Service VIP).

Also we should decide if Service is just targets or also used as a
carrier for a source-selector.

We could add Service as options here, but it's not great because
compat is hard. I have seen at least one proposed impl of Services
that would be non-NAT VIPs end-to-end, but NetPol made it infeasible.

>>> Instead, the user may want to define rules like this:
>>> - service/ingress X can be accessed from cidr 10.10.0.0/16
>>> - pod [x,y] can be accessed via service/ingress X
>>
>> The problem is that Ingress might be implemented as Pods or might not.
>> Policy has to apply equally. Agree that having some global rules for
>> ingress controllers might be a good evolution. "All pods in all
>> namespaces can be accessed by pods in namespace nginx-ingress" for
>> example.
>
> Another option is to make enforcement of "ingress X can only be
> accessed from 10.10.0.0/16" the responsibility of ingress itself. Most
> proxies should already have this capability anyway. I'm not sure about
> cloud provider LBs though. The same can be said for service,
> kube-proxy or whatever implements services can protect itself.

Yes, this is the same as Service which has `loadBalancerSourceRanges`

[Guru Shetty]

On 15 January 2018 at 09:34, Dan Winship <dwin...@redhat.com> wrote:

I have a basic question around pod-label based egress network policies.

Q: If a pod-A is only allowed to speak to pod-B (via label), we allow egress connection from pod-A to pod-B when the destination IP address is pod-B's IP address. Are we also supposed to allow connections to pod-B, if the connection to that pod-B is via a kubernetes service (A VIP)?  The question can be rephrased also as, whether egress policies needs to be applied after load-balancing is done or before it.

 

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.

[Dan Winship]

On 01/29/2018 05:39 PM, Guru Shetty wrote:
> I have a basic question around pod-label based egress network policies.

Pod-label-based egress policies don't really introduce any new wrinkles
here from the perspective of the network plugin: at the cluster level,
"all pods are ingress-isolated but A allows connections from B" behaves
identically to "all pods are egress-isolated but B allows connections to A".

> Q: If a pod-A is only allowed to speak to pod-B (via label), we allow
> egress connection from pod-A to pod-B when the destination IP address is
> pod-B's IP address. Are we also supposed to allow connections to pod-B,
> if the connection to that pod-B is via a kubernetes service (A VIP)? 

Yes. NetworkPolicies act on the original source and final destination of
the traffic.

> The question can be rephrased also as, whether egress policies needs to
> be applied after load-balancing is done or before it.

After.

One of the weird edge case scenarios that has been discussed is: what
happens if service X is backed by pods A and B, and pod C can connect to
pod A but not pod B, and pod C connects to service X? The correct answer
seems to be "half the time it connects to pod A, and half the time it
tries to connect to pod B and fails", although this is a stupid
configuration so I don't think anyone would fault a plugin that
implemented "it always connects to pod A". Maybe you'd even get partial
credit for "the connection always fails". Just as long as it never
successfully connects to pod B.

-- Dan

[Christopher M Luciano]

Trying to resurrect this thread. We are nearing the year mark since we marked NP Egress/IPblock support as beta in 1.8. Can we target the bit-flip for GA status in 1.12 or do we still believe features are missing?

More context can be found in the feature issue.

[Tim Hockin]

I don't think I have heard anything that would block moving the existing features to GA.  Anyone?

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.

[Casey Davenport]

+1 from me.

--

Casey Davenport 
Software Engineer 
Tigera 
ca...@tigera.io | @csdvnprt 
Follow us: Blog | Twitter | LinkedIn 

Zero Trust Network Security & Continuous Compliance for Modern Applications

[Dan Winship]

I had already commented on the github issue, but for anyone not reading
along there, I suggested we should declare egress (including
ipBlock-based egress) to be GA, but we should not declare ipBlock-based
ingress GA, and maybe we should even deprecate it. (Because we still
never defined whether it is supposed to apply to cluster ingress traffic
or not, and if we say "yes", then it's not actually possible for a
plugin to implement, and if we say "no", then AFAIK there isn't actually
any use case for it.)

-- Dan

On 08/08/2018 04:59 PM, 'Tim Hockin' via kubernetes-sig-network wrote:
> I don't think I have heard anything that would block moving the existing
> features to GA.  Anyone?
>
> On Wed, Aug 8, 2018 at 12:42 PM Christopher M Luciano
> <cmlu...@cruznet.org <mailto:cmlu...@cruznet.org>> wrote:
>
> Trying to resurrect this thread. We are nearing the year mark since
> we marked NP Egress/IPblock support as beta in 1.8. Can we target
> the bit-flip for GA status in 1.12 or do we still believe features
> are missing?
>
> More context can be found in the feature issue.

> <https://github.com/kubernetes/features/issues/366#issuecomment-409289151>

> <mailto:kubernetes-sig-ne...@googlegroups.com>.

> To post to this group, send email to
> kubernetes-...@googlegroups.com

> <mailto:kubernetes-...@googlegroups.com>.

> Visit this group at
> https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to kubernetes-sig-ne...@googlegroups.com

> <mailto:kubernetes-sig-ne...@googlegroups.com>.

> To post to this group, send email to
> kubernetes-...@googlegroups.com

> <mailto:kubernetes-...@googlegroups.com>.

[Tim Hockin]

It can't apply to Ingress because that could be totally obscured in an
HTTP header behind TLS encryption, right?

I thought the primary use case was to allow traffic from addresses on
a network (e.g. corp) that are not running in the cluster. For
example, I can use an internal L4 LB or other service
discovery/delivery mechanism to expose kube things ...

[Dan Winship]

On 08/16/2018 06:35 PM, Tim Hockin wrote:
> It can't apply to Ingress because that could be totally obscured in an
> HTTP header behind TLS encryption, right?

Right. (Or even if there's no TLS encryption, some plugins would not be
able to parse a source IP out of an HTTP header.)

> I thought the primary use case was to allow traffic from addresses on
> a network (e.g. corp) that are not running in the cluster. For
> example, I can use an internal L4 LB or other service
> discovery/delivery mechanism to expose kube things ...

I've never heard anyone mention that use case, and AFAICT neither Kube's
docs nor GCP's mention it. Does it work on GCP? Does it work on other
clouds? We don't really specify anything at all about how internal load
balancers work...

-- Dan

[Tim Hockin]

On Fri, Aug 17, 2018 at 5:28 AM Dan Winship <dwin...@redhat.com> wrote:
>
> On 08/16/2018 06:35 PM, Tim Hockin wrote:
> > It can't apply to Ingress because that could be totally obscured in an
> > HTTP header behind TLS encryption, right?
>
> Right. (Or even if there's no TLS encryption, some plugins would not be
> able to parse a source IP out of an HTTP header.)
>
> > I thought the primary use case was to allow traffic from addresses on
> > a network (e.g. corp) that are not running in the cluster. For
> > example, I can use an internal L4 LB or other service
> > discovery/delivery mechanism to expose kube things ...
>
> I've never heard anyone mention that use case, and AFAICT neither Kube's
> docs nor GCP's mention it. Does it work on GCP? Does it work on other
> clouds? We don't really specify anything at all about how internal load
> balancers work...

No, we do not spec LBs outside of kube itself. If your LB is a
proxy, you are SOL for the same reason as Ingress, but if your LB is
DSR you should be OK. But LB is just one mechanism. Pods could
register in Consul or elsewhere, etc. I don't want to proscribe too
much about how Services are implemented, keeping mind that kube-proxy
can be replaced.

We added it based on user and vendor feedback, I'd have expected more
noise about it in this thread...

Maybe worth asking in a broader forum like kube-users or something?

[Dan Winship]

On 08/17/2018 12:27 PM, Tim Hockin wrote:
> On Fri, Aug 17, 2018 at 5:28 AM Dan Winship <dwin...@redhat.com> wrote:
>> I've never heard anyone mention that use case, and AFAICT neither Kube's
>> docs nor GCP's mention it. Does it work on GCP? Does it work on other
>> clouds? We don't really specify anything at all about how internal load
>> balancers work...
>
> No, we do not spec LBs outside of kube itself. If your LB is a
> proxy, you are SOL for the same reason as Ingress, but if your LB is
> DSR you should be OK. But LB is just one mechanism. Pods could
> register in Consul or elsewhere, etc. I don't want to proscribe too
> much about how Services are implemented, keeping mind that kube-proxy
> can be replaced.

Right. But if Services/ingress are going to implemented in a variety of
ways, and most of those ways are incompatible with ipBlock ingress
NetworkPolicy, then should we really have ipBlock ingress NetworkPolicy?

> We added it based on user and vendor feedback, I'd have expected more
> noise about it in this thread...

We added ipBlock-based *egress* based on user and vendor feedback, and
we do know that that feature is being used. But really the primary
argument for ipBlock-based *ingress* was that people wanted to have the
same selectors for ingress and egress...

It appears that IP-based filtering of cluster ingress is available via
the nginx Ingress object
(https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#whitelist-source-range)
and certain LoadBalancer implementations
(https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service).
So maybe the reason no one is complaining about it not working in
NetworkPolicy is because there were other working solutions already.

-- Dan

[Tim Hockin]

I'm all for "less is more", though it's a little weird to leave it languishing.

I'm looking for other voices. I am going to reply and change teh
subject to try to raise attention

[Tim Hockin]

Hey all.

Dan raises some great questions about NetworkPolicy's ingress ipBlock
support. If you use this, can you please write a few sentences about
what you're doing with it?

Tim

[Thomas Graf]

It's definitely a source for regular confusion as exact behavior requires a lot of understanding of low level plumbing but that's not the fault of NP but rather just kube-proxy SNAT behavior and NodePort architecture. We found a way to extract some information on source IP before kube-proxy to improve the situation but it doesn't really work if NodePort adds a 2nd hop in standard installations. We mainly suggest to just filter on X-Forwarded-For headers instead for now.

> >>>>>     send an email to kubernetes-sig-network+unsub...@googlegroups.com
> >>>>>     <mailto:kubernetes-sig-network+unsubscribe@googlegroups.com>.

> >>>>>     To post to this group, send email to

> >>>>>     kubernetes-sig-network@googlegroups.com
> >>>>>     <mailto:kubernetes-sig-net...@googlegroups.com>.

> >>>>>     Visit this group at
> >>>>>     https://groups.google.com/group/kubernetes-sig-network.
> >>>>>     For more options, visit https://groups.google.com/d/optout.
> >>>>>
> >>>>> --
> >>>>> You received this message because you are subscribed to the Google
> >>>>> Groups "kubernetes-sig-network" group.
> >>>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>>> an email to kubernetes-sig-network+unsub...@googlegroups.com
> >>>>> <mailto:kubernetes-sig-network+unsubscribe@googlegroups.com>.

> >>>>> To post to this group, send email to

> >>>>> kubernetes-sig-network@googlegroups.com
> >>>>> <mailto:kubernetes-sig-net...@googlegroups.com>.

> >>>>> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> >>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>
> >>
>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.

[ayodele abejide]

Hi,

> If you use this, can you please write a few sentences about 

what you're doing with it?

IPBlock Ingress NP user here, our use cases are as below:

- ELB subnet whitelisting, once an ingress NP is configured all ingress traffic

are dropped, so we need the ipblock rules to whitelist the ELB subnet in AWS.

- Cross node load-balanced traffic, we observed load-balanced traffic is usually

dropped for a very small window when we deploy our application and the only

way to work-around it was whitelisting our nodes' IP addresses with IPBlock.

- In some of our Kubernetes clusters our pods are partially (and sometimes in

  the future will be fully) routed, so we need to make sure only the nodes in

our cluster we want to reach our pods can actually reach them.

Thanks,

[Dan Winship]

On 08/19/2018 10:02 PM, ayodele abejide wrote:
> Hi,
>
>> If you use this, can you please write a few sentences about 
> what you're doing with it?
>
> IPBlock Ingress NP user here, our use cases are as below:
>
> - ELB subnet whitelisting, once an ingress NP is configured all ingress
> traffic
> are dropped, so we need the ipblock rules to whitelist the ELB subnet in
> AWS.

Huh. OK, yes, that would be needed currently (if you don't just
configure the service to allow *all* connections, which is what people
used to do pre-ipBlock I think).

The fact that this is needed feels like a bug though; if someone
configures Ingress/LoadBalancers/etc, then obviously they want packets
from those servers to be able to reach the corresponding pods, so we
should allow that traffic automatically (just like how we allow kubelet
health checks automatically). Though probably the plugin doesn't
currently have all the information it would need to be able to do that
for all ingress types.

> - Cross node load-balanced traffic, we observed load-balanced traffic is
> usually
> dropped for a very small window when we deploy our application and the only
> way to work-around it was whitelisting our nodes' IP addresses with IPBlock.

That definitely seems like a bug.

> - In some of our Kubernetes clusters our pods are partially (and
> sometimes in
>   the future will be fully) routed, so we need to make sure only the
> nodes in
> our cluster we want to reach our pods can actually reach them.

So, just to clarify, it's possible for some hosts outside your cluster
to access some pods directly by their pod IP (or service IP?), without
passing through any sort of ingress/load
balancing/hostport/nodeport/externalIP, and so the NetworkPolicy can
then act on the host's IP?

-- Dan

[Àbéjídé Àyodélé]

> configure the service to allow *all* connections

This defeats the purpose of Ingress NP.

> So, just to clarify, it's possible for some hosts outside your cluster
> to access some pods directly by their pod IP (or service IP?), without
> passing through any sort of ingress/load
> balancing/hostport/nodeport/externalIP, and so the NetworkPolicy can
> then act on the host's IP?

Yes and in fact we are considering making our pods accessible by
our entire network at some point in the future and using some service
discovery mechanism to make them discoverable.

Abejide Ayodele
It always seems impossible until it's done. --Nelson Mandela

[Casey Davenport]

Yeah - I know of users who do lots of direct routing to pod IPs from non-cluster things, without a service proxy / LB / ingress in the way. To them, ingress ipBlock is an important feature, and declaring that ipBlock policy enforcement must be done by the Ingress mechanism is likely a non-starter. There are also things like NLB on AWS which will preserve source IP.

If anything, I think we might want to define the behavior more tightly so that users know how NP interacts with various proxies - e.g. "selects traffic based on the source IP address as seen by the selected destination pod" or something like that.  Even that might be a bit more restrictive than we want - there is always going to be some tension between the multitude of implementations of both NP and services+ingress that exist and trying to make a full-mesh of precisely defined behavior between them all.

> >>>>>     send an email to kubernetes-sig-ne...@googlegroups.com
> >>>>>     <mailto:kubernetes-sig-ne...@googlegroups.com>.

> >>>>>     To post to this group, send email to

> >>>>>     kubernetes-...@googlegroups.com
> >>>>>     <mailto:kubernetes-...@googlegroups.com>.

> >>>>>     Visit this group at
> >>>>>     https://groups.google.com/group/kubernetes-sig-network.
> >>>>>     For more options, visit https://groups.google.com/d/optout.
> >>>>>
> >>>>> --
> >>>>> You received this message because you are subscribed to the Google
> >>>>> Groups "kubernetes-sig-network" group.
> >>>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>>> an email to kubernetes-sig-ne...@googlegroups.com
> >>>>> <mailto:kubernetes-sig-ne...@googlegroups.com>.

> >>>>> To post to this group, send email to

> >>>>> kubernetes-...@googlegroups.com
> >>>>> <mailto:kubernetes-...@googlegroups.com>.

> >>>>> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> >>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>
> >>
>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

--

[Dan Winship]

So maybe document something like:

In general, ipBlock ingress rules select traffic based on the source
IP address *as seen by the selected destination pod*. For traffic
coming from outside the cluster, this is heavily dependent on the
exact details of the network plugin, cloud provider,
LoadBalancer/Ingress implementation, etc. With some ingress options
you may be able to filter based on the actual original source IP. In
other cases, the "source IP" that the NetworkPolicy acts on may be
the IP of a LoadBalancer or of the pod's node, etc.

The "in general" provides an out for implementations that really want to
do something else (eg, filter based on the original source IP even
though the pod sees a different source IP). Plugins that do that can
just document their behavior in their own docs.


I think we also need a similar clarification for egress ipBlocks with
respect to service IPs. Eg, if I have:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow-to-everything-except-1.2.3.4
spec:
podSelector: {}
policyTypes: [ "Egress" ]
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 1.2.3.4/32

And then I create:

kind: Service
apiVersion: v1
metadata:
name: backdoor
spec:
clusterIP: 172.30.99.99
ports:
- port: 80

kind: Endpoints
apiVersion: v1
metadata:
name: backdoor
subsets:
- addresses:
- ip: 1.2.3.4
ports:
- port: 80

and connect to 172.30.99.99, what happens? By analogy with the behavior
of services-to-pods, it would seem like the connection should be
blocked, but that would require the plugin to implement the policy after
the packet had passed through kube-proxy and been NATted.

So we could say that, like with ingress ipBlock, egress ipBlock only
applies to the destination IP as seen by the source pod. Of course that
means it's always possible to bypass egress ipBlock policies by creating
a Service, but *probably* if I'm able to create a Service, I am also
able to create/edit NetworkPolicies, so this isn't *really* a security
hole. Probably...

-- Dan

> <mailto:cmlu...@cruznet.org <mailto:cmlu...@cruznet.org>>>

> <mailto:kubernetes-sig-network%2Bunsu...@googlegroups.com>
> > >>>>>   
>  <mailto:kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-network%2Bunsu...@googlegroups.com>>.

> > >>>>>     To post to this group, send email to
> > >>>>>     kubernetes-...@googlegroups.com
> <mailto:kubernetes-...@googlegroups.com>

> > >>>>>     <mailto:kubernetes-...@googlegroups.com

> <mailto:kubernetes-...@googlegroups.com>>.
> > >>>>>     Visit this group at
> > >>>>>     https://groups.google.com/group/kubernetes-sig-network.
> > >>>>>     For more options, visit
> https://groups.google.com/d/optout.
> > >>>>>
> > >>>>> --
> > >>>>> You received this message because you are subscribed to
> the Google
> > >>>>> Groups "kubernetes-sig-network" group.
> > >>>>> To unsubscribe from this group and stop receiving emails
> from it, send
> > >>>>> an email to
> kubernetes-sig-ne...@googlegroups.com

> <mailto:kubernetes-sig-network%2Bunsu...@googlegroups.com>
> > >>>>>
> <mailto:kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-network%2Bunsu...@googlegroups.com>>.

> > >>>>> To post to this group, send email to
> > >>>>> kubernetes-...@googlegroups.com
> <mailto:kubernetes-...@googlegroups.com>

> > >>>>> <mailto:kubernetes-...@googlegroups.com

> <mailto:kubernetes-...@googlegroups.com>>.
> > >>>>> Visit this group at
> https://groups.google.com/group/kubernetes-sig-network.
> > >>>>> For more options, visit https://groups.google.com/d/optout.
> > >>>>
> > >>
> >
>
> --
> You received this message because you are subscribed to the
> Google Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to
> kubernetes-sig-ne...@googlegroups.com

> <mailto:kubernetes-sig-network%2Bunsu...@googlegroups.com>.

> To post to this group, send email to
> kubernetes-...@googlegroups.com
> <mailto:kubernetes-...@googlegroups.com>.
> Visit this group at
> https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to kubernetes-sig-ne...@googlegroups.com
> <mailto:kubernetes-sig-ne...@googlegroups.com>.
> To post to this group, send email to
> kubernetes-...@googlegroups.com
> <mailto:kubernetes-...@googlegroups.com>.
> Visit this group at
> https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.
>
> --

> *Casey* *Davenport* 
> Software Engineer 
> Tigera 
> ca...@tigera.io <mailto:ca...@tigera.io> | @csdvnprt
> <http://twitter.com/csdvnprt> 
> Follow us: Blog <https://blog.tigera.io/> | Twitter
> <https://twitter.com/tigeraio> | LinkedIn
> <https://www.linkedin.com/company/tigera/> 

>
> Zero Trust Network Security & Continuous Compliance for Modern

> Applications <https://tigera.io/>

[David Anderson]

On Fri, Aug 17, 2018 at 9:27 AM, 'Tim Hockin' via kubernetes-sig-network <kubernetes-...@googlegroups.com> wrote:

On Fri, Aug 17, 2018 at 5:28 AM Dan Winship <dwin...@redhat.com> wrote:
>
> On 08/16/2018 06:35 PM, Tim Hockin wrote:
> > It can't apply to Ingress because that could be totally obscured in an
> > HTTP header behind TLS encryption, right?
>
> Right. (Or even if there's no TLS encryption, some plugins would not be
> able to parse a source IP out of an HTTP header.)
>
> > I thought the primary use case was to allow traffic from addresses on
> > a network (e.g. corp) that are not running in the cluster.  For
> > example, I can use an internal L4 LB or other service
> > discovery/delivery mechanism to expose kube things ...
>
> I've never heard anyone mention that use case, and AFAICT neither Kube's
> docs nor GCP's mention it. Does it work on GCP? Does it work on other
> clouds? We don't really specify anything at all about how internal load
> balancers work...

No, we do not spec LBs outside of kube itself.   If your LB is a
proxy, you are SOL for the same reason as Ingress, but if your LB is
DSR you should be OK.  But LB is just one mechanism.  Pods could
register in Consul or elsewhere, etc.  I don't want to proscribe too
much about how Services are implemented, keeping mind that kube-proxy
can be replaced.

We added it based on user and vendor feedback, I'd have expected more
noise about it in this thread...

Maybe worth asking in a broader forum like kube-users or something?

FWIW, I use ingress ipBlocks in conjunction with MetalLB on my bare-metal clusters. Since MetalLB relies on kube-proxy for dataplane forwarding, in practice I'm just relying on kube-proxy to do the thing. As stated elsewhere in the thread, it's a bit error-prone because of the way networking is specified in k8s (i.e. not very much, when it comes to handling external traffic). But, if you understand the stack and work within the imposed constraints, it's definitely useful.

I would suggest that this discussion highlights a different problem: LoadBalancer Services are under-specified, in a way that is harmful to both k8s users and implementors of LoadBalancers. Users are left to figure out empirically what does and doesn't work on any particular platform, and implementors have to deal with the fallout of those expectations, in the form of support load and annoyed users ("Hey, your LB is broken, it doesn't support NetworkPolicies!" Well, it's because (a) nobody told me it should and (b) no tests failed when I didn't support it!).

It feels like it would be better to walk back the "no rules, anything goes" approach to LB services a little, rather than treat that as a binding constraint on other useful features. I have some further thoughts on this below, although it's drifting off-topic a little, so if you'd rather have those as its own thread/proposal, I'd be happy to.

For the purpose of this thread, I'd like to voice support for a GA of ingress ipBlocks :).

---

Expanding a little on the above thoughts about LBs being underspecified...

In my ideal world, as both a maintainer of one LoadBalancer implementation and a user of several, we would have a crisper definition of what it means to be a conformant LoadBalancer implementation.

By that I don't mean mandating how the LB works on the inside, but what end-user behaviors MUST, MAY, and MUST NOT work, in RFC terminology. In particular, such a spec could define some facets like NetworkPolicy support as optional, but with less choice than the current "do whatever you want" : either you support it and conform to specified behaviors, or you don't and your cluster declines to support it in a single, specified way.

Taking this NetworkPolicy feature as an example, we could document that:

• It's an optional facet, which particular clusters are free to support or not.
• If the facet is supported, there's a supplemental suite of conformance tests which must pass for the cluster to be conformant.
• If the facet is not supported, there's a supplemental suite of conformance tests that verify that the cluster rejects attempts to set NetworkPolicies on LB endpoints, at admission time so that `kubectl apply` fails with a useful error.

I think this would be to the benefit of LB implementors, LB users, and k8s developers in general:

• Benefit to implementors: there's now a clearly defined set of targets to hit: select the things you can/want to support, and you get a crisp set of conformance tests to pass.
  
• Benefit to users: there's now a clear definition of what LoadBalancers do, and you can use the optional feature matrix when shopping for platforms - or at least, unsupported behaviors fail in an unsurprising way.
• Benefit to k8s developers: it's easier to track what features are well/poorly supported out in the world, which ones might need redoing for better adoption, which ones are good candidates to make mandatory in future releases, and so forth.

This would also be generalizable to other k8s objects that platforms are expected to implement in proprietary ways.

- Dave

> >>>     send an email to kubernetes-sig-network+unsub...@googlegroups.com
> >>>     <mailto:kubernetes-sig-network+unsubscribe@googlegroups.com>.

> >>>     To post to this group, send email to

> >>>     kubernetes-sig-network@googlegroups.com
> >>>     <mailto:kubernetes-sig-net...@googlegroups.com>.

> >>>     Visit this group at
> >>>     https://groups.google.com/group/kubernetes-sig-network.
> >>>     For more options, visit https://groups.google.com/d/optout.
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "kubernetes-sig-network" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send

> >>> an email to kubernetes-sig-network+unsub...@googlegroups.com
> >>> <mailto:kubernetes-sig-network+unsubscribe@googlegroups.com>.

> >>> To post to this group, send email to

> >>> kubernetes-sig-network@googlegroups.com
> >>> <mailto:kubernetes-sig-net...@googlegroups.com>.

> >>> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> >>> For more options, visit https://groups.google.com/d/optout.
> >>
>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.

[Dan Winship]

I've filed https://github.com/kubernetes/website/pull/10151 with both of
the clarifications below (that both ingress and egress ipBlocks may not
do what you were hoping), and also added documentation of
namespaceSelector+podSelector, which I think was previously undocumented
(other than at the API level).

-- Dan

[Tim Hockin]

On Tue, Aug 21, 2018 at 9:56 AM Àbéjídé Àyodélé
<abejide...@gmail.com> wrote:
>
> > configure the service to allow *all* connections
>
> This defeats the purpose of Ingress NP.

I would argue there's three levels of concept to look at.

1) What should be able to access a given pod (other pods, other hosts
on network, etc)

2) What should be able to access a given service

3) What should be able to access a given Service via a service LB

4) What should be able to access a given Service via an Ingress.

Item 1 is covered by NetPol. Item 2 has no Kube abstraction. Item 3
is covered (partly) by service.spec.loadBalancerSourceRanges. Item 4
has no Kube abstraction.

Item 3 is the one that is really problematic to me. There are so many
ways to implement external LB that we never specced the behavior well.
Maybe we should do that now wrt NetworkPolicy, NodePorts, etc.

Item 4 is much harder because Ingress is so limited.

> > So, just to clarify, it's possible for some hosts outside your cluster
> > to access some pods directly by their pod IP (or service IP?), without
> > passing through any sort of ingress/load
> > balancing/hostport/nodeport/externalIP, and so the NetworkPolicy can
> > then act on the host's IP?
>
> Yes and in fact we are considering making our pods accessible by
> our entire network at some point in the future and using some service
> discovery mechanism to make them discoverable.

This is the "normal" mode. I know OpenShift doesn't do it, but it's
what Kubernetes was designed for, what GCP has done since day1, and
what NetPol ingress ipBlocks was aimed at (IMO).

> --
> You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
> To post to this group, send email to kubernetes-...@googlegroups.com.