Key Source IP Restriction on espv2

[Ross G]

I have set up Cloud Endpoints with ESPv2 for GKE to recieve Webhooks from Stripe

To maximise security I have created a key that allow-lists the Stripe public IPs

gcloud alpha services api-keys create \

        --display-name="Stripe Webhooks" \

        --allowed-ips=$STRIPE_PUB_IPS \

        --api-target=service="${STRIPE_ENDPOINT_SERVICE_NAME}"

However the webhooks are blocked due to the source IP being the IP address of my private GKE node. It appears the source IP from Stripe is not preserved. Is this correct? and what can I do to pass the original source IP through?

{
httpRequest: {
latency: "0.026s"
protocol: "http"
remoteIp: "10.146.64.8"
requestMethod: "POST"
requestSize: "1660"
requestUrl: "/stripe?key=XXXXXXXXXXXXXRDUfVThbJUjQQVs5TYvi4v-UI"
responseSize: "217"
status: 403
}
  "insertId": "4195ccbe-35c1-4027-b67d-ebb64b1452084850908905639612439@a1",
  "jsonPayload": {
    "error_cause": "IP address blocked.",
    "api_key_state": "VERIFIED",
    "service_config_id": "2022-07-26r0",
    "service_agent": "ESPv2/2.37.0",
    "api_method": "1.stripe_gke_endpoints_devproj_dev20i_cloud_goog.Stripe",
    "http_status_code": 403,
    "producer_project_id": "devproj-dev20i",
    "response_code_detail": "service_control_check_error{IP_ADDRESS_BLOCKED}",
    "api_key": "XXXXXXXXXXXfVThbJUjQQVs5TYvi4v-UI",
    "api_version": "1.0.0",
    "log_message": "1.stripe_gke_endpoints_devproj_dev20i_cloud_goog.Stripe is called",
    "timestamp": 1658906703.4400651,
    "location": "us-east4-a",
    "api_name": "1.stripe_gke_endpoints_devproj_dev20i_cloud_goog"
  },
  "httpRequest": {
    "requestMethod": "POST",
    "requestUrl": "/stripe?key=XXXXXXXXXXXXXRDUfVThbJUjQQVs5TYvi4v-UI",
    "requestSize": "1660",
    "status": 403,
    "responseSize": "217",
    "remoteIp": "10.146.64.8",
    "latency": "0.026s",
    "protocol": "http"
  },
  "resource": {
    "type": "api",
    "labels": {
      "version": "1.0.0",
      "project_id": "devproj-dev20i",
      "service": "stripe.gke.endpoints.devproj-dev20i.cloud.goog",
      "location": "us-east4-a",
      "method": "1.stripe_gke_endpoints_devproj_dev20i_cloud_goog.Stripe"
    }
  },
  "timestamp": "2022-07-27T07:25:03.440065113Z",
  "severity": "ERROR",
  "logName": "projects/devproj-dev20i/logs/stripe.gke.endpoints.devproj-dev20i.cloud.goog%2Fendpoints_log",
  "trace": "projects/devproj-dev20i/traces/55fc1a4bfdda47e6b42adffe3be37d16",
  "receiveTimestamp": "2022-07-27T07:25:06.150356778Z"
}

[Osvaldo Lopez Acuña]

Have you tried without the alpha version? Do you have a load balancer in front of ESPv2? Have you followed and had any issue related to this setting steps [1]? You can also take a look at this similar case [2].

[1]:https://cloud.google.com/endpoints/docs/openapi/get-started-kubernetes-engine-espv2

[2]:https://github.com/GoogleCloudPlatform/esp-v2/issues/197  

[Ross G]

Confirming there is no load balancer it is just a Cloud Endpoint -> GKE Deployment

It appears the alpha track of gcloud alpha services api-keys is the only one with 'create' available

Yes I implemented the echo tutorial and am now adding production level controls everything is working except for this IP restriction on the API key

Looking at that issue [2] it appears exactly my problem but adding those flags did't work for me. I tried both and the remote IP is still the IP of my GKE node

"--envoy_xff_num_trusted_hops=0",

"--envoy_use_remote_address"

and

"--envoy_xff_num_trusted_hops=1",

"--envoy_use_remote_address"

My full deployment is 

# web-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stripe
  namespace: ${NAMESPACE}
spec:
  selector:
    matchLabels:
      app: stripe
  replicas: 1
  template:
    metadata:
      labels:
        app: stripe
    spec:
      containers:
      # [START esp]
      - name: esp
        image: gcr.io/endpoints-release/endpoints-runtime:2
        args: [
          "--listener_port=9000",
          "--backend=127.0.0.1:8080",
          "--service=${SERVICE_NAME}",
          "--rollout_strategy=managed",
          "--dns_resolver_addresses=169.254.169.254",
          "--ssl_server_cert_path", "/etc/esp/ssl",
          "--envoy_xff_num_trusted_hops=0",
          "--envoy_use_remote_address"
        ]
      # [END esp]
        ports:
        - containerPort: 9000
        volumeMounts:
          - mountPath: /etc/esp/ssl
            name: ssl-certs
            readOnly: true
      - name: run
        image: us-docker.pkg.dev/${PROJECT_ID}/stripe-webhooks/stripe-webhooks
        envFrom:
        - configMapRef:
            name: stripe-configmap
        env:
          - name: STRIPE_API_KEY
            valueFrom:
              secretKeyRef:
                name: stripe-api-key
                key:  api-key
          - name: STRIPE_WEBHOOK_SECRET
            valueFrom:
              secretKeyRef:
                name: stripe-webhook-key
                key: webhook-key
        ports:
        - containerPort: 8080
          protocol: TCP
      serviceAccountName: ${SERVICE_ACCOUNT}
      volumes:
        - name: ssl-certs
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "stripe-ssl-secrets"
      nodeSelector:
        cloud.google.com/gke-nodepool: ${NODE_POOL}

[Teju Nareddy]

If ESPv2 is not reading the correct IP address, then modifying the flag `--envoy_xff_num_trusted_hops` should be enough.

It really depends on your GKE deployment (e.g. if you have another load balancer in front of Cloud Endpoints). 

The default value is 2 for the common case GKE deployment, but you can try modifying it to either 1 or 3 and see what IP address ESPv2 reads. You should see the `remoteIp` field in the access log above changing.

--envoy_use_remote_address is probably not needed, you can leave that at the default value

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/b3f2250c-c10e-447b-b4a6-447859e1723en%40googlegroups.com.

--

Teju Nareddy nare...@google.com Software Engineer

[Ross G]

Thanks. Setting "--envoy_xff_num_trusted_hops=3" worked

Much appreciated