Errors when performance testing AMP. 503 "Service Unavailable" and 403 "Forbidden".

[Jeremy Whiting]

Hi,
 I am performance testing AMP 3scale 2.4 on a dedicated lab. Using a clustered Openshift 3.10 installation.

 When scaling up the load I am seeing two error types in addition to successful responses.
 Am using Buddhi project to configure apicast before the test run. Using the "paths" api to retrieve correctly configured test urls.

 These two errors indicate something went wrong with configuring Apicast in all pods. Or that Apicast has a limitation on the number of clients.

 The pods backend-listener at 4 and backend-worker 2 replicas.

 Checking the logs for apicast I am seeing the following appear sometimes.

[14/Jan/2019:09:47:35 +0000] 074bd837-1808-4660-a9ef-d53ba30f188d.benchmark.acme.com:8080 10.128.0.1:39682 "GET /111?user_key=f4868ab7f2460a0f HTTP/1.1" 502 179 (0.000) 0
2019/01/14 09:47:35 [error] 34#34: *173322 connect() failed (111: Connection refused) while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /1?app_id=4440a535a3570de4&app_key=c60fdc6e16584e44 HTTP/1.1", upstream: "http://xxx.xxx.xxx.xxx:80G1:8080/1?app_id=4440a535a3570de4&app_key=c60fdc6e16584e44", host: "benchserver6"
2019/01/14 09:47:35 [error] 34#34: *182239 connect() failed (111: Connection refused) while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /111?app_id=7f562f08bd9b7ce9&app_key=70bf4e7e7592d469 HTTP/1.1", upstream: "http://xxx.xxx.xxx.xxx:80G1:8080/111?app_id=7f562f08bd9b7ce9&app_key=70bf4e7e7592d469", host: "benchserver6"
2019/01/14 09:47:35 [error] 34#34: *181266 connect() failed (111: Connection refused) while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /11?app_id=ff639c0c4f84a678&app_key=487099fd8bc8e550 HTTP/1.1", upstream: "http://xxx.xxx.xxx.xxx:80G1:8080/11?app_id=ff639c0c4f84a678&app_key=487099fd8bc8e550", host: "benchserver6"
[14/Jan/2019:09:47:35 +0000] 95218b57-d521-4ef4-a8c6-dea73cbf19f0.benchmark.acme.com:8080 10.128.0.1:39602 "GET /1?app_id=4440a535a3570de4&app_key=c60fdc6e16584e44 HTTP/1.1" 502 179 (0.000) 0

Checking back to the Buddhi service to retrieve the configuration I see this.

# curl -v localhost:8089/admin/api/services/074bd837-1808-4660-a9ef-d53ba30f188d/proxy/configs/production/latest.json
{"proxy_config":{"content":{"id":"074bd837-1808-4660-a9ef-d53ba30f188d","backend_authentication_type":"provider_key","backend_authentication_value":"11839bce-8dba-4634-a62f-75988edd2396","backend_version":"1","proxy":{"api_backend":"http://benchserver6G1:8080/","hosts":["074bd837-1808-4660-a9ef-d53ba30f188d.benchmark.acme.com"],"backend":{"endpoint":"http://172.30.141.128:3000"},"proxy_rules":[{"http_method":"GET","pattern":"/1","metric_system_name":"5382dbd7-9109-4f33-9d5e-d73ae0588f00","delta":1},{"http_method":"GET","pattern":"/11","metric_system_name":"d46dfe9f-2f74-4412-8996-e9a2aeb0b62b","delta":1},{"http_method":"GET","pattern":"/111","metric_system_name":"ef401de7-9cd0-45c1-ad11-af1566fb5af8","delta":1}]}}}}
#

 It seems apicast will mangle the api_backend url. When it attempts to substitute the hostname with an IP.
 

"api_backend":"http://benchserver6G1:8080/"

 becomes

upstream: "http://xxx.xxx.xxx.xxx:80G1:8080/1?app_id=4440a535a3570de4&app_key=c60fdc6e16584e44"

 the context and parameters are not a concern. It's the hostname substitution. The hostname resolution worked fine. I redacted the IP address to xxx.xxx.xxx.xxx

 At a guess I'd say the regex parsing does not interpret correctly the initial number in the hostname. That is wrongly interpreted as the port number.

 I am using Apicast image quay.io/3scale/amp@sha256:de778af728c945a68cdc3ce24cc90ecd3cacdb8e95eeead4721483aca0b38a48

 Has this been seen before ?

Jeremy