KEYS/GET problem between AWS regions

[Turbo Fredriksson]

I have setup a new region (London) in our AWS environment and are
trying to connect one of instances in there to our ElastiCache (Redis3.2)
cluster which is located in London.

I’ve connected the two regions with a Strongswan IPSEC VPN, running
on m4.large EC2 instances.


I get latencies of 20ms (not sure if that’s to much or if it’s just a fluke).

EC2 Instance (Redis client London) -> EC2 Instance (London VPN) -> VPN -> EC2 Instance (Ireland VPN) -> Redis

From any instance in Ireland, I get a latency of less than 1ms…

EC2 Instance (Redis client Ireland) -> Redis

If I try it from my laptop at work, it’s about 15ms.

Laptop (Redis client Office) -> VPN -> EC2 Instance (Ireland VPN) -> Redis

A coworker trying that last attempt, from his laptop at home instead of
mine at the office, get a latency number of ~25ms, so it might be latency
problems.


If I strace the redis-cli command on ‘Redis client London’, I see that the
initial connect completes without any (noticeable) delay. However, GET
on a value (~14kb), never completes. If I do it from a telnet session directly
to the Redis cluster, the GET eventually completes (after one and a half
minutes).

$ strace redis-cli -h elasticache.domain.tld -n 3 keys \*
connect(3, {sa_family=AF_INET, sin_port=htons(6379), sin_addr=inet_addr("10.127.1.88")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll([{fd=3, events=POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
fcntl(3, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(3, F_SETFL, O_RDWR) = 0
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [15], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [5], 4) = 0
setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = 0
write(3, "*2\r\n$6\r\nSELECT\r\n$1\r\n3\r\n", 23) = 23
read(3, "+OK\r\n", 16384) = 5
write(3, "*2\r\n$4\r\nkeys\r\n$1\r\n*\r\n", 21) = 21
read(3, "*524\r\n$62\r\n", 16384) = 11
read(3,

If I try to do a GET on a value that’s ~14kB, the same thing occurs - that
last write goes through, but not the following read()…


I’ve done several checks (and I’m also discussing this on the Strongswan
list), but I’m not sure what else to check - it might not be the VPN after all...

[hva...@gmail.com]

Small reads/writes work okay through the VPN, but large ones are extremely slow or time out?  That sounds like an MTU problem when going through the VPN.