Iris on CoreOS / EC2 - initial findings

[James Cooper]

Hi folks,

Executive summary: it works!

Github repo with notes/scripts if you want to try it: https://github.com/coopernurse/iris-bench

- Iris works when run on the CoreOS host OS (not in a container)

- You have to run in a VPC.  If you're not in a VPC, then assumptions Iris makes about seeds being in subnets aren't valid. I think this may be solvable, but I'd like to get Peter's opinion.

- Convergence time is awesome.  I'm seeing around ~5 seconds.

- Docker containers can talk to the Iris relay and pass messages to processes on other CoreOS hosts in the same cluster (!!)

I will push a github repo with some scripts and notes if others want to try and repeat this test.  I made a simple Go Iris client/server benchmark and packaged that as a docker image.

Some things that would be interesting to discuss:

(a) Do we want to support non-VPC EC2 deployments?  The behavior I saw was pretty simple: the bootstrapper found the etcd nodes, but their IP addresses weren't in the host's subnet, so they were skipped.

Personally I don't think this is a huge deal.  VPCs are free, and you should probably be using them anyhow.

(b) Do we want to support running Iris as a container?

This seems like a reasonable idea.  I did write a user-data script that will install Iris as a binary on the host OS, and this works fine, but it would be nice to fully embrace the "container" concept if possible.  The issue appears to be the random high ports that are used by the pastry code.  I don't have a good understanding of why random ports are used.

(c) Do you need a wide open security group?

When I saw the random ports I just made the security group completely open between all nodes in the security group.  This doesn't bother me that much, but I wonder if that's required.

But  overall this is great stuff and I want to keep experimenting with it.

-- James

[Péter Szilágyi]

Hey James,

  Great to hear :)

  Relating to the issues you've raised:

1. The problem with not running inside a VPC (or same virtual network on GCE), is that you could have ugly problems caused by firewall rules. If all nodes reside within one single VPC, I can assume that they are all accessible from one another. However, if they are in different firewall zones, and one blocks outside access, then this will lead to a weird/corrupt/unstable overlay network. It's similar with running cross cloud/availability zone deployments, or spanning multiple physical networks. There are a lot of things that can go wrong so I'd like to keep such setups explicit and not seamlessly integrate into the overlay.
2. Yes, I would like to containerize Iris. This is a known design "issue" of Iris (being born before the whole docker hype), that it selects ports at random to avoid collisions with other services on the host machine. The bootstrapping ports of Iris are fixed (pool of 3 currently), but all others are picked randomly, and advertised during bootstrapping. This ensures that only one of the bootstrap ports need to be free, others cannot collide. Unfortunately, if you need firewall pass-through, then this decision is a problem. Iáll have to think on how best to work around it without too many ugly hacks.
   
3. Currently pretty much yes due to the above design decision. Imho this shouldn't bee a problem, as most cloud services will either way run in isolated virtualized networks. Firewalls can really mess up things, so if you're already isolated, and communication is encrypted via Iris anyway, I see no issue with the open system. But I'm definitely open to be proven otherwise :)

  I'll try and look at the code closer too, but I'm more of a Google Compute Engine guy, so that's the easier one for me :)

Cheers,

  Peter

--
You received this message because you are subscribed to the Google Groups "Iris cloud messaging" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-iris...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.