Running Tsuru in production (questions)

[Marcelo Freitas]

Hey guys,

I can't emphasize enough how excited I am to have found Tsuru. I hope it is / can become the great project I think it is. I just found it, and I'm love with it. After trying to re-invent the wheel (direct docker, fleet, own scheduler, node scaling, crazy nights, etc) and researching a few more PaaS I can tell Tsuru made good decisions. Node scaling/auto-scaling, app metrics, service API. Awesome job!

I'm playing with it, with a local 3-machine Vagrant cluster on my local machine and I have some questions, if someone happens to have the answers. Sorry, the docs are not complete and I could only read so much code in the last 2 days... appealing for the community good will ;-)

- Can I have more than 1 server per cluster ? My best guess is that can only be 1, right? If so, what happens if it crashes? Do I need to bring up another server, restore a mongo db backup and re-register all the nodes to the new server, or there is an easier/faster/more fault-tolerant way to do it?

- I'm using Tsuru-now script to provision my machines, besides saying in the project page I can provision a load balancer node only (without other server components), I couldn't see the option in the bash script (--template all | dockerfarm | server | client). Is it missing from the script by mistake, I didn't see it or cannot be done? If cannot be done (hipache/redis dependency, routes replication or something) I can just have one load balancer per cluster?

Thanks in advance,

[Cezar Sá Espinola]

Just realised I answered only to Marcelo, forwarding to the group.

-

Hi Marcelo,

By servers you mean tsuru API servers, right? You can have as many as you want, as long as they all share an identical tsuru.conf file. It's actually recommended to have at least 2 API servers in a production environment. It's a good idea to have a load balancer in front of them, if you're deploying a tsuru cluster on Amazon you could use the ELB service.

To make MongoDB fault tolerant it's recommended using replica sets, you can find more about them in http://docs.mongodb.org/manual/replication/.

tsuru-now script is nice for creating a demo environment, however it's not the recommended way to create a production environment. A better option for now is using puppet [1] or ansible [2] to provision your environment.

A basic overview of the components you would need for a production environment is:

Hipache router: You can have multiple servers pointing to the Redis instance. You'll need a load balancer (ELB is an option if deploying on AWS) if you have more than one server. More details in http://docs.tsuru.io/en/stable/installing/hipache-router.html

Redis: Used by Hipache router and referenced in tsuru.conf.

MongoDB: You can use have many servers configured as a replica set. It should be referenced in tsuru.conf file.

Gandalf: Optional. If you want to use git push to deploy your applications you need a gandalf server installed. You can have multiple gandalf servers behind a load balancer, however as Gandalf rely on file system storage so you'll have to make sure all servers share the same storage for repositories (using NFS mount points is a good option). More details in http://docs.tsuru.io/en/stable/installing/gandalf.html

Docker registry: Docker registry installation can also span multiple servers as long as they share the same storage. See https://docs.docker.com/registry/deploying/ and http://docs.tsuru.io/en/stable/reference/config.html#docker-registry

Tsuru server: As I said before, you can also have multiple API servers under a load balancer. Just make sure they all share the same tsuru.conf file. Details in http://docs.tsuru.io/en/stable/installing/api.html

Docker nodes: The best option for creating docker nodes is letting tsuru use an IaaS provider to dynamically create and manage them, this will allow you to enable node healing and scaling for a more robust environment. More details in http://docs.tsuru.io/en/stable/installing/adding-nodes.html#managed-nodes

[1] - https://github.com/tsuru/puppet-tsuru

[2] - https://github.com/jblancett/tsuru-ansible-role

Best,

--

Cezar Sá Espinola

--
You received this message because you are subscribed to the Google Groups "tsuru" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tsuru-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marcelo Freitas]

Hey Cezar,

Thanks for the excellent answer. That was exactly what I needed to know. The inner workings of the different components and their relations.

Regards,

Marcelo

P.S: The suggestions were also invaluable

[Josh Blancett]

Just thought I'd add to Cezar's post about replication:

For docker-registry I use the latest version from the docker repos that supports s3 file storage

I don't know if this is officially supported by the tsuru team but I have had no issues with it so far

[Marcelo Freitas]

Hey Josh,

Very nice. I'll definitely use that.

I was also checking the new AWS Elastic file system (still in preview) for the Gandalf/Docker Registry server. It looks like a good option. At least using S3 for the registry is a lot cheaper and flexible

[Cezar Sá Espinola]

Josh, using the docker-registry with s3 or swift works just fine, we rely use NFS currently but we'll soon migrate to swift on our internal tsuru environment.

Concerning Gandalf, having to use the filesystem is really a pain. I didn't know about Elastic file system, it seems a really good option if deploying on AWS.

Other options worth exploring for Gandalf would be using a library to upload to an object storage. I found some experiments around [1] [2] but nothing seems quite production ready yet. 

[1] - http://techs.enovance.com/6643/openstack-swift-as-backend-for-git-part-2

[2] - https://github.com/mbr/amazing-git

[Cezar Sá Espinola]

Another addition to the list of possibilities for Gandalf, using libgit2:

http://blog.deveo.com/your-git-repository-in-a-database-pluggable-backends-in-libgit2/

Cezar Sá Espinola