Ideally I'd like to be able to spin up a clean CoreOS cluster running fleet and then just start scheduling units and magic happens. What I'm finding is that I often need a bit of scripting before my unit kicks off my docker container. Even just running a simple docker container needs some scripting because you do something like "docker start -a || docker attach || docker run" (yeah... launching docker from systemd is neither fun, easy, or reliable, but that's a whole d ifferent thread).
launching docker from systemd is neither fun, easy, or reliable, but that's a whole different thread).
I was thinking something like a subset of cloud-config.
I'm not sure I'm quite convinced that this is a separate issue. If docker integration was seamless (and automatically garbage collected), doesn't it provide exactly the kind of thing you're describing? A self-contained, self-describing environment that's fully usable with a single entry point; what else would you really need to bootstrap?
Could you explain what other kind of things you would want to achieve with this beyond your write_files examples? (Since again I would assume those could be handled within the container, either during the container build or by whatever the entrypoint is).
Well, docker/systemd integration is not seamless and I see no solution that will make it so in a reasonable amount of time, so I don't know what to say. It's kind like saying, "If it did everything you needed, wouldn't that be sufficient?" I'm not trying to be obnoxious, its just that relating a docker container to a systemd service today doesn't fit well at all. In a separate thread I can enumerate all my issues. You can see a small bit of my frustration at https://github.com/dotcloud/docker/issues/6791
write_files is the main use case. I don't think I follow what you are saying about "container build".
Maybe I'm alone here on this one. For example. Say I do "docker run --name myservice someimage:42". Now later I want to deploy the same service built off of someimage:43. What I'd like to do in the start up is do "docker ps" and see if the container exists and is using an old image, if it is delete it and create new container from the new image using the same data volumes.
ExecStart=/usr/bin/docker run busybox /bin/sh -c "while true; do echo Hello World; sleep 1; done"
Alex,
Thanks for responding and I'm glad we can have this discussion. I'm having very similar discussions with the Docker folks in parallel, so don't just think that I'm picking on CoreOS. I'm encouraged at the responses I've been getting and I think we can all come to an agreement. Your email touches on so many points that you'll have to bear with me if this email goes a bit long. I'll address this is in two main parts, first systemd/docker integration, and second fleet.
Red Hat is in a really crummy situation from what I see. I really empathize with them. As the “owners” of systemd, SELinux, and OpenShift they had all the technology they needed to accomplish their goals. I'm sure they were quite content with the path they were on but then this cute little hack called Docker came onto the scene and captured the hearts of the users. They have no choice but to be involved and use Docker, but honestly Docker is far more immature than what they had and additionally it's design does not play well with their current architecture. With the current investment they have they'd be silly and stupid to not try to leverage as much as they can from systemd, SELinux, etc. The path that Red Hat has chosen and their fight for systemd seems somewhat natural given their circumstances.
Whenever some new technology comes around there's always a natural reaction to look at the current technology you have, move some pieces around, and declare it just as good that the new shiny technology. Why, is this? Because there's no such thing as new technology these days. It was all invented many, many years. Today we don't innovate with technology, but with paradigms. And this is why the shiny new technology is better, it was assembled and tweaked under a new paradigm. As you shuffle around the old technology to match the new, you dissect and normalize the paradigm and often lose the essence of it. So while systemd and Docker are both assembled on the same building blocks, there is just enough of a paradigm shift with Docker that it has really resonated with users. You really need to respect that. It's potentially dangerous to mold Docker to fit systemd because, in the process, you could lose the essence of what made it great to begin with.
I feel this is one of the difficulties of moving Docker to a standalone architecture. It's not just the technical effort, it's more about how this change could compromise the vision of Docker. From the perspective of Docker you're trying to fit a legacy paradigm while Docker is trying to move forward with their new view of the world. While technically feasible, the subtleties in the design of Docker are the tricky part, but those subtleties are the magic of Docker. You can't really fault Docker for their desire to own the fork/exec. It is the natural approach to have a daemon that spawns and manages its children. systemd does the same thing, it just happens to be PID 1.
I'm currently working with the Docker folks to fully flush out the design of standalone mode. I think at this point, it’s essential that we fully explore this option with real intent of implementing it. I think there could be some advantages to this model, but if, and only if, it stays consistent with the vision of Docker. If I feel it's an unnatural distraction in the progression of the Docker architecture I will be opposed to it.
While both sides may have their ideal solution, my gut feeling is that there will be a very simple compromise that will get 80% of what both sides need. Then later you'll find out the other 20% wasn't really all that important. I'm also pursuing with Docker what I think would be a short term solution also.
Should fleet talk directly to docker? Absolutely. Imagine if the current landscape today was that we didn't have systemd but instead Upstart and SysV. If you were going to build a container orchestration tool, do you think you would have opted to have tight integration into Upstart? Probably not. It just happens to be that systemd includes so much container oriented functionality that as architects and programmers we feel there must be a way to happily merge these twos. Now having said that, I don't think fleet should stop managing systemd units either, but more about that later.
To conclude on the systemd/docker integration path. I'm glad to hear you guys are open to different approaches and I'm actively working to resolve this and so I think we can join forces. But... if I was to trust my spidey sense, I'd say that systemd and docker will largely stay independent. Time will tell.
Now to fleet.
If you were to ask me what is fleet I'd say fleet is a systemd scheduler. I'm sure it has a grander vision than that but from an outside perspective that is what it appears to be. I struggle with the usefulness of that. I think the easiest way to describe why I struggle with that is to propose a slightly different vision for fleet. Instead of saying fleet is a scheduler or an orchestration tool, I would propose that it's role is to be the CoreOS cluster manager. It's the brains that glues everything together everything. If you create a node all you should need to do is join it to fleet. Then, based on the configuration in fleet the node should be configured and tasks deployed to it. This means fleet should pull together and offer up to the users more first class concepts than just systemd units. It should mergeor wrap functionality from cloud config, confd, systemd management, docker management, concepts from Kelsey’s ipxe profiles, etc.
CoreOS should be the ideal platform for running containers. It comes with all the plumbing needed so you can largely focus on your workload. But... that doesn't mean CoreOS should be heavily involved in scheduling/orchestration right now. Eventually, maybe, right now no. I say this for two reasons. First, for smaller size clusters, the scheduling is quite simple and so regardless fleet should work for smaller setups. For a larger size clusters people are going to gravitate towards a higher level scheduler or orchestration system like Mesos or Kubernetes. Second, if you get heavily into a scheduling/orchestration it usually come with some opinionated view on how to do things. For example, Kubernetes introduces pods and a different networking model.
Taking these points into consideration I think that instead of focusing on scheduling/orchestration, fleet should be a very unopinionated tool that can bootstrap all the other frameworks. Because of the features built into fleet you can easy run a command and have Kubetnrtes, Mesos, or Deis running fully containerized and managed. Not like you have today where you’re largely dropping files in /opt and the writing unit files. In the end, CoreOS will be the best OS to run all these systems or if you have basic needs you can just use bare CoreOS.
The ability to bootstrap these other systems requires a bunch of functionality Docker and CoreOS don't have today. Most people think of containers as a thing to run an application. But there's a different mode that I'd like to call system containers. The basic idea is that you create a container that shares namespaces with the host or other containers to add services to them. Example use cases would be monitoring tools, security frameworks, software defined storage and networking components, or, most important of all, agents for a higher level orchestration tools. The scheduling of these containers are quite different too. Most of the time you just want to define one template and then say instantiate this template once per node.
So to some this all up, my opinion is that
1. Fleet should be the cluster manager that gives you a single point to configure and control your nodes.
2. Fleet should expose more first class concepts than just units files. Pull in concepts from cloud config, confd, ipxe profiles, etc
3. Fleet should provide basic scheduling needs for small simple clusters
4. Fleet should be able to easily bootstrap high levels orchestration tools.
I think if you focus on making fleet more like a tool like kubernetes you doing a disservice. You don’t need to compete or provide an alternate perspective but instead encourage the wide ecosystem that is growing and ensure that CoreOS is the best platform to run all of it.
Darren
Systemd has an extremely rich syntax that can describe the attributes of a particular service. Your services can express hard or soft dependencies, the order of launch relative to those dependencies, and identify conflicting services. Docker containers are much easier to manage when you can specify whether they automatically restart per container and customize the timing for restarting.