[server] Stability of server-full
Tarek Ziade
People had some issues lately with server-full. Cloning our repo and
building the app was producing a non-working app.
That's basically because it's pulling the bleeding-edge version of every
component involved in building a Sync Server.
We usually don't have this issue internally because we use the RPM
system and are able to maintain our own list of
versions for every dependency, which is specific to our own deployments.
We use a few options when we build our apps.
See
http://docs.services.mozilla.com/server-devguide/release.html#building-the-rpms
But for the non-RPM version -- the one you're getting when you do "make
build", you will get a tip version for
our packages, and the latest available version of every dependency.
One problem we've hit lately was a regression in WebOb 1.0.8 that
affected server-full, but also our own deployments.
The latter is easy to fix, because we can 'pin' the version of WebOb
when we do our RPMs, but for people that run
their own server using "make build", that's not good enough as they
might get the latest WebOb version when they
refresh their servers using "make build".
In other words, we're missing a way for people to get a "stable" version
of our apps.
"stable" means here, something very *similar* to what we run in
production. I am saying _similar_ because we may
run our own versions at Mozilla in some environments, the community
might not want to run.
But the "development" version also needs a bit of stability: when you
want to pull the development version
of server-full, you don't want to be bothered with a regression in WebOb
1.0.8 -- You want to get the development
version for Mozilla's own packages, and pull the required set of
dependencies without worrying.
= Proposal =
I want to make an initial proposal for discussion based on how I think
we develop our apps. It's based on the idea of channels.
We add a "channel" option for the "make build" command, and we introduce
three channels: dev, beta, stable
people will be able to build the server using this new option:
$ make build dev
# builds the Sync server using the "bleeding-edge" version of each
dependency -- bleeding-edge means here that you get the same
version than what the project's developer use to work/add features to
the project. It might be the "tip" of internal dependencies, like
server-core, but maybe not. It might but the latest published version of
WebOb but
maybe not. It's up to the project team to define what "dev" means in
this context
$ make build beta
# builds the Sync server using a stable set of dependencies and a
version of the application that is almost stable but not yet in production.
I am thinking here about a new version of the application that might be
pushed in production very soon.
$ make build stable
# builds a Sync Server that is like the one we run in production. So you
get the bug fixes and features that made it to our own production
server, after
a QA pass and being used by a lot of people out there.
$ make build # run the stable build by default
$ make build SPEC_FILE
# do the building using a custom spec file that provides the versions to
use.
Technically speaking, developers on each project will maintain for each
channel a file that contains the required versions (possibly point to
some repositories).
"make build SPEC_FILE" will also allow anyone that's outside the team to
provide it's own custom spec file to the build process.
This problem is already solved in the Python community with different
tools and techniques (zc.buildout, pip requirements, etc).
For this proposal, a spec file would just be a text file that contains a
list of dependencies, and for each the version or where to get it (a git
repo
a hg repo or whatever -- technically it will probably be based on Pip's
requirements file --
see http://www.pip-installer.org/en/latest/requirement-format.html
For the RPM builds, we'll have the same options
$ make build_rpms dev
$ make build_rpms beta
$ make build_rpms stable
$ make build_rpms
$ make build_rpms HERES_THE_LIST.txt
Please comment !
Cheers
Tarek
--
Tarek Ziade - Mozilla Services
_______________________________________________
Services-dev mailing list
Servic...@mozilla.org
https://mail.mozilla.org/listinfo/services-dev
JR Conlin
Generally, the problem with a system like that is that it takes a great
deal of effort to ensure that build files stay up-to-date and that
there's a valid product created. It requires external package
maintainers to lock down versions of their code (or at least their
interfaces), notify dependents of new changes or breakages, and ensure
that things are back in working order. Tools like Jenkins can help,
obviously, but even those need to be well crafted to reflect the sorts
of dependencies that have grown up.
At Previous Co., we resolved it by locking down each release phase to
explicit version numbers of packages, with a bi-weekly upgrade check
that everyone hated doing.
tl,dr; It can be done, but it's thankless and a pain in the ass.
_______________________________________________
Tarek Ziade
> +1 (with qualifications)
>
> Generally, the problem with a system like that is that it takes a
> great deal of effort to ensure that build files stay up-to-date and
> that there's a valid product created.
- we have all the versions settled -- the project happily evolves on this
- there's no bi-weekly upgrades, but rather:
- a new project starts, uses new features for a dependency. we're
looking if we want to update other projects.
- a security fix or a bug that impacts us is fixed
A good example is SQLAlchemy. The 0.7.x line is the stable line for a
couple of months, but we're still on the 0.6.x line because there is no
killer feature in 0.7 and moving to it requires a bit of work because
it's backward compatible.
> It requires external package maintainers to lock down versions of
> their code (or at least their interfaces), notify dependents of new
> changes or breakages, and ensure that things are back in working
> order. Tools like Jenkins can help, obviously, but even those need to
> be well crafted to reflect the sorts of dependencies that have grown up.
>
> At Previous Co., we resolved it by locking down each release phase to
> explicit version numbers of packages, with a bi-weekly upgrade check
> that everyone hated doing.
>
> tl,dr; It can be done, but it's thankless and a pain in the ass.
so, if the pain in the ass part is about the upgrade frequency, I think
we don't want to upgrade dependencies that often.
e.g. once the versions are pinned down everywhere, they should work for
a while and I suspect we would need to do regular updates maybe every
quarter ? (and exceptional updates once in a while if something bad
happens I guess)
Maybe if we maintain a central place where we define the "recommended
versions", that would help when anyone changes a pin : she can check the
recommended versions list, maybe update it, and sends a head up here ?
But definitely I see your point about the maintenance burden
Cheers
Tarek
--
Tarek Ziade - Mozilla Services
_______________________________________________
Tarek Ziade
> Hey,
>
> People had some issues lately with server-full. Cloning our repo and
> building the app was producing a non-working app.
We've worked on this in the last weeks, and improved our build tools and
release process.
1/ server-full will now stay on the "latest stable" version when you run
"make build" in it -- basically the same versions we run in production.
2/ Until this point, the way our release process worked, was impacting
anyone that was updating the code during the release work. Since is not
the case anymore.
3/ if you want to jump into a bleeding-edge version, you can do it by
switching to the "dev channel".
All the gory details are here:
docs.services.mozilla.com/server-devguide/release.html