Snakebite: Beyond Buildbot
Trent Nelson
week or so. I started this reply to him this morning, but by the
time I'd finished it I had mentioned a lot of the neat ideas I've
got in mind for Snakebite and the concept of "beyond Buildbot...",
so I figured I'd send it to the list for everyone to benefit.
It's too late for a PyCon US presentation on this topic this year
but hopefully my OSCON one gets accepted; I'll go into a lot more
detail about our long term plans specifically for this integrated
development environment alluded to in the thread below.
Enjoy!
Trent.
On Wed, Feb 04, 2009 at 06:54:10PM -0800, Mike Melanson wrote:
> Trent Nelson wrote:
> >On Tue, Feb 03, 2009 at 05:57:07PM -0800, Mike Melanson wrote:
> >>And if it helps in the decision-making process, I thought I would
> >>mention that the build client part of the continuous integration
> >>architecture is written in Python. :)
> >
> > That certainly helps a lot more than say, if it were written in
> > Perl ;-P Will you be attending OSCON this year? I've submitted
> > a proposal for a couple of Snakebite talks; one in particular,
> > "Beyond Buildbot...", talks about the long term vision we've got
> > for revamping the notion of continuous integration etc.
>
> Had no intention of attending OSCON this year.
>
> About CI, though: I have been working on my CI system for over a year.
> But I only recently figured out that what I was doing was called CI. Now
> that I know what to look for, I'm actually evaluating other CI packages
> to see if they might fit my needs.
>
> I gather you don't recommend BuildBot?
Oh no, not at all, Buildbot is great for what it is. The things
we've got in mind for Snakebite, though, are light years ahead of
Buildbot's humble aspirations ;-)
Let's say you're hacking away on some ffmpeg code on Linux with
vim or emacs. In the background, you've got a little Snakebite
daemon running, closely watching your source code directory.
Each time source code files are saved locally, the daemon ferries
off the deltas to an exact replica of your source tree that's on
your home drive on the Snakebite network. (It'll automagically
figure out how best to get the deltas from your local system to
Snakebite regardless of what sort of network connectivity you
have; you could be at home on a 20Mb DSL line, or you could be
at work with access to no more than a HTTP proxy on port 80.)
Now, after 20-30 minutes of coding, you kick off a build. You
might just do this via make; perhaps you're working in Windows,
though, and build the solution via Visual Studio. Or XCode...
Your build process has been slightly augmented to send a little
note to the Snakebite daemon each time you invoke a build. The
result? Each local build you invoke automatically triggers a
remote build on every one of your project's supported platforms.
The results from all of these remote builds are automatically
aggregated by the daemon and presented for your consumption. In
an instant, you get feedback that your changes don't compile on
Solaris 10 SPARC with Sun's Compiler, or on DragonFlyBSD 2.1 w/
the experimental Intel compiler port. You're told whether or
not the change you've just made compiles on every single one
of your supported platforms/configurations, without lifting a
finger.
Finally get to the point where your code is compiling cleanly
on every platform? Next step is to test. You might just run
a couple of unit tests via a script, or you could launch an
IPython console and do some interactive testing on the API
changes you've just made via your Python bindings (or ctypes).
Or maybe you're nearing the end of this particular change and
want to run your entire regression test suite.
As your test framework has been augmented to support Snakebite
(one of the pre-requisites of being brought onto the network),
any form of testing you do locally will be mirrored on all
platforms. The test results are aggregated the same way as the
build results; you get immediate feedback on any platforms that
have tests that now fail after your changes. (Note the subtle
distinction between detecting if a test has failed versus detec-
ting that a failed test that used to pass before your change.)
And that's only the beginning. I think it's safe to say that
the majority of open source developers don't primarily develop
on Windows, and that a lot of projects generally have issues
debugging Windows problems much more often than Unix-xy type
problems, mainly due to lack of resources.
Let's say your changes that you've been working on locally in
Linux compile fine everywhere. So, you decide to run your
entire regression test suite. After a few seconds, you get
a little notice saying that one of the tests crashed your app
on Windows. However, by the time you get that message, the
Snakebite worker process running on the Windows box has actu-
ally restarted the test and placed a breakpoint a couple of
methods before the point the crash occurred.
So when it tells you that a test crashed on Windows, what it
will actually say is "Test xxx crashed on Windows 7. Debug?";
You press 'Yes' to debug. Moments later, Visual Studio appears
on your desktop. As in, literally, Visual Studio, just as if
you were in Windows and you'd launched it directly. And guess
what? It's in debug mode -- it's just hit the breakpoint that
was automatically set based on the test that crashed. (Behind
the scenes, your local Snakebite daemon and the remote worker
on the Windows box have collaborated to launch an RDP session
for Visual Studio.)
Without any effort whatsoever (except for the effort involved
in writing code that crashes on Windows), you've gone from
kicking off a local regression test suite run, to being at the
helm of an-about-to-crash-a-few-lines-down-from-here Visual
Studio, ready for you to investigate.
Now that, my friends, is a brief glimpse at how Snakebite will
forge the future of open source development.
Trent.
Brett Cannon
>
> Mike Melanson (ffmpeg) and I have swapped a few e-mails the past
> week or so. I started this reply to him this morning, but by the
> time I'd finished it I had mentioned a lot of the neat ideas I've
> got in mind for Snakebite and the concept of "beyond Buildbot...",
> so I figured I'd send it to the list for everyone to benefit.
>
> It's too late for a PyCon US presentation on this topic this year
> but hopefully my OSCON one gets accepted; I'll go into a lot more
> detail about our long term plans specifically for this integrated
> development environment alluded to in the thread below.
>
You could do a lightning talk or Open Space on it.
> Enjoy!
>
I have no clue how fleshed out anything below is in your head, Trent,
but I am going to comment anyway. =)
I assume this is controllable as to only occur when I successfully
build locally? No point in trying to build all over the place before I
can get something built.
> The results from all of these remote builds are automatically
> aggregated by the daemon and presented for your consumption. In
> an instant, you get feedback that your changes don't compile on
> Solaris 10 SPARC with Sun's Compiler, or on DragonFlyBSD 2.1 w/
> the experimental Intel compiler port. You're told whether or
> not the change you've just made compiles on every single one
> of your supported platforms/configurations, without lifting a
> finger.
>
How are you going to handle that granularity? For instance, if I am
hacking on CPython, some extension modules work in some places and not
others.
> Finally get to the point where your code is compiling cleanly
> on every platform? Next step is to test. You might just run
> a couple of unit tests via a script, or you could launch an
> IPython console and do some interactive testing on the API
> changes you've just made via your Python bindings (or ctypes).
> Or maybe you're nearing the end of this particular change and
> want to run your entire regression test suite.
>
> As your test framework has been augmented to support Snakebite
> (one of the pre-requisites of being brought onto the network),
> any form of testing you do locally will be mirrored on all
> platforms. The test results are aggregated the same way as the
> build results; you get immediate feedback on any platforms that
> have tests that now fail after your changes. (Note the subtle
> distinction between detecting if a test has failed versus detec-
> ting that a failed test that used to pass before your change.)
>
Woohoo! This would be nice as I know I get tired of buildbot
notifications that put me on the blamelist when the failing tests were
already failing before I came along.
Holy crap! If you pull that off, Trent, ... wow.
-Brett
Antoine Pitrou
> Woohoo! This would be nice as I know I get tired of buildbot
> notifications that put me on the blamelist when the failing tests were
> already failing before I came along.
In all honestly, it doesn't seem to happen very often.
More annoying is that we have some flaky tests which fail erratically on
some platforms, and as a consequence people probably don't care a lot
about the buildbot failure messages.
Brett Cannon
I should have been more specific, but the flaky tests fall under that
category as well. Once importlib is squared away I plan on tackling
the test issue to hopefully eliminate the flaky tests so that buildbot
failures actually start to matter again.
-Brett
Trent Nelson
> I have no clue how fleshed out anything below is in your head, Trent,
> but I am going to comment anyway. =)
Hah, moderately! Glad it piqued your interest though; it was what I
was hinting at when I sent my "Ohhh Brett, so innocent!" response a
few weeks ago ;-P
> > Let's say you're hacking away on some ffmpeg code on Linux with
> > vim or emacs. In the background, you've got a little Snakebite
> > daemon running, closely watching your source code directory.
> >
> > Each time source code files are saved locally, the daemon ferries
> > off the deltas to an exact replica of your source tree that's on
> > your home drive on the Snakebite network. (It'll automagically
> > figure out how best to get the deltas from your local system to
> > Snakebite regardless of what sort of network connectivity you
> > have; you could be at home on a 20Mb DSL line, or you could be
> > at work with access to no more than a HTTP proxy on port 80.)
> >
> > Now, after 20-30 minutes of coding, you kick off a build. You
> > might just do this via make; perhaps you're working in Windows,
> > though, and build the solution via Visual Studio. Or XCode...
> >
> > Your build process has been slightly augmented to send a little
> > note to the Snakebite daemon each time you invoke a build. The
> > result? Each local build you invoke automatically triggers a
> > remote build on every one of your project's supported platforms.
> >
>
> I assume this is controllable as to only occur when I successfully
> build locally? No point in trying to build all over the place before I
> can get something built.
Indeed; aggregating the information will only be a small part of
the battle. Deciding when (and how much) to display to the user,
though, will be key. And the less interaction we require from the
user to help make that decision, the better. Last thing you want
is to get out of Flow 'cause you were bombarded with information
you didn't care about.
(If only the little daemon could also read minds eh; I'd prefer
it to figure out that I don't care about something without me
actually having to tell it.)
> > The results from all of these remote builds are automatically
> > aggregated by the daemon and presented for your consumption. In
> > an instant, you get feedback that your changes don't compile on
> > Solaris 10 SPARC with Sun's Compiler, or on DragonFlyBSD 2.1 w/
> > the experimental Intel compiler port. You're told whether or
> > not the change you've just made compiles on every single one
> > of your supported platforms/configurations, without lifting a
> > finger.
> >
>
> How are you going to handle that granularity? For instance, if I am
> hacking on CPython, some extension modules work in some places and not
> others.
There needs to be some concept of project profiles and delineated
'development sessions', which would be backed to a given profile.
That is, before you start hacking away on something, you give the
little Snakebite daemon a bit of a helping hand with regards to
stipulating what it is you're actually working on (i.e. via your
project's pre-configured profiles).
A SQLAlchemy development session, where you care less about the
OS, but much more about being able to run code against multiple
database vendors (with the same intelligent result aggregation),
is going to behave significantly different to a core CPython dev
session where you're hacking away on internals.
> > Finally get to the point where your code is compiling cleanly
> > on every platform? Next step is to test. You might just run
> > a couple of unit tests via a script, or you could launch an
> > IPython console and do some interactive testing on the API
> > changes you've just made via your Python bindings (or ctypes).
> > Or maybe you're nearing the end of this particular change and
> > want to run your entire regression test suite.
> >
> > As your test framework has been augmented to support Snakebite
> > (one of the pre-requisites of being brought onto the network),
> > any form of testing you do locally will be mirrored on all
> > platforms. The test results are aggregated the same way as the
> > build results; you get immediate feedback on any platforms that
> > have tests that now fail after your changes. (Note the subtle
> > distinction between detecting if a test has failed versus detec-
> > ting that a failed test that used to pass before your change.)
> >
>
> Woohoo! This would be nice as I know I get tired of buildbot
> notifications that put me on the blamelist when the failing tests were
> already failing before I came along.
The fact that there are so many false positives when it comes to
our buildbots is very disheartening. It only takes a handful of
successive false positives before you start losing confidence in
those failure notices entirely. It shouldn't be like that!
(The idea behind Snakebite is that you don't need to wait until
you've checked in your code to potentially find out that there
are problems on esoteric platforms; you get all that information
right from the get-go.)
I do like to live with my head in the clouds, I admit ;-) That
said, everything I've alluded to is possible with today's technol-
ogy. There's no reliance on something wacky like requiring each
developer to be connected to a neural EEG whilst developing, such
that brain waves can be data-mined by the little Snakebite daemon
so it can figure out what you mean without you having to explicitly
say it. Predictive text 2.0, if you will.
(If a company can produce head gear that can differentiate brain
signals to the point where you can control your little character
in Halo just by willing it, surely we can adopt that to differen-
tiate between the brain waves emitted when you think "no, sorry,
I could not care less about that Solaris 10 compilation failure
you've just reported to me", versus "crap, broke Windows again,
ok, launch another remote session".)
See, now when you factor that sort of stuff in, my original e-mail
doesn't seem so far fetched after all eh? :-P It's all relative!
> -Brett
Trent.
Jesus Cea
Hash: SHA1
On 05/02/09 14:32, Trent Nelson wrote:
>
> Mike Melanson (ffmpeg) and I have swapped a few e-mails the past
> week or so. I started this reply to him this morning, but by the
> time I'd finished it I had mentioned a lot of the neat ideas I've
> got in mind for Snakebite and the concept of "beyond Buildbot...",
> so I figured I'd send it to the list for everyone to benefit.
>
> It's too late for a PyCon US presentation on this topic this year
> but hopefully my OSCON one gets accepted; I'll go into a lot more
> detail about our long term plans specifically for this integrated
> development environment alluded to in the thread below.
This email is almost 2 years old. The website still says "Snakebite will
be publicly announced soon".
I guess the project is dead. Isn't it?.
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
jc...@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:jc...@jabber.org _/_/ _/_/ _/_/_/_/_/
. _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQCVAwUBTOPpx5lgi5GaxT1NAQK8kgP/VnkFIgLaY8D0hVXhlw6zIRvWLBooOGrx
F8xu0c4xbjy9x+jX+9ROpYH8WVCRdMh06gUGtmDZEt67aCVsXE5md/hE9vedGVY/
hwLfVD1y+F3qkeTwHpqIRztDwn7rrt9rMcygr1X69oXxX2BhRjqyA4AHni+/UJOq
N2zc0Dr0tfk=
=BwSk
-----END PGP SIGNATURE-----
C. Titus Brown
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 05/02/09 14:32, Trent Nelson wrote:
> >
> > Mike Melanson (ffmpeg) and I have swapped a few e-mails the past
> > week or so. I started this reply to him this morning, but by the
> > time I'd finished it I had mentioned a lot of the neat ideas I've
> > got in mind for Snakebite and the concept of "beyond Buildbot...",
> > so I figured I'd send it to the list for everyone to benefit.
> >
> > It's too late for a PyCon US presentation on this topic this year
> > but hopefully my OSCON one gets accepted; I'll go into a lot more
> > detail about our long term plans specifically for this integrated
> > development environment alluded to in the thread below.
>
> http://www.snakebite.org/
>
> This email is almost 2 years old. The website still says "Snakebite will
> be publicly announced soon".
>
> I guess the project is dead. Isn't it?.
No, no, it's just pining for the fjords...
Seriously, I'll let Trent answer with details, but the short answer is
that Trent is still working on it. There was actually a presentation
at the testing-in-python BoF/PyCon 2010 about some of the, umm, hardware
challenges; I think Trent decided to stop overpromising and try to just
get stuff working. And it's taken a long time.
--titus
--
C. Titus Brown, c...@msu.edu
Trent Nelson
Hey, neat, the mailing list still works ;-)
Seriously though -- it couldn't be further from being dead. And Titus
is spot on regarding the decision to stop overpromising and just crack on.
The two biggest issues I had to face were a) dodgy hardware, and b) the
temperature increase that comes with putting three racks of servers in
an extremely small room. Now that these two things have basically been
addressed, the project is picking up lots of traction, once again.
But I'm still not going to start foaming about it publicly again until
there's something tangible (network wise) doing something useful
(software wise) ;-)
Trent.
Nick Coghlan
Perhaps the front page at Snakebite could be updated along these
lines? A "watch this space (but not yet)" message would give a far
more accurate impression of the current situation than the old message
that is there at the moment (the only reason I didn't share Jesse's
assumption that the project was dead before bringing it up on
python-dev recently was that I remembered Trent saying something about
it earlier in the year).
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Trent Nelson
The website will be receiving some TLC from me very soon :-)
Trent Nelson
....and in the mean time, in the spirit of JFDI, I've put a little
update there along the lines you suggested :>
Trent.
Jesus Cea
Hash: SHA1
<http://www.acronymfinder.com/Tender-Loving-Care-%28TLC%29.html>
<http://www.acronymfinder.com/Just-Focus-and-Do-It-%28JFDI%29.html>
<http://www.acronymfinder.com/Just-Freakin%27-Do-It-%28polite-form%29-%28JFDI%29.html>
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
jc...@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:jc...@jabber.org _/_/ _/_/ _/_/_/_/_/
. _/_/ _/_/ _/_/ _/_/ _/_/
"Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
"My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQCVAwUBTOQBw5lgi5GaxT1NAQKi7gP/UB7AqxJ9mJKBK+npIPsJlnMJBkaFYxpC
5Tgr3suwH3W8a7rOU43xOL31uR+KfliXfvFZLG6JVxDeinK3cXCrHbtk74/jJW+O
hCdKh0uiPoea9QjnkEm5HnVFbLzbsCb9sWLhFIwM+fxREKOfZNEsDletRAuSYEP8
pEQ+PSELsmw=
=1hRO
-----END PGP SIGNATURE-----
Jesse Noller
Did I say it was dead? I don't remember that :(
Nick Coghlan
> Did I say it was dead? I don't remember that :(
Whoops, sorry. s/Jesse/Jesus/ (It was late*!)
Regards,
Nick.
*Or early, depending on your point of view...
Nick Coghlan
> ....and in the mean time, in the spirit of JFDI, I've put a little update
> there along the lines you suggested :>
Much better :)