Soliciting process suggestions
pgwilson5
suggestions on process options to move the BigBlueButton product to
Enterprise grade quality.
There is currently an activity underway to enhance the product quality
through minimal process additions. The following areas are being
considered. Please provide suggestions and input. All input
would be appreciated. It is expected this will move quickly for the
ongoing product additions and existing support mechanisms.
- BigBlueButton Feature/Capability additions
- BigBlueButton Patches to correct defects
- Normal updates
- Critical updates
- Continuous Improvement
PW
Fred Dixon
In looking at improving our process to move towards an “enterprise
grade” product, it’s helpful to understand the our current process as
a starting point.
There have been nine releases of BigBlueButton to date; Blindside
Networks has coordinated the activities of each release. Our process
has much evolved from the early days, and you could say the current
process reflects the skills of the committers, who include:
Jeremy Thomerson: API
Richard Alam: Client and Server
Tiago Jacobs: Server (red5)
Denis Zgonjanin: Client
Fred Dixon: Packaging
DJP, Xie, Chen: Localization
Here is the process we went through for our last major release: 0.71
(I’ll give some comments at the end how it could be further improved):
1. Planning and Design
We start each release with a planning meeting among the committers
(using BigBlueButton of course!). All committers are invited. During
the meeting, Blindside Networks gives a presentation that covers the
proposed features for the upcoming iteration. We select the features
using the following process:
http://code.google.com/p/bigbluebutton/wiki/FAQ#When_will_feature_X_be_implemented?
During the discussion, we cover
- impact of features
- major design trade-offs
- assignment of responsibility for implementation
We also look back at the previous release and talk about what
worked/what did not. We cover points that would be *very* familiar
with any software developer (more code inspection, testing, better
documentation, etc.).
After the meeting ends, we post to the bigbluebutton-dev mailing list
an edited copy of the presentation (removing those features that were
not going to make the release). For example, see:
http://groups.google.com/group/bigbluebutton-dev/browse_thread/thread/6ffa48133ea7d2e6#
At this point, we know what to do, but not always how to do it. Most
developers start working on a design and prototypes. There are still
lots of unknowns; some take weeks to figure out. Sometimes we post
preliminary designs to the mailing list (Jeremy was good at this for
API updates) to solicit feedback.
As none of us are user interface designers, we try to follow the
mantra of “keep it simple” in our UI design. If we can design a new
feature and at same time further simplify the UI for BigBlueButton, we
know we’re headed in the right direction.
2. Development
At this point, the developers go “heads down” and start working on
implementation.
During this phase, we incorporate and review contributions from
others. For example:
- Neeraj Kumar extended the chat module to use Google Translate for
real-time translation. Denis mentored Neeraj and reviewed (and
rewrote parts of) his code for inclusion in the release.
- Leif Jackson added support for FreeSWITCH as a voice conference
server. When Leif was ready, he sent Richard a pull request from his
Git repository and Richard incorporated it into his local Git
repository.
We try to do some refactoring of the code as well. For example, Denis
managed to remove the last parts of PureMVC dependencies in the client
in 0.71. None of the refactoring is visible in the end product, but
we know it’s vital to keeping the architecture flexible and
maintainable.
3. Internal Testing
At this stage, developers are responsible for testing their own code.
However, since Richard, Denis, and I all work together, Richard would
often say “Could you go to 32”, which is code word for 192.168.0.32.
We all open our browser and help test a build on his local development
environment.
Much of this testing is not visible to the community, nor are we
posting any status reports yet.
4. Packaging
The internal development and testing is far enough along to start
pushing code to our external Hudson server at:
http://ubuntu.bigbluebutton.org:8080/
Hudson does a lot for us: it automatically pulls from GitHub the
latest code, rebuilds from source, runs the packaging scripts, and
updates our devbuild server:
http://devbuild.bigbluebutton.org/.
Before any code is pushed to GitHub occurs, we review the code for any
updates that are required to packaging.
The inclusion of FreeSWITCH, for example, meant that we needed a
dependency for FreeSWITCH in our packaging (hence the creation of
bbb-freeswitch-config). We update the package scripts to account for
new configuration files, directories, permissions, properties, etc.
A lot of configuration issues get resolved at this point. The goal is
to get that first complete install of everything on an clean Ubuntu
10.04 32-bit (or 64-bit) install. Once that occurs, the package
scripts are updated on Hudson and the code is pushed.
At this point we check:
1. Hudson has successfully built the code (for example, did the war
file for bbb-web successfully compile?)
2. The package scripts have successfully built the associate package
(did the bbb-web package build?)
3. The Hudson server has successfully updated the devbuild (did the
upgrade scripts work?)
We fix any errors at this point with the above process until
everything is updating correctly on devbuild.
Testing has now moved away from the individual developer to a separate
build servers -- both devbuild and internal build servers.
Development is occurring in parallel, with lots of feedback back and
forth.
Once the internal builds are working well, and the product is looking
stable enough for others, we’re now ready for external testing.
5. External Testing
At this point, we go through external testing in four phases, which
are covered in detail
http://code.google.com/p/bigbluebutton/wiki/071Overview
A. Install 0.71-beta on devtest.bigbluebutton.org for limited testing
The devtest server is a snapshot of devbuild. (We don’t use devbuild
yet because there are still lots code updates being pushed and the
updates may break devbuild, which isn't helpful for external testing).
We post to bigbluebutton-dev asking for users to try out devtest. We
watch it closely for any code that breaks breaks (processes stop,
exceptions thrown in the log files, etc.) and if anyone reports
problems with the client (freezes, hangs, etc.)
The initial traffic is usually high, but it quickly tapers off, which
is OK as we’re now fixing any immediate bugs that were discovered by
external testing.
B. Install 0.71-beta on demo.bigbluebutton.org for broader testing
The build is now stable enough for more public testing, and we update
demo.bigbluebutton.org. This server gets a *lot* of traffic, and any
obvious errors (and sometimes not so obvious errors) are quickly
spotted. Again, we’re watching the log files anything that suggest
the server code isn’t working correctly, and we frequently login to
the server and interact with visitors to get informal feedback.
C. Release the 0.71-beta Ubuntu packages for local installation and testing
This is a big step. At this point, we’re now ready for testing on
third-party servers. The product is given the "beta" label.
By this time the development is starting to settle down. The
packaging has stabilized - we’ve installed the packaging many
timesinternally, on devtest, and devbuild -- and the product is stable
enough for extended use (as determined by our watching demo). There
are four scenarios for packaging that we are looking to externally
test:
On Ubuntu 10.04 32-bit:
Clean install of 0.71
Upgrade from 0.70 -> 0.71
On Ubuntu 10.04 64-bit:
Clean install of 0.71
Upgrade from 0.70 -> 0.71
The goal of the packaging is to be a non-event: it just works. We
want developers and administrators to focus on testing the new
features of BigBlueButton. We post URLs for the packaging, and again
we watch the forums for feedback.
D. Release a 0.71-beta VM for download and testing
Since the VM also doubles as the development environment, this step is
actually much bigger than it sounds.
We first make sure that all developer tools on the VM are updated (if
needed). Next, we make sure the VM is able to build the latest
version of BigBlueButton client, web apps for tomcat6, and server apps
for red5. Richard and Denis usually switch over to using the new VM
for their internal development.
Since anyone can run the VM with VMWare Player, we get a broader class
of users testing now. After the release, we look for problems with
- installation of the VM
- updating the VM to the latest builds
- using the VM as a development environment
- using BigBlueButton itself
Historically, it usually takes us about four weeks of external testing
to get to this point. We’re now getting to the smaller stuff:
- checking the language files for missing strings (which we always
miss a few :-)
- making sure the versions numbers are all correct
- updating the documentation (install instructions, configuration
of the client, branding, etc.)
- updating the issue tracker, making sure everything that is fixed
is marked as such.
If there are any major problems reported, we’ll fix them, and push our
changes to GitHub. We’re now posting updated builds almost daily to
the forum, asking for people to test on their server (sudo apt-get
update/sudo apt-get upgrade). demo is being updated each day to get
the broadest possible testing.
Once the problem reports slow to a trickle, we’re just finishing up
the loose ends.
The decision to release is made on quality.
6. Release and Support
To actually put out the release, we tag Git, make one last change to
bbb-conf (so it checks out the tagged release), do all the package
installs one last time to make sure we didn’t miss anything, then,
move over the dev repository to the release repository.
The release notice goes out; everyone can now update form 0.70 to 0.71.
Reflection:
You’ll notice there are actually two processes at work: the
development processes and release processes. They are related, but
yet distinct: packaging could work perfectly, but if there are bugs in
the code, the perfectly installed product will not work.
It’s those bugs that we want to find and fix. Here are some areas for
improvement:
1) Code Reviews
We do them informally, but it’s spotty. We’ve relied a lot on the
skill of Denis, Jeremy, Richard, and others to write solid code. But
we’re only human, and doing more code reviews will yield a better
product.
2) Code Documentation
Nuff said.
3) Unit tests
We refer to this as the classic “back up problem.” Before we can have
unit tests, we have to back up and have a framework for unit tests.
Before have a framework, we have to back up and decided how we are
going to run this framework. Before we can run the framework, we have
to setup the tools and processes to support it.
There some some unit tests in BigBlueButton (such as for presentation
conversion), but this too is spotty and run manually. We need a
system to automate the running of unit tests.
4) Stress testing
This is the big one. BigBlueButton is a server, and as such there are
very high demands on its stability (this is what Paul refers to as
"enterprise grade").
A lot of our stress testing is still manual. We've developed some
tools, but we need a Hudson-like process to automatically test each
build of the BigBlueButton server under heavy load, with test scripts
that simulate users, voice, video, etc. in various scenarios.
These automated tests need to run constantly and publish reports (like
Hudson does) on anything that fails so we fail, find, fix, and test as
quickly as possible. Any errors in the log files should be included
in the reports.
Also, as we have more automated testing, we can get closer to a more
agile model of development.
5) Public Specifications
Any design could be improved with more eyes, and we’ve got 750 members
in our development mailing list.
Publishing a spec as part of the design phase should save us effort on
the back end.
6) VoIP Refactoring
This is another big part; high-quality voice is critical to
BigBlueButton’s success. We've worked for months on refactoring and
optimizing the handling of VoIP packages, but there are others who
know more about VoIP than we do, and we look forward to working with
them.
7) Best Practices
We can look at other successful open source projects and incorporate
some of their best practices into BigBlueButton.
We can also better use the tools we have at hand. We can further
exploit Git to make it even easier for others to develop and test in
their own branch. We could use the voting of issues in Google Code to
get more feedback on those issue and enhancements are highest
priority.
Regards,... Fred