Guys:

From a brief study of the game industry, I might list a few impediments to
"agile" software development. Whereas this essay will define "agile" as
"good", we should not make the mistake to think that any agile "best
practice" must slavishly mirror into game development. Achieving the results
shown by "agile" development in other industries - reduced odds of failure,
accurate schedule estimates, and absurdly low defect rates - will require
adjusting the practices to fit game programming topologies.

    Planning Game, Early Delivery
With a few exceptions, the game industry could not sustain Early Delivery.
Games are entertainment, and changing levels every few months would destroy
the tenuous addictions that designers so carefully foster. The game industry
has more in common with the motion picture industry than the other sectors
of programming. A game starts as a pitch, very similar to a screenplay
pitch, and ends with a rollout. If the game succeeds it funds its studio's
failures. Like a screenplay, a big game design, up front, must pitch a known
genre to a very clearly defined demographic. That helps a studio's successes
exceed its failures. This practice excuses each game's development from much
internal risk management. A successful game generates enough cheddar to fund
rewriting the next game's engine. (A studio simultaneously shipping two
games generates trade literature headlines.)

Frequent, regular internal releases have many more benefits than merely
preparing for frequent delivery. Frequent releases depend on a steady stream
of feature requests, where each feature can cover any subset of database,
middleware, engine, logic, art (assets), and front-end. Sorting these
requests in order of business value has a powerful effect on design, because
the high-priority items get tested, reviewed, integrated, and reinforced the
most during the remaining development. However...

    Continuous Integration, Daily Deployment, Frequent Releases
Game development works with very large teams, split by function into groups
with similar abilities. The business world, working on projects requiring a
narrower range of abilities, has learned to avoid this tactic, and to
colocate very small teams of diverse abilities. Splitting teams by function
requires goal donors (designers and technical leaders) to split feature
requests by function, not by business value. So these forces, coupled with
the incredibly high volume of data born in each game title, powerfully
dis-integrates production, making continuous integration much more difficult
than it could be. A game's source code and data file can take several
computer-hours to rebuild.

So these forces, in turn, inspire game developers to re-invent a practice
that business programmers are learning to avoid - rampant code branching. A
game team might have a mini-branch for the sound guys, a mini-branch for the
AI guys, etc. The cost of re-integrating each teams' new outputs, to finally
create the integrated feature the designer requested, delays the most
important feedback. Designers should review the effect of each request, on
all game aspects, as soon as possible to permit that review to accurately
and usefully drive the new requests.

    Pair Programming, Shared Code Ownership
The game industry is widely reputed to program game engines using only the
worst practices of both Waterfall and Code-and-Fix. Designers request engine
abilities far in advance of reviewing the levels that exploit them.
Engineers perform Big Design Up Front to the best of their abilities. When
any Waterfall system runs off the end of its planned design, developers are
forced to switch to Code-and-Fix. Those of us experienced with the long-term
results of Waterfall have all seen projects with many clever designs, many
over-designed parts, and many parts riddled with spaghetti, cross-patches,
and horrid cruft.

I lack the tact that other agilistas exercise daily when I admit that game
development attracts and retains programmers with ... special skills. Pride
of code ownership must preceed the all-night coding sessions that keep such
code afloat. Game developers learn their systems very well, and the industry
relies on their experience shipping games to request them to ship again,
with a slightly better engine each time. Within these mega-iterations,
source code loses many opportunities for collaboration to force in new
flexibilities. "We can't do X until Q completes Y" is a common impediment.

    Test-Driven Development, Customer Tests
Over time, game projects grow a huge "decompression debt" of easily-fixed
bugs. Shipping a game requires an early feature-freeze and then a very long
party to stomp insects. However, within the many layers that make up a game
are many latent tests, and an extraordinarily huge number of side-effects
that could easily become "characterization tests". But only pure TDD from
scratch can beat legacy test retrofits. Pure TDD will generate full-feature
test fixtures capable of testing arbitrarily complex aspects of play that
entry-level testers could only dream of.

And all games have a scripting layer of some kind. The Lua language is the
leading candidate for this role. And the great irony of game development is
this: Most business projects don't have a scripting layer, and have Onsite
Customers who should not write source code. Agile development relies on
Customer Tests, written by customers, to constrain each feature they
request. Game projects, by contrast, require designers to author Lua scripts
to add the hooks and triggers to each level. So game projects have Onsite
Customers who can write scripting code, and game projects also have the
scripting layer for them to write with. To close this gap, game projects
need only require designers to author the tests for each high-level feature
they request. And game projects must tune their scripting layers to make
these techniques easy and robust.

    A Prescription
Start a game small, not super-big. Maybe start with a designer, two engine
guys, and two asset guys. They finish a level, incrementally adding
challenges, and maintaining a zero-tolerance for bugs. They measure their
velocity, and their gold-owner adds resources until the velocity levels off.
The game remains playable and deliverable at all times. It might not have
all its levels and its resolution cinematics, but it will have zero bugs and
all playability features for the levels it grows.

This recommendation turns traditional development side-ways. The storyboard
becomes the schedule, not the bottleneck, and levels not yet completed are
the ones most easy to change. Shortening the feedback between feature
requests and results, by any means necessary, will lead to agility for the
game industry.

--
  Phlip
  http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

I'd like to point out a couple of areas where your brief study is
inconsistent with my own experience, but first...

You're assuming that these are big problems in game development. In
my experience, other problems dwarf these factors.

Failure:

Few games that pass the green light stage get canceled before
completion. In my experience I have only worked on 1 game in 10 that
got a green light and was later canceled. Getting a green light is
usually quite tough so this weeds out most poor game designs before
they get too far. The most common cause for canceled projects (in my
admittedly limited experience of such things) are for marketing and
budget reasons unconnected with the game's development progress. I
suspect this is outside the scope of things agile methods are likely
to cover. That's not to say that failures due to technical problems
don't occur, but I've not heard of any. As far as I'm aware, all of
the recent studios that closed in the UK did so in spite of
delivering their games, not because they couldn't.

Accurate Schedule Estimates:

Contrary to popular belief, most game development teams are fairly
good at estimating project schedules. The problem is usually that
these accurate estimates aren't used. Instead most publishers
substitute ridiculous estimates based around delivery dates dictated
for (usually very questionable) marketing reasons. For example, in a
project that I worked on, our technical director came up with a spot
on estimate for development time based on years of development
experience. This was then cut in half by the publisher and later
extended month by month until we delivered pretty much bang on our
estimate date - in other words late! This is how most experienced
teams deliver accurate estimates and yet paradoxically miss their
deadlines. Again, it's difficult to imagine where agility would help
here.

Low Defect Rates:

I've played a lot of console games over the years, and in that time
I've only ever found one crash bug in a published game. That was
after playing continuously for many hours and then starting over. On
the other hand, I've lost count of the amount of times I've lost
work and had to reboot my PC while using perfectly standard "mature"
development tools. Now it's true that PC games also suffer from this
instability, but these games are mostly written and tested by the
exactly the same people who write the console games. That makes me
think this is a PC / OS problem rather than defects inherent in PC
game code or the game development process.

A defect in game development is a vague concept. A bug can mean
anything from a wrong shade of green on a texture to an inability to
complete the game. The priority of bugs is usually somewhat
arbitrary: a single incorrect spelling of some on screen text can
render the game unshippable; while extremely annoying consequences
of bizarre design choices are often waived as "unimportant" as the
deadline closes. In fact, many of the "bugs" that ship with a game
are emergent in nature. They don't arise from a mistake in the
coding, but from an unintended consequence of the specification of
rules of the game.

Okay, back to your essay:

I'm not sure where you got this impression. It does happen. (And
probably more than you realise precisely because it does not
generate trade literature headlines when it does?)

True. And so many game development teams are anything but agile by
definition. However, there are still many smaller game developments
that could benefit.

I've never seen that myself. Almost every project I've worked on, or
even heard about used simple source control with everybody working
on the same branch. Although I'm not sure what branching has to do
with agility. Is there some pet peeve you have with this practice?

In my experience, code ownership is imposed on programmers. In that
sense it's more like code responsibility than ownership. Also, in my
experience, programmers frequently collaborate (in secret if
necessary) to get the job done. It's not pride on the part of
programmers, it's a manager's need for a name to blame.

This is usually the policy, but in practice most games are adding
features right up to the burn of the final master. It's particularly
easy to introduce emergent bugs at this point.

But does the game industry need agility? And what for?

In my experience, speeding up the feedback loop hasn't improved much.
If anything, it led to experimentation without purpose. At
best it has had no effect. I once made every predefined gameplay
constant editable from a single text file. The designer took that
version away to "tweak" the design. When he brought it back the
following week he had changed one parameter by 0.01 of a second.
Nobody noticed.

I think game development could benefit from many of the techniques
of agile methods, but I don't think agility in itself offers much.

Phlip,

It worked for Id Software's Quake 3. Perhaps that's one of the
exceptions...

You could always release (early and often) to a small, select, trusted
population of early players.

Laurent

One myth of XP is it requires early and frequent delivery.

You get all the benefits if you release regularily to a small crowd of
users.

One goal for my essay was skipping over some of these myths, before they
caused the usual diversionist threads.

--
  Phlip
  http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

You need to understand Phlip that Game development is by far the most
complicated form of programming there is! Period. Most of the games out
there today are at the cutting-edge and do well compared to the lame crap
business developers write.

Maybe we should be learning something from them? Also, are you coming from
the philosophy that XP and agile development is the "right" way to program?
If so, I think game developers show it is at least not the only way.

Stede

news:mu3md.21811$C74.2854@newssvr31.news.prodigy.com...

Stede Troisi <st...@verizon.net> loquated like no one had ever loquated
before with:

Sorry, but there's plenty of non-game development software that is just as
complex.  Maybe you meant to say 'pressure driven and competitive' instead
of 'by far the most complicated form of programming there is.'  It is
certainly equal to other development tasks, but it is the
time/money/resource constraints that make development harder than most other
forms of development, not the complexity of the code itself.

    WTH

I listed the three major reasons why "agile" development has been reputed to
be "good". Having never seen a game project fail, up close (for various
definitions of "fail"), I will personally rely on "good". Maybe "avoiding
all-nighters" should be the primary goal.

If I had seen such a project fail, I suppose I could find a way to recast
each failure point back onto my list of agile results just as easily as I
could recast them away from them. It's all "good".

What top three problems would you list? Be brutal - many shops grow
accustomed to some situations, just as ones eyes grow accustomed to the
dark.

My untenable premise is a shop builds their engine, builds a game, and
starts a re-write on the engine. I'm not saying that's rampant (or
especially bad). I suspect that truly flexible and bug-free code would allow
such an engine to obey the "Software Product Lines" style.

I don't understand. Are you saying the teams _must_ split, and that therefor
process improvement must be ... "less" agile?

Why yes. I have seen code forking churn non-game projects into complete mud.
Proper respects for observing unbranched game projects, and I should not
behave as if I had determined it were the industry average.

Ah, I forgot to blame the managers, too. Point.

Per the "crunch mode" thread, it seems I tried to describe "crunch mode"
here. Feature freezes are not bad (it's just a game, not air traffic
control).

But if scheduling were working, were are the "crunch modes" coming from? My
non-game crunch modes came from not maintaining a zero-tolerance policy
against even minor bugs, early in development.

Let's assume they currently have "bad", and need "good". Neither of us have
disagreed, so far, on those fundamental issues. However,

That's not feedback. You didn't get early feedback on whether your tool was
needed.

The recommendations, again:

 - start small, colocate a team, and pair
 - mix the team; design, art, and math
 - seamless integration, no code forking
 - leverage the script layer for design-tests
 - one level at a time, one trigger at a time
 - TDD, refactoring, shared code, daily deploment, etc.
 - absolute zero tolerances for anything remotely resembling a bug

Within that cycle, I suspect that a design could tweak a parameter via
several different scenarios, including user stories, story tests, pairing
with a developer, tweaking the scripts, etc.

If you complain about any one item in the list, then you play CTips's game
of describing some practice going out of balance, while ignoring the
opposing forces from the other practices.

Maybe I didn't say, "we should not make the mistake to think that any agile
'best
practice' must slavishly mirror into game development," in the first
paragraph.

--
  Phlip
  http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

You have a good point but I still feel game development is in a field
of its
own. First, almost anyone can write a payroll system (think C3) but
inventing cutting-edge game graphics and sounds that have never been
done
before is tough. Add all the other factors that go into game design and
I
think it would be almost impossible to do XP at companies like EA,
Activision and Blizzard.

Then again, your math is probably better than mine and it probably is
easier
for you. It would be interesting to see a AAA title done with XP
though.
Working 90 hour weeks minimum and enough teenagers to make 1 game a day
for
the next 500 years I don't think it will happen any time soon.

You really *are* the offensive twit you appeared to be on the xp mailing
list. At least *consistency* is some kind of virtue.

--
Chris "electric hedgehog" Dollin

news:Rvgmd.20851$Rf1.20444@newssvr19.news.prodigy.com...

[Paul's message doesn't appear to have made it to my server...]

I'm not sure where you're going with this one, but several examples
of engines that underlie multiple games come to  mind. LucasArts'
SCUMM engine, for example. The Z-machine that not only ran
all the early Zork games, but was reimplemented, ported to just
about every platform in existance and is ubiquitous in the
interactive fiction camp. There are numerous other examples.
I suspect that the norm is to use an existing engine to crank out
mid-level games quickly. It's only the real high end games where
the intention is to push the state of the art where you really want
to do wholesale revisions to existing engines.

I don't necessarilly think of this as bad. While I've heard of
XP teams that are larger than about a dozen people, the
"conventional wisdom" seems to put an upper limit on
an effective team that is somewhere around that point.
Beyond that, you need a strategy to coordinate multiple
teams, or operate in some other fashion.

IMO, crunch mode usually comes from not having a production
quality deployable at frequent intervals throughout the development
process.

For me, the question simply comes down to:

"Would you rather be able to ship a bug-free product that's missing
a couple of levels that you had on the storyboards in time for Christmas,
or would you rather ship the complete game your designers had in mind
and miss the Christmas shopping window?"

This is simply one of the basic XP questions, with  the blanks
filled in from the game development lexicon.

However, the question only makes sense if the process is
capable of producing that production quality deployable
every couple of weeks, with a new level or two, fixes
for previous levels from the play testers, game engine
enhancements and so forth.

The basic issue is: which feedback loop needs tweaking? The
only way you tell that is to find the bottlenecks and work on
them. In this example, it sounds like gameplay engine change
requests weren't a bottleneck.

On the other hand, they may well have been. I can very
easily envision a scenario where the engine people were
being burried under a load of change requests for constants
that didn't actually do anything. Externalizing that would
have removed the change request queue.

John Roth

Heh, I love this attitude.  This is exactly why game development is
mired in the shit that it is right now.  "Game development isn't
rocket science, it's /way/ harder than rocket science."  LOL.  Fuck
yeah!

Actually, what we can learn is how to make software better.  The state
of the software in games is miserable, in general.  Game studios
regularly ignore the masters; Brooks told us that adding more people
to a late project is death, yet we still do it.  We all know that
developers' efficiency goes down the longer they go without rest, yet
we (as an industry) still "mandate" seven-day work weeks.

Game development is NOT rocket science.  Game development is easy.
That's the only reason why we're able to do it with the hackery that
we did.  If game development were actually hard, we wouldn't be able
to hack through it like we do, and either a sustainable process would
have come to us earlier, or we wouldn't be making games.

-tom!

Tom, Easy? Can you at least say unique or specialized? I mean game
programmers have to know Physics and a heavy, heavy dose of math. That
doesn't even touch on the optimization wizardry they produce?

Maybe programming is hard when the field you are working in is hard.
Maybe a complex stock market simulator would also be hard. It just
seems to me that game programming requires a much deeper understanding
of the machine then a client/server application.

How when you cannot see that game development and traditional business
development is different?

produced?

Stede

Paul Sinnett responding to Phlip:

November 2004 issue of Game Developer.

-tom!

Show me a game that has done some sort of programming "cutting-edge"
graphics work that has never been done before.  Maybe something from
id or Epic, but other than that, everyone's reinventing the wheel but
it's not exactly an unsolved problem anymore.  (Please note that I
don't consider reinventing the wheel to be doing something that has
never been done before.  Despite the fact that NotInventedHere is
rampant in the game industry, we're still all building on the work of
those who have come before us.)

What are some examples of cutting-edge game sounds that have never
been done before?  How does the implementation of these sounds
illustrate technology that has never been done?

He's talking programming, not game design.  BTW, Blizzard says they're
doing XP.  Do we not believe them?

The company that I work for will be starting down this path very soon.
 We've already moved toward many agile processes, and each switch we
flip makes development go that much faster.

???

Anyway, it's not hard to think of systems that have much more
stringent requirements than game development.  Consider the back-ends
for credit card companies.  Despite perhaps billions of dollars in
transactions a day, I have never bought something on my credit card
and not ended up getting the bill for it.  I can go to my credit
card's website and see my transaction history without waiting for more
than a couple of seconds.  Consider the software behind any
spacecraft; sure, the tech isn't super-advanced, but the requirements
of not killing people make the software's functionality that much more
important.

When WikiWords attack! :-)

*Hah*. That said, I'd love to see better reference material available on
the sort of stuff that goes into eg the average Lexicon or TC effects
unit. And one of these days I'll learn enough DSP to write a half-decent
softsynth, honest...

Quite. Games might be a step above typical business code, but so're 101
other things.

--
fli...@flippac.org

Hi Tom,

You win this argument. I must agree. Blizard doing XP makes sense. Their
games are always better designed and contain less bugs then most of their
competitors.

Can I assume if Blizzard is doing XP they can't have programming sweatshops
like EA?

Would anything in your opinion need to be changed/amended to XP to make it
easier to fit into the game development lifecycle?

Stede

news:7d209056.0411161335.6442bb2f@posting.google.com...

In article <tjwmd.11685$tI3.5087@trndny01>,

   Reading some of the recent postings, it seems that the sweatshop
mentality at EA comes from a management attitude of "you will work
long hours." [I can't comment on the validity of those assumptions,
just presenting my limited understanding of them.] The number of hours
worked seems to be irrelevant to XP-- it can be combined with few or
many hours. A manager of a company using XP could very easily turn it
into a sweatshop. Or, the employees could do it to themselves-- giving
dirty looks to people who leave early.

   Don't use hours worked as a plus/minus for XP. It's tangential, and
irrelevant. Silver bullets are usually tarnished.

Nathan Mates
--
<*> Nathan Mates - personal webpage http://www.visi.com/~nathan/  
# Programmer at Pandemic Studios -- http://www.pandemicstudios.com/
# NOT speaking for Pandemic Studios. "Care not what the neighbors
# think. What are the facts, and to how many decimal places?" -R.A. Heinlein

Did you really work on Star Wars: Battlefront? If so, it is an amazing game.
I can't stop playing it! I wish it had its own level editor though and
worked with a joystick.

Stede

news:419aa140$0$218$a1866201@visi.com...

Heinlein

Suppose, instead of rescuing those meek programmers from their tyrant
bosses, or proving XP is good for everything, we instead set a goal of
"rapidly produce awesome games, and kick our competition's butt."

How would you (plural) target that goal?

--
  Phlip
  http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

news:419aa140$0$218$a1866201@visi.com...

I wouldn't be so sure about that. There's quite a bit of evidence
that you can't work more than about six hours doing straight-out
TDD and pairing. If you try to, you start making so many many
mistakes from mental fatigue that it isn't worth it.

In other words, trying to get more than six hours a day out of
your developers with XP is going to send your velocity plumeting
toward zero.

The other two hours, of course, are e-mail, meetings and other
stuff that has to be done but isn't development work.

John Roth

In article <10pld9ef5j0j...@news.supernews.com>,

   There's quite a bit of evidence that there's a limit without
XP. Usually a bit higher (and depends on the person), but still under
the number of hours worked during crunch mode on games. Once you work
past a certain number of hours, productivity goes negative as you
have to spend time cleaning up after the mistakes of yesterday.

   You can argue this point till you're blue in the face. But, when
management doesn't care, then XP, non-XP, whatever you want to call
your development process is irrelevant. XP is *NOT* a silver bullet
for working sane hours. It's tangential to the real issue.

Nathan Mates
--
<*> Nathan Mates - personal webpage http://www.visi.com/~nathan/  
# Programmer at Pandemic Studios -- http://www.pandemicstudios.com/
# NOT speaking for Pandemic Studios. "Care not what the neighbors
# think. What are the facts, and to how many decimal places?" -R.A. Heinlein

news:419abbb1$0$236$a1866201@visi.com...

In that sense, you're quite right. If you've got that issue, there
are a number of formal studies (beginning with some at the
US DoD) that demonstrate that error rates go sky high as
fatigue sets in.

The medical community doesn't seem to have learned this either,
from how they schedule interns and residents.

John Roth

Not working long hours is the silver bullet for working long hours.

Studies (by game shops) have shown that if you actually track velocity, it
does not go up as the hours get longer.

I suggested "design code and finish one level at a time". I'm not sure, but
I suspect games grow breadth-first.

Just like working sane hours, there are people who will reject such radical
ideas simply because they have never been done.

Look to "lean hardware design", aka lean manufacturing. You delay decisions,
and empower the workers who collected the raw data about things.

--
  Phlip
  http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

Phlip <phlip_...@yahoo.com> loquated like no one had ever loquated before
with:

You're missing Nathan's point Philip.  If XP was a magic bullet, and you
gave it to EA, they'd just double their game output and continue to work
people into the ground.  Hours are 'tangential' to software methodology in
EA's case (copyright NM 2004)

    WTH

Good management, in general, might help EA-style projects.

Pure XP would help too - the practice called variously "40 hour weeks",
"sustainable pace", and "energetic work" forbids chronic overtime.

Whether or not XP is a magic bullet, if EA did XP with chronic overtime then
it wouldn't be XP.

However, some folks see "agile" or "XP", don't read any farther, and then
assume someone thinks they have a magic bullet.

I don't know about EA's management, but managers who can be helped are
advised to read /Slack/ by Tom DeMarco.

--
  Phlip
  http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces