RenderMan on the Frontier
Sean Cier
project that I've been working on lately that takes rendering
in a few new directions. The renderer is called Vegas, is
partially RenderMan compliant (that is, it renders RIBs, but
includes only a subset of the features of a fully
RenderMan-compliant renderer). It's based on Monte Carlo
global illumination algorithms -- specifically, Bidirectional
Path Tracing ala Eric Veach -- to achieve beautiful, realistic
light transport. It's entirely Java-based. Most significant,
though, is the platform it runs on -- a new Internet-based
distributed system called Frontier, developed by
Parabon Computation (yes, I work for Parabon).
For the click-happy: http://www.Parabon.com/vegas.jsp
For a bit of background, Frontier uses the idle time of
Internet-connected desktop computers to form a huge,
general-purpose virtual supercomputer. If you've heard of
SETI@Home, this system takes the concept to the next level, as
it's commercially available, runs *any* task (you could develop
code and submit jobs from your laptop), and will soon pay
people to run the screensaver-like 'compute engine', Pioneer,
on their desktop machines. For more info on Frontier, see
http://www.parabon.com.
So, back to Vegas. Before I get any farther, let me make it
clear that Vegas isn't currently a commercial product, so
don't go looking for where you can download it; I'm just
describing a cool project here, not giving a sales presentation.
The concept behind the renderer is that as Frontier can bring to
bear a lot of power on-demand, cheaply, it suddenly makes more
realistic rendering algorithms computationally feasable. At the
same time, it's one of very few RenderMan-based renderers that
employ Monte Carlo GI. There are some simple example images at
http://www.Parabon.com/vegas.jsp (note that the page is primarily
intended for the layperson).
Now, there are obviously caveats with rendering on a system such
as this. For example, scene size becomes a consideration. Given
the advent of broadband and large amounts of RAM on desktop
machines, though, this isn't a barrier for many applications; and
further, the ability to use more advanced algorithms means that
less fakery and unneccessary complexity is required to achieve
results which still look better. Another consideration is
intellectual property; some studios would never want to send their
scene files out on the internet, no matter what the payback. But
the nature of the system, which breaks jobs up into millions of
small tasks and sends each to a different machine, helps a lot
with this. Beyond the layers of security, strong encryption, and
obfuscation that Frontier uses, though, the fact is that even a
highly skilled hacker would end up with, at best, one piece of a
very large puzzle (which some would even consider nothing worse
than free advertisement). On the other hand, the many advantages
to rendering this way should be obvious for both large studios and
individual artists working in their free time out of their
basements -- you can dial up as little or as much power you need
at any given time, and pay only for that much power; no need to
buy a huge render farm that'll be idle most of the time and will
start to depreciate before it's even set up.
But I'm getting carried away; my goal here isn't to sell y'all on
anything (heck, I'm not even in marketing), but to tell you a bit
about what I personally find to be a really cool application of
some new technology, even if I am biased. If any of this provokes
discussion, all the better. I'll be happy to answer any questions
anybody has (I can do better with the engineering side than
business, though), either here or in private email.
Disclaimer: I do not speak for Parabon; all information in this
and subsequent messages is merely my fallable opinion.
-spc
--
/- Sean Cier <sc...@PostHorizon.com> -\
( If only you could see what I've seen with your eyes )
\- http://www.PostHorizon.com/scier -/
Skint
Sounds interesting - do you have any idea how much it will cost to
put a render out on this system? And can you run it standalone on
a non-networked PC? Not so sure of professionals, but students etc who
can't normally afford a renderfarm will definitely be interested in
this.
Simon
On Mon, 06 Nov 2000 01:32:09 -0500, Sean Cier <sc...@PostHorizon.com>
wrote:
http://www.rendermania.com/
UIN 11123737
al...@r-m-c.removethis.ru
And many people were negative about the prospects of a similar
project. And the reason they noted was the significant differences
between, for example, seti@home and 3d rendering. Amount of input
data for a real frame of 3d animation can be measured in gigabytes.
So how do you plan to resolve this issue?
Sean Cier
Four points. Take each with a grain of salt. Perhaps no single one of
these factors alone will eliminate the issue, and each involves some
amount of ifs and guesswork, but just a few combined could make the
technology feasable for some applications -- maybe a few, maybe most.
First, not all scenes are gigabytes in size -- this depends on the
studio and the scene itself. Maybe every frame of Pixar's "Finding
Nemo" won't fit in an average desktop box's available RAM, but maybe
Square USA's next film will; or maybe Independent Filmmaker Joe Smith's
film will fit perfectly.
Second, often much of the complexity is really pretty extraneous and
exists simply to make the scene 'look good' to the eye (something that's
admittedly tough to quantify), and the detail might become unncessary
when higher-quality rendering is made available -- or at least, the
final quality lost by decreased scene detail could be surpassed by that
offered by more computationally intense algorithms.
Third, there's a lot of technology that can be used to reduce the size
of scenes -- geometry compression, texture compression, higher-order
procedural surfaces and shading, high-level articulated models, even use
of subdivs rather than NURBS; much of this technology isn't used simply
because it isn't neccessary in traditional renderfarms (e.g., why bother
to use jpgs for textures when network bandwidth is nearly free?).
Fourth, The capabilities of the network and computational nodes are
increasing; broadband is fast becoming commonplace, disk storage is
increasing by leaps and bounds, and system RAM is exhibiting expected
geometric rates of increase.
-spc
--
/- Sean Cier <sc...@PostHorizon.com> -\
( Dreams of falling, dreams of flying; )
( a man who never dreams goes slowly mad )
\- http://www.PostHorizon.com/scier -/
Sean Cier
>
> Sounds interesting - do you have any idea how much it will cost to
> put a render out on this system?
I don't have numbers, sorry (in part becuase they have yet to be worked
out, and in part because I don't pay attention to whatever we have
worked out sales-wise). I can say that it'll be somewhere between
'cheaper than buying/maintaining a render farm' and 'dirt cheap'.
> And can you run it standalone on
> a non-networked PC?
Ya; Vegas itself runs standalone just fine (albeit more slowly ;-)
> Not so sure of professionals, but students etc who
> can't normally afford a renderfarm will definitely be interested in
> this.
Indeed... that was part of the idea; maybe we can offer professional
studios a cheaper way to do the same thing, maybe we can offer them a
way to do *new* stuff, but also -- and perhaps most exciting -- we can
offer the possibility of rendering on a powerful platform to people who
could never have considered laying out the capital to put their own farm
together.
-spc
--
/- Sean Cier <sc...@PostHorizon.com> -\
( Dreams of falling, dreams of flying; )
( a man who never dreams goes slowly mad )
\- http://www.PostHorizon.com/scier -/
Larry Gritz
Sean Cier <sc...@PostHorizon.com> wrote:
> First, not all scenes are gigabytes in size -- this depends on the
>studio and the scene itself. Maybe every frame of Pixar's "Finding
>Nemo" won't fit in an average desktop box's available RAM, but maybe
>Square USA's next film will; or maybe Independent Filmmaker Joe Smith's
>film will fit perfectly.
Dunno about Square's *next* film, but rumor is that the current one
is at least as complex as a typical Pixar production.
I think that professionals working on large films will never go for
this, for reasons that I outlined in great detail in a post a few
weeks back. You're smoking crack if you think a big studio will
ever be able (for technical, practical, and political reasons) to
use such a service. "Use Seti@Home for rendering a film" is the
subject of many jokes in the industry.
So, if you're going to make a go at it, you have to cater to people
with fairly simple scenes, who can't afford their own equipment (*).
The main "Catch-22" here is that the intersection of people with
sufficiently simple scenes, yet enough money to afford any service at
all, is probably very small. Such people generally don't have a lot
of money, so the service would have to be exceptionally cheap.
(*) See my previous post for an explanation of why a real studio
would prefer a smaller, in house, tightly controlled farm to a
much larger one that they don't have control over.
Personally, I think that there is likely a living to be made at widely
distributed computing, but it's just not in rendering for film.
> Second, often much of the complexity is really pretty extraneous and
>exists simply to make the scene 'look good' to the eye (something that's
>admittedly tough to quantify), and the detail might become unncessary
>when higher-quality rendering is made available -- or at least, the
>final quality lost by decreased scene detail could be surpassed by that
>offered by more computationally intense algorithms.
I can't imagine what definition of "higher-quality rendering" could
possibly make this true. If there's any lesson to be learned from CG
production, it's that higher quality algorithms and larger
computational resources tend to ruthlessly drive scenes to
exponentially greater geometric complexity.
> Third, there's a lot of technology that can be used to reduce the size
>of scenes -- geometry compression, texture compression, higher-order
>procedural surfaces and shading, high-level articulated models, even use
>of subdivs rather than NURBS; much of this technology isn't used simply
>because it isn't neccessary in traditional renderfarms
Huh? When we cite >GB scene sizes, that *is* using high-order
surfaces, subdivs, compression of absolutely everything, procedural
geometry, procedural shading, etc. And it doesn't count the size of
textures at all.
> (e.g., why bother
>to use jpgs for textures when network bandwidth is nearly free?).
We *are* smart enough to compress textures (though not with JPEG,
which is lossy, has horrible compression artifacts, and only supports
8 bits [I think]).
Network bandwidth is *not* nearly free. Network bandwidth (and disk
bandwidth) are among the most important bottlenecks in real studios
working on big projects. The big studios invest considerable
resources into streamlining these to avoid having an hour of every
frame just for shipping texture tiles over the network. It's a
struggle, even with high-speed lines all in the same building. That's
why the idea of trying to ship the data off-site is so ludicrous.
> Fourth, The capabilities of the network and computational nodes are
>increasing; broadband is fast becoming commonplace, disk storage is
>increasing by leaps and bounds, and system RAM is exhibiting expected
>geometric rates of increase.
I'd love to see the back-of-the-envelope calculations for how much
bandwidth a real scene would take, how many machines "out there" would
have the resources to handle it. If the argument is that you're not
talking about studio films or other large projects, but rather small
hobby projects, then work backwards from the available bandwidth and
resources to deduce exactly what kind of scene might be practical.
-- lg
--
Larry Gritz Exluna
l...@exluna.com Berkeley, CA
Stephen H. Westin
<snip>
> Network bandwidth is *not* nearly free. Network bandwidth (and disk
> bandwidth) are among the most important bottlenecks in real studios
> working on big projects. The big studios invest considerable
> resources into streamlining these to avoid having an hour of every
> frame just for shipping texture tiles over the network. It's a
> struggle, even with high-speed lines all in the same building. That's
> why the idea of trying to ship the data off-site is so ludicrous.
I certainly agree with that. When I had such a renderfarm, the frames
were collected on one server, connected via 10BaseT. We were
negotiating for 100BaseT for that machine, fully aware that it would
still be the bottleneck. When we started making estimates, the Ford
network people started looking scared. Our geometry was ridiculously
simple compared to the typical entertainment rendering (~100K polys),
and we could reuse it for every frame. Texture demands were small,
too, but starting the farm (downloading texture and geometry)
saturated the network for tens of minutes before we would see the
first frame.
The problem with postulating a solution in ever-increasing bandwidth
is that everything else grows, too: the modelers will have bigger,
faster machines to generate models that will still saturate the
network.
<snip>
--
-Stephen H. Westin
Any information or opinions in this message are mine: they do not
represent the position of Cornell University or any of its sponsors.
3dwa...@my-deja.com
>Dreams of falling, dreams of flying;
The dawn of science, the age of reason.
This is the voyage of the mind's eye.
-3DWB
Sent via Deja.com http://www.deja.com/
Before you buy.
Sean Cier
Larry Gritz wrote:
>
> I think that professionals working on large films will never go for
> this, for reasons that I outlined in great detail in a post a few
> weeks back. You're smoking crack if you think a big studio will
> ever be able (for technical, practical, and political reasons) to
> use such a service.
I'm not betting my livelihood on it, no -- and yes, I think that it
would take a great deal to make this technology feasable for a Pixar or
a Square to use as their primary render farm for all-CG feature films.
But on the other hand, anybody who thinks that the CG industry in 15
years (much less 'ever') will look *anything* like it does today, and
hence that we can make reliable predictions about it, probably has some
psychotropic substances in his system as well.
> "Use Seti@Home for rendering a film" is the
> subject of many jokes in the industry.
Yeah, but wouldn't it be cool anyhow? ;-)
> So, if you're going to make a go at it, you have to cater to people
> with fairly simple scenes, who can't afford their own equipment
That's the other end of the spectrum -- though "can't afford their own
equipment" isn't quite the same as "isn't big enough that buying and
supporting a 1500+ processor renderfarm is chump change". There *is* a
middle ground, and a *lot* of niches (for instance, good light transport
in relatively simple scenes is important for some architectural
conceptualization). Given IP issues, scene size, importance of
high-quality rendering algorithms, value of decreased manpower (e.g.
less TD effort) vs. decreased physical control of renderers, 'spikiness'
of rendering demand, etc, we have a potentially really complex map of
rendering needs. I would be insane to argue that this technology works
for everybody, but the converse is also true. I'll be interested to see
just what parts of that map this tool ends up being valuable to.
> Personally, I think that there is likely a living to be made at widely
> distributed computing, but it's just not in rendering for film.
Let me be clear -- Parabon's business does *not* rest on the CG
industry. In fact, rendering was originally tried basically as a
proof-of-concept, and the fact that we think we had some mild success
doesn't mean that we expect everybody to come flocking with their
rendering needs (some have, though). We're far more concerned about
bioinformatics/genetics, pharmaceutical research, financial modelling,
etc, as these industries tend to have problems that map a bit better
(factors such as what we refer to as the compute-to-data ratio, for
instance, which is basically what we're discussing here). Still, I
think it's a really cool application, and even if it turns out not to be
a tool that the entire industry drops what they're doing and adopts (I'm
not even saying it should do so -- the tradeoffs might just not be worth
it for many or even most applications), it's still not worth dismissing
out-of-hand. Remember, even in the entertainment industry, technology
is what drives the limits of possible creativity -- new technologies
with new benefits and hence the *possibility* of permitting better
storytelling should generally take priority over unrelated concerns.
And yes, it's a business too -- but in the long run, business
considerations must be driven by audience demand. Okay, I'm getting
really philisophical and vague here, sorry.
> > Second, often much of the complexity is really pretty extraneous and
> >exists simply to make the scene 'look good' to the eye (something that's
> >admittedly tough to quantify), and the detail might become unncessary
> >when higher-quality rendering is made available -- or at least, the
> >final quality lost by decreased scene detail could be surpassed by that
> >offered by more computationally intense algorithms.
>
> I can't imagine what definition of "higher-quality rendering" could
> possibly make this true. If there's any lesson to be learned from CG
> production, it's that higher quality algorithms and larger
> computational resources tend to ruthlessly drive scenes to
> exponentially greater geometric complexity.
It's all about bottlenecks. Different tools have different advantages,
and different bottlenecks. If somebody decides to work with a new tool,
then the different bottlenecks of that tool mean mean that they'll
actually work *differently*. The trend you mention is driven by the
bottlenecks of that paradigm; but a new paradigm, with different wins,
could mean that larger computational resources mean people use more
instances of articulated characters (larger crowds or more leaves in a
forest), or more precise volumetric light transport through skin. And
workflow changes, too -- instead of spending time adding fake lights to
simulate subtle indirect lighting effects, you could let your GI
algorithm worry about that and spend time tweaking the BRDF of this
wooden table or the musculature of that character.
> Huh? When we cite >GB scene sizes, that *is* using high-order
> surfaces, subdivs, compression of absolutely everything, procedural
> geometry, procedural shading, etc.
Right, but there's still more to go (and not *everybody uses subdivs
yet, you know). Compression doesn't mean just gzip; it could mean a
per-frame perspective-based surface simplification prepass, as is used
in high-complexity realtime algorithms (maybe that's what you meant; to
be honest, I don't know for certain *what* pre-render geometric
compression technologies Pixar, for instance, uses).
> > (e.g., why bother
> >to use jpgs for textures when network bandwidth is nearly free?).
>
> We *are* smart enough to compress textures (though not with JPEG,
> which is lossy, has horrible compression artifacts, and only supports
> 8 bits [I think]).
Da, I think it's 8 bits as well -- and yes, there are better
algorithms, agreed. But don't be so quick to dismiss lossy algorithms,
either; that's exactly my point -- different tradeoffs. Very slight
loss of texture quality vs. a 10-fold decrease in size could mean, given
*different bottlenecks*, quicker turnaround or the ability to turn up
samples per pixel for the same turnaround.
> Network bandwidth is *not* nearly free. Network bandwidth (and disk
> bandwidth) are among the most important bottlenecks in real studios
> working on big projects.
Okay, I overspoke; what I mean was that it's cheap compared to internet
distribution (which I'm sure you agree with ;-).
> I'd love to see the back-of-the-envelope calculations for how much
> bandwidth a real scene would take, how many machines "out there" would
> have the resources to handle it.
Your calculations are better than mine, I'm sure -- you know the
numbers much better. Oh, and I'll mention that in my opinion, RAM is
actually a much bigger issue than network bandwidth. For instance: a
broadband connection could transfer a 1GB scene in a matter of hours,
while rendering another frame, and then spend a day rendering it; plus,
with data reuse, e.g. through articulated models, the next frame can be
*much* cheaper to transfer. However, putting that 1GB scene in the RAM
of your average desktop box is simply not an option right now, or in the
next couple years. That problem is addressable too -- not just through
limiting the size of scenes once again, but by intelligent
simplification, caching and memory-coherent ray tracing, etc. It's
still a really hard problem, though.
-spc
--
/- Sean Cier <sc...@PostHorizon.com> -\
( If only you could see what I've seen with your eyes )
\- http://www.PostHorizon.com/scier -/
Sean Cier
>
> Sean Cier <sc...@PostHorizon.com> wrote:
> >Dreams of falling, dreams of flying;
>
> This is the voyage of the mind's eye.
Good call ;-)
-spc
--
/- Sean Cier <sc...@PostHorizon.com> -\
( If only you could see what I've seen with your eyes )
\- http://www.PostHorizon.com/scier -/
Simon Bunker
wrote:
>al...@r-m-c.REMOVETHIS.ru wrote:
>>
>> A while ago there was a similar thread here.
>> And many people were negative about the prospects of a similar
>> project. And the reason they noted was the significant differences
>> between, for example, seti@home and 3d rendering. Amount of input
>> data for a real frame of 3d animation can be measured in gigabytes.
>> So how do you plan to resolve this issue?
>
> Four points. Take each with a grain of salt. Perhaps no single one of
>these factors alone will eliminate the issue, and each involves some
>amount of ifs and guesswork, but just a few combined could make the
>technology feasable for some applications -- maybe a few, maybe most.
>
> First, not all scenes are gigabytes in size -- this depends on the
>studio and the scene itself. Maybe every frame of Pixar's "Finding
>Nemo" won't fit in an average desktop box's available RAM, but maybe
>Square USA's next film will; or maybe Independent Filmmaker Joe Smith's
>film will fit perfectly.
Well true - not everyone goes to the lengths Pixar does....
>
> Second, often much of the complexity is really pretty extraneous and
>exists simply to make the scene 'look good' to the eye (something that's
>admittedly tough to quantify), and the detail might become unncessary
>when higher-quality rendering is made available -- or at least, the
>final quality lost by decreased scene detail could be surpassed by that
>offered by more computationally intense algorithms.
It's not always down to the lighting - OK so you'll need less lights,
but most of the pressure is on the renderer processing the lights than
actually storing the light call - which is trivial.
Complex procedural textures take up the same storage space, however
complex or not it is set at - the algorithm is still the same! And if
you want to use geommetry you'd better have a damn good Level of
Detail system!
> Third, there's a lot of technology that can be used to reduce the size
>of scenes -- geometry compression, texture compression, higher-order
>procedural surfaces and shading, high-level articulated models, even use
>of subdivs rather than NURBS; much of this technology isn't used simply
>because it isn't neccessary in traditional renderfarms (e.g., why bother
>to use jpgs for textures when network bandwidth is nearly free?).
Not a good idea using JPEG's anyway - they aren't designed for
rendering - you're more likely to get aliasing and they are usually
not all that sharp - especially in the blue channel.
When it comes to photorealistic rendering quality is
one thing you can't afford to skimp on.
> Fourth, The capabilities of the network and computational nodes are
>increasing; broadband is fast becoming commonplace, disk storage is
>increasing by leaps and bounds, and system RAM is exhibiting expected
>geometric rates of increase.
I wouldn't say broadband is becoming commonplace over here yet - then
again the UK has always been backwards when it comes to new things.
Also even though it may be faster it isn't necessarily consistantly
fast - I think everyone knows how the internet can jam up at peak
times.
I'm not saying don't try this - I definitely want to try it myself -
however I don't have the same overheads a a big studio - just don't
rely on this as the big Vegas feature - I'd concentrate more on the
capabilities - a fast, not very noisy Monte Carlo renderer beats
network rendering anyday - making it portable across platforms is an
added bonus.
my 2 cents
Simon
>-spc
>
>--
> /- Sean Cier <sc...@PostHorizon.com> -\
>( Dreams of falling, dreams of flying; )
>( a man who never dreams goes slowly mad )
> \- http://www.PostHorizon.com/scier -/
http://www.rendermania.com/
UIN 11123737
Alex Segal
I would like to note that small projects can be easily rendered in-house.
Most of the 3D shots our small studio does (in TV res) are done over a night
or two on 4 our PIII's. So why do we need to hire Seti@home? I guess it 'll
take a week or so just to download the rendered frames via 56K modem :-)
--
Alex Segal. RMC Advertising
+7(3472) 52 59 66, 53 3133
al...@r-m-c.ru http://www.r-m-c.ru
Kevin Bjorke
BWAAAAAAAAAAAHAHAHAHAHAHA
sorry
> > Second, often much of the complexity is really pretty extraneous and
> >exists simply to make the scene 'look good' to the eye
Generally "look good" is the IDEA
> > Third, there's a lot of technology that can be used to reduce the size
> >of scenes -- geometry compression, texture compression, higher-order
> >procedural surfaces and shading...
Who do you think developed those techniques? For the most part, the very
people you imply are wasting resources and bandwidth. Believe me, these
are well-researched and tested topics!
Perhaps SOME project will come along that can be rendered on a simple
textureless raytracer (I'm thinking something the size of a business
card, heh) that could be broadly distributed. But most projects will not
fit that model.
This topic comes up so often, it almost begs to be put into the FAQ
(maybe on c.g.algorithms or c.g.rendering, or both).
*
kb FF LT Supv
Square Pictures
Stephen H. Westin
<snip>
> > > Second, often much of the complexity is really pretty extraneous and
> > >exists simply to make the scene 'look good' to the eye
>
> Generally "look good" is the IDEA
Well, I think he's claiming that better rendering will be a substitute
for geometric complexity. To date, I'm afraid that complexity wins out
over rendering in most cases. Perhaps a better way to put it is that
it's more practical to accomodate for a renderer that falls short of
physical accuracy that it is to accomodate for one that limits the
complexity of geometry and textures.
> > > Third, there's a lot of technology that can be used to reduce the size
> > >of scenes -- geometry compression, texture compression, higher-order
> > >procedural surfaces and shading...
>
> Who do you think developed those techniques? For the most part, the very
> people you imply are wasting resources and bandwidth. Believe me, these
> are well-researched and tested topics!
Well, yeah. I really think most folks can't imagine what it takes to
render even a few minutes of film-quality frames. They don't realize
that 100 megabits per second is still way too slow, much less the
uncertain bandwidth of the Internet.
Kevin Smith
You guys are all such buzzkills. 1. The internet will continue
to follow it's own version of Moore's Law. 2. Not everything people
do with computers is "Bug's Life". People do stylized/simplified stuff
all the time.
I think that anyone serious about distributed computing on the
internet will concede that it's not a great idea, right now. I think it
will be a great idea in the future, and guess who will have an established
infrastructure.
--
"..Theism's "continuing hold on the minds of many reasonable
people is surprising enough to count as a miracle in at
least the original sense.."
-- J.L. Mackie, The Miracle of Theism
Mark VandeWettering
>You guys are all such buzzkills.
Wow, I guess I am getting old, I don't have the vaguest idea what
that means.
>1. The internet will continue to follow it's own version of Moore's Law.
It is unclear that bandwidth follows anything approaching Moore's law,
and cheap bandwidth is the thing which could make distributed rendering
feasible.
>2. Not everything people do with computers is "Bug's Life". People do
> stylized/simplified stuff all the time.
The margins for most kinds of computer animation are very low.
>I think that anyone serious about distributed computing on the
>internet will concede that it's not a great idea, right now. I think it
>will be a great idea in the future, and guess who will have an established
>infrastructure.
On the contrary, there are many situations even know where distributed
computation can be quite useful and effective. Unfortunately, it is
pretty clear that the economics of film production aren't favorable.
In short, if your work isn't complicated, you can do it yourself for
so little money that outsourcing it doesn't matter, and if your work is
complicated and/or expensive, you are unlikely to be able to provide
sufficient capacity to be competitive, as well as providing additional
communications overhead (human, not computer) and security issues.
Mark
--
This signature has eight As, two Cs, three Ds, thirty Es, eight Fs, seven
Gs, nine Hs, fourteen Is, four Ks, two Ls, four Ms, nineteen Ns, thirteen Os,
two Ps, fifteen Rs, thirty one Ss, twenty four Ts, seven Us, six Vs, seven
Ws, two Xs, and four Ys. Mark VandeWettering <ma...@telescopemaking.org>
Sean Cier
forgetting to turn that thing off. They *were* all by my, I assure you,
despite any 'invalid signature' messages!
-spc
--
/- Sean Cier <sc...@PostHorizon.com> -\
( We are here to save the Erf )
\- http://www.PostHorizon.com/scier -/
Stephen H. Westin
<snip>
> > >2. Not everything people do with computers is "Bug's Life". People do
> > > stylized/simplified stuff all the time.
> >
> > The margins for most kinds of computer animation are very low.
>
> I'm not sure I see the connection here -- could you expand?
That those who are doing quick-and-dirty animation just aren't likely
to be a significant revenue source.
> > On the contrary, there are many situations even know where distributed
> > computation can be quite useful and effective. Unfortunately, it is
> > pretty clear that the economics of film production aren't favorable.
>
> This does appear to be the case, at least for a significant chunk of
> film rendering. However, remember -- things change.
>
> > In short, if your work isn't complicated, you can do it yourself for
> > so little money that outsourcing it doesn't matter
>
> This I don't buy; just because one's scene models aren't Pixarian in
> size, that doesn't mean that more complex rendering isn't desirable.
> There might be a correlation *right now* between those two extremes
> (e.g., Mainframe's production timelines mean streamlines,
> cartoon-quality production, and simplistic rendering simply because
> that's all there's time for; I mean, they introduced *shadows* a couple
> years ago simply because it became feasable computationally to turn that
> flag on; that's exactly what I'm talking about), but that won't
> neccessarily *always* be the case.
>
> > and if your work is
> > complicated and/or expensive, you are unlikely to be able to provide
> > sufficient capacity to be competitive, as well as providing additional
> > communications overhead (human, not computer) and security issues.
>
> If you're big enough that render farm purchase and maintainance are
> small blip on your radar, then no, such a significant paradigm shift
> quite probably *won't* outweight the risks, security issues, et cetera
> right now, at least not enough to justify switching the entire pipeline
> over.
Except that there's still the maintenance headache with a
"render-over-the-Internet" approach. Far bigger headaches, in fact,
since you don't have control over your machines. When you're trying to
render on a lot of machines, this is one of your biggest problems:
some machines will have the software installed wrong, others will have
their licenses expire, some will fill their disks and be unable to
render frames, etc. If you can't even log in to these machines to fix
the problems, you're up a creek. With your own render farm, you have
at least a fighting chance to keep all machines on the same release of
software, with identical default parameters, etc.
One thing that is widely misunderstood is the effect of scale: a
solution that is great with 10 machines will probably fail horribly
with 100. One vendor brought us render farm software that was
basically non-functional in our environment of 40-80 4-processor
Onyxes; we eventually found out that their test network had 4
workstations on it. We had dropped frames; we had partial frames; we
had frames that rendered wrong because of missing textures; we had
long waits as the software waited for that last frame that was never
going to come because the machine was down, or hung, or overloaded.
gri...@my-deja.com
network? Couldn't you have a rendrer that processed data intensive
elements locally but sent certain computationally intensive elements
out to the distributed network? As scenes become ever more complex it
sees that splitting them down into individual tasks is the way foreward.
Graham
Mark VandeWettering
>Is there any reason why the whole scene has to be sent out over the
>network? Couldn't you have a rendrer that processed data intensive
>elements locally but sent certain computationally intensive elements
>out to the distributed network? As scenes become ever more complex it
>sees that splitting them down into individual tasks is the way foreward.
Make this work, and collect your PhD. I'd say "Nobel Prize", but there
are no Nobels for Computer Science.
To put some reality on this, suppose you are rendering frames with
2M pixels (images are 2K x 1K or thereabouts). Further imagine
that you divide the images into 8x8 buckets, and can efficiently
dispatch individual buckets to remote machines for rendering (you
can't but let's pretend). That's 32K buckets. Consider that you
have a budget of (say) 1 hour turn around time. That's 3600 seconds,
which means that you only have .11 seconds or so per bucket.
Compare that to typical network latencies over the Internet, and
it becomes clear that in most cases, the network latencies dominate.
Try to increase bucket size? That reduces load balancing. Also
try to consider texture access from remote clients. It would be
nice if such accesses occurred automatically (to make work flow
more straight forward) and quickly (obviously).
There are *tons* of ordinary, boring systems and network level issues
that need to be considered here. It is most of what renderer writers
actually deal with at studios.
>Graham
Sean Cier
>
> gri...@my-deja.com <gri...@my-deja.com> wrote:
>
> >Is there any reason why the whole scene has to be sent out over the
> >network? Couldn't you have a rendrer that processed data intensive
> >elements locally but sent certain computationally intensive elements
> >out to the distributed network? As scenes become ever more complex it
> >sees that splitting them down into individual tasks is the way foreward.
>
> Make this work, and collect your PhD.
True. Sigh. That would make so many things *so* much easier. There
are ways to *approach* the problem -- e.g., divide up an image into
overlapping 4x4 regions, derive a scene file for each segment by
applying agressive perspective-based simplification, render the regions
from their simplified scene files, then combine using weighting in the
overlap regions to 'hide' the inevitable mismatch at the edges -- but
right now, they're still imperfect.
> To put some reality on this, suppose you are rendering frames with
> 2M pixels (images are 2K x 1K or thereabouts). Further imagine
> that you divide the images into 8x8 buckets, and can efficiently
> dispatch individual buckets to remote machines for rendering (you
> can't but let's pretend). That's 32K buckets. Consider that you
> have a budget of (say) 1 hour turn around time. That's 3600 seconds,
> which means that you only have .11 seconds or so per bucket.
> Compare that to typical network latencies over the Internet, and
> it becomes clear that in most cases, the network latencies dominate.
> Try to increase bucket size? That reduces load balancing. Also
> try to consider texture access from remote clients. It would be
> nice if such accesses occurred automatically (to make work flow
> more straight forward) and quickly (obviously).
Several points; such a division would only be ideal if you had on the
order of 32K * N frames number of machines to render on -- and even
then, as you say, network bandwidth issues on the result-gathering side
quickly become the bottleneck. But unless you're rendering on precisely
one box, I don't understand how latency becomes an issue. In any case,
I agree that 1 hour is a limiting case for the efficiency of any
internet-based computation; we usually think of turnaround on the order
of a day -- tasks of perhaps a few hours to a day in length, and jobs
(sets of tasks) lasting the better part of a day to weeks or even
months. (I'm not just being contrary here -- I'm interested in hearing
your thoughts on each of these three points, Mark).
-spc
--
/- Sean Cier <sc...@PostHorizon.com> -\
( Yield, he told the silver triangle. Cough up arcane secret. )
\- http://www.PostHorizon.com/scier -/
Matt Pharr
Bill Heiden <bi...@anim.dreamworks.com> writes:
> Sean Cier wrote:
>
> > True. Sigh. That would make so many things *so* much easier. There
> > are ways to *approach* the problem -- e.g., divide up an image into
> > overlapping 4x4 regions, derive a scene file for each segment by
> > applying agressive perspective-based simplification, render the regions
> > from their simplified scene files, then combine using weighting in the
> > overlap regions to 'hide' the inevitable mismatch at the edges -- but
> > right now, they're still imperfect.
>
> It's pretty easy to render regions in a tiff format and then join the
> pieces.
Note that the RenderMan spec is actually quite careful to require that
pixels that are rendered at the edges of a crop-window match what they
would have been in the absence of a crop window. So there should be no
such mismatch in a good RenderMan-type renderer.
To get this right, each render process either needs to expand the width of
its tile by the filter's extent and compute samples in that region of the
screen as well, or (to save computation), each process should send back
unfiltered screen sample values, which are then filtered down into pixel
values.
-matt
--
Matt Pharr <URL:http://graphics.stanford.edu/~mmp>
===============================================================================
In a cruel and evil world, being cynical can allow you to get some
entertainment out of it. --Daniel Waters
Bill Heiden
> True. Sigh. That would make so many things *so* much easier. There
> are ways to *approach* the problem -- e.g., divide up an image into
> overlapping 4x4 regions, derive a scene file for each segment by
> applying agressive perspective-based simplification, render the regions
> from their simplified scene files, then combine using weighting in the
> overlap regions to 'hide' the inevitable mismatch at the edges -- but
> right now, they're still imperfect.
Been doing this with both Maya and Renderman for quite some time.
It's pretty easy to render regions in a tiff format and then join the
pieces.
--
--------------------------------------------------------------------------
Bill Heiden
bi...@anim.dreamworks.com
--------------------------------------------------------------------------
Bill Heiden
> Imagine an experimental studio, ten years into the future. One person
> puts together a story and script. Half-a-dozen sculptors and painters
> create characters and an environment, lighting it simply, realistically;
> perhaps a sparse theatre feel, perhaps not. A small crew runs audio and
> motion-control gear in a modest studio to capture the cast acting out
> the story. The animation is edited, and score added. The resulting
> scene files are preprocessed on a couple boxes in-house, sent over the
> 'net for a pre-render, the results reviewed and tweaked, and the entire
> thing sent out for a final four-month render, all using powerful GI
> algorithms to take care of realistic light transport. A studio reviews
> the preview, decides to release it in a month; the power is turned up,
> the result is done in two weeks, final edits performed, and the final
> cut reviewed and sent to theatres.
You really have a simplified vision of how a film is made. You have
left out storyboard/animatic artists; you just don't sit down and
make an animated film without this step. You neglect to mention
a producer, a director, coordinators, a finance person to handle
payments, probably a couple of PAs to run errands, some sort
of engineering support, an editor, etc... Even with a highly
stylized look (read simple), the making of a cg film still requires
a number of people. We are talking about a collaborative effort, with
many hands contributing to the final product.
> That's a much smaller technical team than a photographic film crew,
Horsefeathers. A photographic crew keeping it simple by using
availble light and existing locations can be very small.; cameraman,
soundman, gofer/reflector holder, and director (who also probably
is the cameraman). Other than talent, traditional fim crews
can be very small and obviously very large depending on
what is being shot.
> My point is, *don't make assumptions about the future of film
> production*. Make decisions for the present only; 'never' rarely has a
> place in the creative arts.
The people in this newsgroups don't make assumptions,
but rather judgements based on years of experience and
trends they see. Believe me, if an internet distributed
service could provide reliable, cost-efficent, and
secure rendering, they would have the interest of many
people. Your online resume has no mention of
film making, so it comes across pretty arrogant when
you dimiss the opinions of professionals in this
newsgroup that have many years of experience.
Sean Cier
>
> Sean Cier wrote:
>
> > True. Sigh. That would make so many things *so* much easier. There
> > are ways to *approach* the problem -- e.g., divide up an image into
> > overlapping 4x4 regions, derive a scene file for each segment by
> > applying agressive perspective-based simplification, render the regions
> > from their simplified scene files, then combine using weighting in the
> > overlap regions to 'hide' the inevitable mismatch at the edges -- but
> > right now, they're still imperfect.
>
> Been doing this with both Maya and Renderman for quite some time.
> It's pretty easy to render regions in a tiff format and then join the
> pieces.
Really? Using different scene files for each tile? I'd be interested
to hear more about what kind of results you see; perhaps it deserves to
be taken into a different thread.
Mark VandeWettering
>Mark VandeWettering wrote:
>>
>> gri...@my-deja.com <gri...@my-deja.com> wrote:
>>
>> >Is there any reason why the whole scene has to be sent out over the
>> >network? Couldn't you have a rendrer that processed data intensive
>> >elements locally but sent certain computationally intensive elements
>> >out to the distributed network? As scenes become ever more complex it
>> >sees that splitting them down into individual tasks is the way foreward.
>>
>> Make this work, and collect your PhD.
>
>are ways to *approach* the problem -- e.g., divide up an image into
>overlapping 4x4 regions, derive a scene file for each segment by
>applying agressive perspective-based simplification, render the regions
>from their simplified scene files, then combine using weighting in the
>overlap regions to 'hide' the inevitable mismatch at the edges -- but
>right now, they're still imperfect.
>
>> 2M pixels (images are 2K x 1K or thereabouts). Further imagine
>> that you divide the images into 8x8 buckets, and can efficiently
>> dispatch individual buckets to remote machines for rendering (you
>> can't but let's pretend). That's 32K buckets. Consider that you
>> have a budget of (say) 1 hour turn around time. That's 3600 seconds,
>> which means that you only have .11 seconds or so per bucket.
>> Compare that to typical network latencies over the Internet, and
>> it becomes clear that in most cases, the network latencies dominate.
>> Try to increase bucket size? That reduces load balancing. Also
>> try to consider texture access from remote clients. It would be
>> nice if such accesses occurred automatically (to make work flow
>> more straight forward) and quickly (obviously).
>
> Several points; such a division would only be ideal if you had on the
>order of 32K * N frames number of machines to render on -- and even
>then, as you say, network bandwidth issues on the result-gathering side
>quickly become the bottleneck. But unless you're rendering on precisely
>one box, I don't understand how latency becomes an issue. In any case,
>I agree that 1 hour is a limiting case for the efficiency of any
>internet-based computation; we usually think of turnaround on the order
>of a day -- tasks of perhaps a few hours to a day in length, and jobs
>(sets of tasks) lasting the better part of a day to weeks or even
>months. (I'm not just being contrary here -- I'm interested in hearing
>your thoughts on each of these three points, Mark).
If your turn around times are on the order of a day for frames,
and often more, you already aren't moving in the direction that
studios need. Turnaround time for shading and lighting tests needs
to be minimized to get effective use out of your human resources
(lighting and shading technical directors), which tend to dominate
purely IT costs (it is much more difficult to double the number of
tds than to double the number of machines in your renderfarm).
Regarding latency, my assumption is that you are trying to return
individual frames as quickly as possible (not the only possible
goal, but certainly a useful one). If you rely on a screen space
subdivision scheme based on buckets as I hypothesized above, then
there is a distinct tradeoff: the time to send work to a remote
host and get it back, versus the time spent just rendering it
yourself. Latency is your enemy in this case, often times the time
to actually send and receive a bucket to a remote site over the
network exceeds the 100ms or so that I hypothesized above. While
it is true that you could POTENTIALLY overcome this with parallelism,
network bandwidth quickly becomes and issue. It becomes exceedingly
hard to make schemes like this work in a reasonable way.
Another diversion. Raytracing is often listed as the perfectly
parallelizable application, but actually making this work for
arbitrarilly complex models is very difficult. Virtual memory
issues rapidly become an issue, and often schlepping data around
any kind of loosely coupled network is more time consuming than
doing the necessary graphics computations.
Mark
>-spc
Sean Cier
replying to.
Bill Heiden wrote:
>
> Sean Cier wrote:
> > My point is, *don't make assumptions about the future of film
> > production*. Make decisions for the present only; 'never' rarely has a
> > place in the creative arts.
>
> The people in this newsgroups don't make assumptions,
I think that's a very funny thing to say ;-)
> Your online resume has no mention of
> film making, so it comes across pretty arrogant when
> you dimiss the opinions of professionals in this
> newsgroup that have many years of experience.
I'm *not* dismissing opinions, and I have *openly said* numerous times
on this thread that I have far less experience in the film industry than
several others in this group; and beyond that, I respect their
opinions. But this *doesn't mean you're always right*, it doesn't mean
I can't disagree with you especially when making decade-plus
projections, and it doesn't mean that resorting to attacks on my
background improves the value of a discussion one iota. I don't
understand why you're offended by what I have to say; if you re-read my
posts, I think you'll see that I didn't march in here to tell you you're
all idiots and that you should be calling my company's sales
department. If I've come off as venemous or egotistical in your eyes,
then that was a failure to communicate on my part, not my intention.
> You really have a simplified vision of how a film is made. You have
> left out storyboard/animatic artists; you just don't sit down and
> make an animated film without this step. You neglect to mention
> a producer, a director, coordinators, a finance person to handle
> payments, probably a couple of PAs to run errands, some sort
> of engineering support, an editor, etc... Even with a highly
> stylized look (read simple), the making of a cg film still requires
> a number of people. We are talking about a collaborative effort, with
> many hands contributing to the final product.
[...]
>
> > That's a much smaller technical team than a photographic film crew,
>
> Horsefeathers. A photographic crew keeping it simple by using
> availble light and existing locations can be very small.; cameraman,
> soundman, gofer/reflector holder, and director (who also probably
> is the cameraman). Other than talent, traditional fim crews
> can be very small and obviously very large depending on
> what is being shot.
Excuse me? You just completely disagreed with yourself from one
paragraph to the next. Stop accusing me of being arrogant if you're
just disagreeing with every statement I make for the sake of
disagreement. The few paragraphs you were quoting were referring to an
*experimental studio a decade into the future* -- things change, which
was my point. You don't have storyboard/animatic artists in community
theatre -- that's silly. Hence it is *not* a neccessary step in
creating a piece of visual storytelling, it just happens to be an
efficient means to an end in the current process of creating an animated
film. Who would have thought Toy Story could have been made with a
fraction of the number of animators required for a traditional film?
Technology can improve workflow (it doesn't always, but it often does --
dramatically so); is it unreasonable to posit a return to storytelling
by a small group of people, even without a reduction in technical
quality? Yes, it *is* a collaborative effort; my point was that it
could be a collaboration betwen a few dozen people rather than a few
hundred. I didn't try to list every single person neccessary for the
production; that was extraneous detail.
So, to restate my point in this little branch of the thread: in the
future, technology may enable high-quality films to be created by a
small group of people, requiring a relatively small budget, and in an
environment such as this the eceonomics of on-demand internet computing
may prove a large incentive when compared to the purchase and
maintenance of a large render farm -- perhaps a large enough incentive
to outweight some of the disadvantages, such as distribution of IP and
lack of physical control. It would not neccessarily solve technical
difficulties, but I'm talking here of business considerations. I'm open
to being disagreed with, of course; but please address my point, not the
details of my delivery.
I'm fully expecting to be flamed now for living on the East coast and
hence not being able to understand the fundamentals of storytelling as a
business.
-spc
--
/- Sean Cier <sc...@PostHorizon.com> -\
( If only you could see what I've seen with your eyes )
\- http://www.PostHorizon.com/scier -/
Sean Cier
>
> If your turn around times are on the order of a day for frames,
> and often more, you already aren't moving in the direction that
> studios need. Turnaround time for shading and lighting tests needs
> to be minimized to get effective use out of your human resources
> (lighting and shading technical directors), which tend to dominate
> purely IT costs (it is much more difficult to double the number of
> tds than to double the number of machines in your renderfarm).
Yeah, I agree the numbers work out much better for final renders than
for dailies, as it were. In your opinion, could such tests be performed
effectively on programmatically simplified models (e.g., simplified
geometry outside of particular areas of interest)? This could allow for
efficient reduced turnaround times. On ths flip side of this coin, it
means that if the numbers do work out for quick daily turnaround of test
renders for some project, this means that much more computation could be
thrown at the tests -- that is, if you can render a lighting test
overnight, you can probably render at full quality, with all the shading
et cetera turned on. I'd also hope that solid, albeit computationally
intense GI algorithms, as they require less 'faking', would reduce the
number of testing iterations required -- again impacting the TD effort;
but that's something that'd probably have to be tried to know for sure.
> Regarding latency, my assumption is that you are trying to return
> individual frames as quickly as possible (not the only possible
> goal, but certainly a useful one).
I generally consider that completion times should be minimized for a
set of frames. Is this not a good goal?
> If you rely on a screen space
> subdivision scheme based on buckets as I hypothesized above, then
> there is a distinct tradeoff: the time to send work to a remote
> host and get it back, versus the time spent just rendering it
> yourself. Latency is your enemy in this case, often times the time
> to actually send and receive a bucket to a remote site over the
> network exceeds the 100ms or so that I hypothesized above. While
> it is true that you could POTENTIALLY overcome this with parallelism,
> network bandwidth quickly becomes and issue. It becomes exceedingly
> hard to make schemes like this work in a reasonable way.
Agreed; this is why I tend to prefer parallelization across frames,
rather than considering them serially. Data is more coherent this way.
Let's create a scenario where we need to render 16 frames across 16
nodes, and only care when all the frames are done. We can either (A)
divide each frame into 4x4 sections, and send each node a region from
each frame, or (B) send each node one frame. (Of course, it's actually
a continuum). If each frame scene file is 100MB (let's ignore data
re-use across frames for now), (A) results in 1.6GB transferred to each
node and 25.6GB going through the server, while (B) results in 100MB
transferred to each node, and 1.6GB going through the server. The same
amount of computational work is performed in each case, but (B) results
in a much lower probability of data transfer becoming a bottleneck.
This math is obvious, of course, it just seemed an easier way to explain
my thoughts.
> Another diversion. Raytracing is often listed as the perfectly
> parallelizable application, but actually making this work for
> arbitrarilly complex models is very difficult. Virtual memory
> issues rapidly become an issue, and often schlepping data around
> any kind of loosely coupled network is more time consuming than
> doing the necessary graphics computations.
Right; so the way to make it efficient is using more computationally
intense algorithms. This is backwards, of course; you don't waste
computation just because you have it, or just because the numbers work
out better. But when you flip it back around, it says that the
penalties of using a networked approach stay the same, but the benefits
increase with more expensive algorithms. For instance, film production
doesn't use real GI because it's too expensive computationally. But, if
you swallow the tradeoffs of a networked approach, you get GI's benefits
nearly for free -- better results, quite possibly less TD effort, et
cetera. That's what I've meant when I've mentioned 'different
bottlenecks' in a couple messages; if you can accept the new bottlenecks
(e.g. bandwidth), some of the old ones go away.
-spc
--
/- Sean Cier <sc...@PostHorizon.com> -\
( If only you could see what I've seen with your eyes )
\- http://www.PostHorizon.com/scier -/
Larry Gritz
Sean Cier <sc...@PostHorizon.com> wrote:
> Right; so the way to make it efficient is using more computationally
>intense algorithms. This is backwards, of course; you don't waste
>computation just because you have it, or just because the numbers work
My experience contradicts this assertion.
> For instance, film production
>doesn't use real GI because it's too expensive computationally.
I think that as has been discussed here in the past, expense is
significant but by no means the sole reason (or even the main reason?)
that GI is eschewed in high-end production.
Your assertion would carry significant weight if it turned out that
greater resources were actually correlated with use of GI. However, I
would be willing to bet that if you were to compare studio size/budget
and algorithms across a wide range of well-known projects, you'd find
a *negative* correlation between GI and budget. In other words, it's
the largest studios with the biggest budgets and most expansive render
farms that have most vehemently stayed away from GI (except for a bit
of dabbling or a few particular effects). The place you'll see a
strong positive correlation is between size/budget/farm and geometric
complexity.
Sean Cier
>
> In article <3A1317D0...@PostHorizon.com>,
> Sean Cier <sc...@PostHorizon.com> wrote:
> >intense algorithms. This is backwards, of course; you don't waste
> >computation just because you have it, or just because the numbers work
>
> My experience contradicts this assertion.
Heh; yeah, taken the way I think you parsed it I suppose it's true --
but I meant it to parse "one generally should not plan to waste
computation just because one can". In reality, once the computation
becomes available, things *do* expand to use it, alas; but that was
along the lines of my point anyhow.
> > For instance, film production
> >doesn't use real GI because it's too expensive computationally.
>
> I think that as has been discussed here in the past, expense is
> significant but by no means the sole reason (or even the main reason?)
> that GI is eschewed in high-end production.
>
> Your assertion would carry significant weight if it turned out that
> greater resources were actually correlated with use of GI. However, I
> would be willing to bet that if you were to compare studio size/budget
> and algorithms across a wide range of well-known projects, you'd find
> a *negative* correlation between GI and budget. In other words, it's
> the largest studios with the biggest budgets and most expansive render
> farms that have most vehemently stayed away from GI (except for a bit
> of dabbling or a few particular effects).
Interesting; why is this? Is it because there is generally risk in
using relatively untested algorithms, and high budgets call for low
risk?
> The place you'll see a
> strong positive correlation is between size/budget/farm and geometric
> complexity.
I'd definitely expect this; high budgets mean you can afford both the
talent to create additional geometry as well as the hardware to render
it. Or conversely, a high budget means you can afford lots of hardware,
and given current tradeoffs, the best way to take advantage of a
marginal increase in computational power is to increase complexity, as
you've said. (If you had a *lot* more computation, the best way to take
advantage of it would probably be different algorithms; the mathematics
I'm visualizing that suggest this warrant a whole other discussion, but
it can be seen in the limit of cheap computation at least: the price in
manpower of producing geometric complexity remains linear in the limit,
hence the marginal returns of better algorithms as the price of
computation goes to zero must surpass the marginal returns of more
complexity. Where this crossover occurs is an interesting question, and
one I don't have nearly the hands-on experience to offer a guess about).
-spc
--
/- Sean Cier <sc...@PostHorizon.com> -\
( Dreams of falling, dreams of flying; )
( a man who never dreams goes slowly mad )
\- http://www.PostHorizon.com/scier -/
Neil Okamoto
Sean Cier <sc...@PostHorizon.com> wrote:
> nodes, and only care when all the frames are done. We can either (A)
> divide each frame into 4x4 sections, and send each node a region from
> each frame, or (B) send each node one frame. (Of course, it's
actually
> a continuum). If each frame scene file is 100MB (let's ignore data
> re-use across frames for now), (A) results in 1.6GB transferred to
each
> node and 25.6GB going through the server, while (B) results in 100MB
> transferred to each node, and 1.6GB going through the server. The
same
> amount of computational work is performed in each case, but (B)
results
> in a much lower probability of data transfer becoming a bottleneck.
> This math is obvious, of course, it just seemed an easier way to
explain
> my thoughts.
... assuming each node picks up jobs at random, and no scene data is
cached at the node. If you cache data locally and "line up" similar
jobs on the same node then data xfers could be reduced considerably (at
the expense of serializing the frames on fewer processors and increasing
the total turnaround time). It really depends on what problem you're
trying to solve.
I would say, "He who is willing to send render jobs out over the
internet, can't be picky about fast turnaround." I mean, what if someone
decides to turn off their machine in the middle of my job? How quickly
would the queue manager realize this and restart on another machine?
What if someone picks up my job on a 486DX/66 and works on it for a few
days before finishing? etc. etc.
n
Mark VandeWettering
>Larry Gritz wrote:
>>
>> In article <3A1317D0...@PostHorizon.com>,
>> Sean Cier <sc...@PostHorizon.com> wrote:
>> >intense algorithms. This is backwards, of course; you don't waste
>> >computation just because you have it, or just because the numbers work
>>
>> My experience contradicts this assertion.
>
> Heh; yeah, taken the way I think you parsed it I suppose it's true --
>but I meant it to parse "one generally should not plan to waste
>computation just because one can". In reality, once the computation
>becomes available, things *do* expand to use it, alas; but that was
>along the lines of my point anyhow.
Even with your revised parsing it is also not true. We have had shots
with 18 hour render times which we don't bother tweaking because "well,
the shots are going through, and its only 50 frames, and while we
wouldn't want to do this with every shot, by tomorrow it will all be
forgotten." And that is ABSOLUTELY the right decision to make for us.
>> > For instance, film production
>> >doesn't use real GI because it's too expensive computationally.
>>
>> I think that as has been discussed here in the past, expense is
>> significant but by no means the sole reason (or even the main reason?)
>> that GI is eschewed in high-end production.
>>
>> Your assertion would carry significant weight if it turned out that
>> greater resources were actually correlated with use of GI. However, I
>> would be willing to bet that if you were to compare studio size/budget
>> and algorithms across a wide range of well-known projects, you'd find
>> a *negative* correlation between GI and budget. In other words, it's
>> the largest studios with the biggest budgets and most expansive render
>> farms that have most vehemently stayed away from GI (except for a bit
>> of dabbling or a few particular effects).
>
> Interesting; why is this? Is it because there is generally risk in
>using relatively untested algorithms, and high budgets call for low
>risk?
No, for Pixar I suspect it is mostly a control issue: we get better control
over lighting by having highly non-physical lights. In other words, cheating
illumination is the norm, and global illumination often makes that more
difficult.
>> The place you'll see a
>> strong positive correlation is between size/budget/farm and geometric
>> complexity.
>
> I'd definitely expect this; high budgets mean you can afford both the
>talent to create additional geometry as well as the hardware to render
>it. Or conversely, a high budget means you can afford lots of hardware,
>and given current tradeoffs, the best way to take advantage of a
>marginal increase in computational power is to increase complexity, as
>you've said. (If you had a *lot* more computation, the best way to take
>advantage of it would probably be different algorithms; the mathematics
>I'm visualizing that suggest this warrant a whole other discussion, but
>it can be seen in the limit of cheap computation at least: the price in
>manpower of producing geometric complexity remains linear in the limit,
>hence the marginal returns of better algorithms as the price of
>computation goes to zero must surpass the marginal returns of more
>complexity. Where this crossover occurs is an interesting question, and
>one I don't have nearly the hands-on experience to offer a guess about).
Mark
Kevin Bjorke
> internet, can't be picky about fast turnaround."
Also, he who is willing to send render jobs out over the internet
probably doesn't care when the frames pop up unannounced on A.I.C.N.
Bill Heiden
> Bill Heiden wrote:
> >
> > Sean Cier wrote:
>
> I'm *not* dismissing opinions, and I have *openly said* numerous times
> on this thread that I have far less experience in the film industry than
> several others in this group; and beyond that, I respect their
> opinions.
You have *no* experience in film business, which is considerably
different than having *less* experience in the film business. :-)
> But this *doesn't mean you're always right*, it doesn't mean
> I can't disagree with you especially when making decade-plus
> projections, and it doesn't mean that resorting to attacks on my
> background improves the value of a discussion one iota.
Don't post an url to your resume and expect me not to
consider it. BTW, I only mentioned that your resume
had no film making experience and that you dismissing
the opinions of the professionals in the group comes
across arrogant.
> > You really have a simplified vision of how a film is made. You have
> > left out storyboard/animatic artists; you just don't sit down and
> > make an animated film without this step. You neglect to mention
> > a producer, a director, coordinators, a finance person to handle
> > payments, probably a couple of PAs to run errands, some sort
> > of engineering support, an editor, etc... Even with a highly
> > stylized look (read simple), the making of a cg film still requires
> > a number of people. We are talking about a collaborative effort, with
> > many hands contributing to the final product.
> [...]
> > > That's a much smaller technical team than a photographic film crew,
> >
> > Horsefeathers. A photographic crew keeping it simple by using
> > availble light and existing locations can be very small.; cameraman,
> > soundman, gofer/reflector holder, and director (who also probably
> > is the cameraman). Other than talent, traditional fim crews
> > can be very small and obviously very large depending on
> > what is being shot.
>
> Excuse me? You just completely disagreed with yourself from one
> paragraph to the next.
No I didn't. The first paragraph you speak of the staff needed for a
CG project, the second paragraph you make an assumption about
the size of a technical staff needed to make a live action film. These
are two different subjects.
> The few paragraphs you were quoting were referring to an
> *experimental studio a decade into the future* -- things change, which
> was my point. You don't have storyboard/animatic artists in community
> theatre -- that's silly. Hence it is *not* a neccessary step in
> creating a piece of visual storytelling, it just happens to be an
> efficient means to an end in the current process of creating an animated
> film.
Storboarding is an essential part of making an animated
film as well as a traditional film. It is the previsualization
of the finished product, a very necessary step. It is
the initial direction for the animators. You may think
this step can be omitted, but the final product will
suffer from the lack of a unified direction.
> technology may enable high-quality films to be created by a
> small group of people, requiring a relatively small budget,
Define small group of people, small budget, and high quality film. Also,
are you speaking of a full 90 minute feature or a 5 minute short? A
small group of is totally capable of producing an interesting short, but
a feature is another thing.
> I'm fully expecting to be flamed now for living on the East coast and
> hence not being able to understand the fundamentals of storytelling as a
> business.
It's not about where you live...
Kevin Smith
>
Sean, if nothing else, you've got vision. Don't let the nay-sayers get you down.