clustered hugin
michael.grant
of hugin.
I'm wondering if there's been any development work on this.
I have some general ideas I'd like to toss out.
Could hugin be split up to run part of it's stitching remotely? For
example, a stitch server would sit there and accept requests up to the
number of CPUs it had available. The front end would feed pieces to
as many stitch servers that were available. The server could run on a
linux/unix server or as a windows service.
Ultimately it would be really cool if people could run some public
stitch servers (like grid computing) people could use the spare
compute power of idle computers to stitch their images.
Michael Grant
Bruno Postle
>
>Could hugin be split up to run part of it's stitching remotely?
Yes, the Makefile stitching system used by Hugin is very suited to
distribution over multiple machines.
--
Bruno
Roger Goodman
There used to be a product from SUN, called Sun Grid Engine (SGE)
that was made to distribute jobs over multiple computers for
processing. It was free a couple years ago, I haven't looked lately,
now that Oracle owns them. It might be worth looking into. As I
recall, it would run on Windows, Linux, or Solaris systems, but they all
had to be the same OS in the grid.
Roger Goodman
michael.grant
We could create our own grid. There's also things like BOINC, but
BOINC is a research grid. BOINC is something you download and install
on your machine and it allows people to run stuff on your machine.
There's a control panel and you decide which projects to donate your
spare computrons to. For example, you could say 50% to Einstein@home
and 50% to Seti@home. Then, you ignore it. When you're computer's
screenlock comes on, your computer starts working on these tasks. It
sits there, gets a new task, and when it's done, it returns the
results. As soon as you come back to your computer, it stops the grid
task. You never know it's there. I've been running it for years on
my computers and never had a problem with it.
I don't know if BOINC would let us build something to let people
submit panos to a grid that used the BOINC platform. This may be
overkill, and it may take a central server somewhere.
In the shorter term, it would be very worth while if hugin could be
split into two parts, a server and a client which could submit things
to the server(s).
Bruno, when you talk about a makefile model, where can I see that? I
see, on windows, there's a make.exe, and I'm familiar with make.
Where/how are these makefiles generated that make is run on?
Would the other machines need access to all the images in the pano?
Could only the overlapping parts of the images be sent to the other
server for stitching?
Michael
Bruno Postle
>
> Bruno, when you talk about a makefile model, where can I see that?
> I see, on windows, there's a make.exe, and I'm familiar with make.
> Where/how are these makefiles generated that make is run on?
There is some documentation here:
http://wiki.panotools.org/Panorama_scripting_in_a_nutshell#Makefile_stitching_system
Basically Hugin constructs a list of all the temporary files and the
rules to assemble them, then writes it all to a Makefile. The
stitching process is then managed by gnu make, you can
close Hugin, or start a new project during stitching, or run
stitching entirely on the command-line later or on another machine.
> Would the other machines need access to all the images in the
> pano? Could only the overlapping parts of the images be sent to
> the other server for stitching?
The simplest way to do it is to use something like 'distmake', this
would require that all the photos were on a shared filesystem
--
Bruno
Simon Oosthoek
At my current work a tool is being developed for this kind of
distributed computing only it's intended for matlab based computations.
Part of the system is also built in C, perhaps it can be adapted to
suite hugin?
Have a look at http://fieldtrip.fcdonders.nl (look in the FAQ section on
peer computing).
Cheers
Simon
michael.grant
So getting everything in one shared partition and running distmake
over many machines is one method. I have some questions about this:
Is there some part of the process (i.e. one of the commands in the
makefile) that require to have the entire image in memory? And is
that on every machine? How can I estimate how much memory each
machine would need on each machine?
Are there parts of the process that will only run on one machine that
will block it from using multiple machines? For example the emblend?
There is also a parallel make where you start make with a --jobs flag
and it does something similar to distmake except locally. Let's say
we used that but instead of running the commands like nona -z LZW -r
ldr -m TIFF_m -o "IMG_5290-IMG_5440" -i 127 hug63C.tmp, it could
prefix it with something, let's call it rrun, that would package up
the necessary files off to another machine, run them, and return the
result. This would be more grid-like. It might enable us to create
some sort of pano server to process large panos on lots of machines
without actually needing to have shared disk.
One of the big questions in my mind though is if this could even work
because at some point, one of these processes on such another machine
might need access to everything, or it might need somehow to merge
it's result into a larger file.
Has anyone ever built a script to reserve some number of amazon ec2
instances, fire off a job, let it run, and return the result and shut
down the instances? Such a run might only cost a few dollars (I
hope!) and run very very fast.
And by the way, this is the sort of thing one might do to stitch say
thousands of images, something that a single machine could take weeks
to do.
Michael
On Jan 12, 11:05 pm, Bruno Postle <br...@postle.net> wrote:
> On Wed 12-Jan-2011 at 13:28 -0800, michael.grant wrote:
>
>
>
> > Bruno, when you talk about a makefile model, where can I see that?
> > I see, on windows, there's a make.exe, and I'm familiar with make.
> > Where/how are these makefiles generated that make is run on?
>
kevin
spreading the stitching programs across them to see how much of a
speed up I can get. All the tests I've done are with a final
stitch that's 500Mpixel in size and uses 174 images which are stacks
of 3 bracketed images. I picked this size stitch because it doesn't
swap at all. I know at 750Mpixel enblend gets big enough that it
needs some drive space to work, and once you start hitting the drive
like that it's going to slow down a lot.
The specs for the three machines are:
A - Core i7 2.6GHz 4 core, 24G memory, Nvidia GTX460, Slackware 64-bit
B - Core 2 2.4GHZ 4 core, 8G memory, Nvidia 9600GT, Slackware 64-bit
C - Core 2 2.0GHz 2 core, 4G memory, Nvidia 8400GS, Slackware 64-bit
Here's what I've learnt so far - now this is relevant for how I
stitch together, for someone else I'm sure they'd get different
numbers. I run hugin to make both the "Exposure fused from stacks"
and "Exposure fused from any arrangement".
There are five stitching parts:
1 - nona for layer images
2 - enfuse for ldr images
3 - enblend for exposure images (needs layer images)
4 - enfuse for blended_fused image (needs exposure images)
5 - enblend for fused image (needs ldr images)
nona - the single biggest speedup for nona is being able to use the
GPU. I've also found that while nona is threaded, I get the fastest
times by running the same number of nona as the number of cores on
the machine - all set to use the GPU. So for a 4-core machine, 4
nonas at a time all with the -g option. A 2-core machine, 2 nonas at
a time both using the GPU.
enfuse - this one is similar to nona, it's threaded but I get the
fastest times by running the same number of enfuse processes as the
number of cores on the machine.
enblend - the most speedup I've gotten from enblend is using the '-a'
option. I've tried using the --gpu but it always crashes at some
point. I've tried the --gpu with and without '-a', but still
crashes.
For stages 3, 4, & 5, because the final images they produce are large
in size (500MPixel) I have to run them on machine A, otherwise they
start swapping. Now I can turned off the -a to enblend and that
will let me run stage 3 on the 8GB memory machine. However, I did
some tests and not a huge benefit, that will come later. If I had
other machines with 24G of memory then I could spread them out.
Here's the numbers, hopefully this will come out looking ok.
1st column - labels for stage (1-5) and a total (t).
2nd column - time in min/sec using all 3 machines
3rd column - % of time compared to total in col 2
4th column - time in min/sec just running locally
5th column - % of time compared to total in col 4
Column 4 and 5 would be the times taken if you just pressed the
stitch button in hugin.
1 3m29.959s 4.86% 21m37.362s 21.85%
2 4m06.984s 5.71% 20m16.820s 20.50%
3 24m37.791s 34.18% 24m37.791s 24.89%
4 4m50.763s 6.73% 4m50.763s 4.90%
5 7m13.954s 10.04% 7m13.954s 7.31%
44m19.451s 78m36.690s
As you can see using two extra machines I was able to reduce the time
to around 56% of original. Stages 1 and 2 are trivial to run
remotely in parallel and they speed up very nicely. And the
processes while they are running don't blow up to huge sizes, so you
don't need a machine with a huge amount of main memory to run them.
Stages 3-5 are a different story though. For my test stitch stage 3
enblend gets run 3 times to create three different exposure images.
Each exposure image is made up of a mutually exclusive set of ldr
images from stage 2. Looking at this problem you'd think you could
just run them in parallel and be done with it, but you can't. Since
the final image is big, the images produced in this stage is big
too. When running one of these enblends on machine B the process got
up to 12G in size, so much more then the main memory of machine B, so
started swapping. But I could call enblend without the -a option and
I could run it on machine B, because then enblend only got up to
about 5-6G in size. If I ran all three serially on machine A using
-a it ok 24m37.791s. If I ran two serially on machine A using -a and
ran one on machine B without -a, it took 24m5.534s. Not a huge
amount of speed up, the -a makes enblend run faster, if you can
use it. And if I was doing a 750MPixel stitch, I'm sure the run on
machine B would taken even longer.
What I've determined is that as I stitch larger and larger, it's
always going to take more time, it's not possible to keep the time
constant while increasing the final image size by throwing more
machines at the problem. The reason being while with stages 1 & 2
more machines will always decrease the time, stages 3-5 require a
machine with an increase in main memory to correspond to an increase
in final stitch size. For a real increase in speed with enblend it'd
need to be designed so that the problem it's solving can be broken
into pieces that don't require all the information of the entire
image. That way when those pieces are worked on by remote machines
it wouldn't require all the remote machines to have a large amount of
main memory. But I don't know if enblend could even been designed
that way.
Bruno Postle
> For a real increase in speed with enblend it'd need to be designed
> so that the problem it's solving can be broken into pieces that
> don't require all the information of the entire image. That way
> when those pieces are worked on by remote machines it wouldn't
> require all the remote machines to have a large amount of main
> memory.
Thanks for this, yes the bottleneck to distributed stitching is that
enblend has to assemble the final image and this requires a lot of
memory.
One experiment we tried a long time ago was to split the enblend job
up, i.e. instead of blending images 0,1,2,3 in one go, you can blend
0,1 and 2,3 as separate processes and then blend these together as a
third process.
The disadvantage of this is there is more IO and processing, and the
final blend still requires lots of memory. The advantage is that
due to the way Hugin uses 'cropped TIFF' intermediate files, the
other blends use less memory and some of the processing can be
distributed.
--
Bruno
Jeffrey Martin
paul womack
> On Sun 16-Jan-2011 at 06:10 -0800, kevin wrote:
>
>> For a real increase in speed with enblend it'd need to be designed so
>> that the problem it's solving can be broken into pieces that don't
>> require all the information of the entire image. That way when those
>> pieces are worked on by remote machines it wouldn't require all the
>> remote machines to have a large amount of main memory.
In any case, if it is required to send all images to all remote machines,
this may (of itself) cause an I/O bottle neck.
BugBear
kfj
On 16 Jan., 23:25, Bruno Postle <br...@postle.net> wrote:
> Thanks for this, yes the bottleneck to distributed stitching is that
> enblend has to assemble the final image and this requires a lot of
> memory.
>
> One experiment we tried a long time ago was to split the enblend job
> up, i.e. instead of blending images 0,1,2,3 in one go, you can blend
> 0,1 and 2,3 as separate processes and then blend these together as a
> third process.
images are blended at different resolutions, so that low frequencies
are treated differently to high frequencies. This would imply that the
parts of the blending process which work at high resolution (high
frequencies) don't need so much lateral context and could be parcelled
off to be blended somewhere else, whereas the blending at lower
frequencies could be done at reduced resolution locally, needing much
less processing power. Splitting the images up with something like a
DWT and then treating the different levels individually might be an
option.
Kay
kevin
have all the images, then it's a losing battle. But if there is some
way to break the problem up so that a remote machine is only working
on a certain area of the final image and only needs the information
just in that area, that might speed things up. I'll have to give a
shot at what Bruno said, that the blending can be broken into stages
and see if that makes any difference.
paul womack
In the case of a Giga-Pixel matrix/grid style pano, it's quite simple.
But some fish-eye based panos are much less simple.
Even a spherical made from small shots (e.g. 35mm FOV)
is quite complex, since the shots don't for a uniform grid.
BugBear
Bruno Postle
>
> Correct, if it's going to always require all the remote machines
> to have all the images, then it's a losing battle.
This isn't necessarily the case with 'make' (or distmake), the
necessary files only need to be available via NFS or whatever
filesystem you are using, they won't be moved over the network
unless they are actually used.
--
Bruno
michael.grant
the advantage of make here. Especially since if you stop the job at
this point, you have to start the emblend all over again.
Distributing the other pieces is a small job in comparison to the
emblend which seems to take the most time.
So it seems that emblend is keeping the entire image in memory. What
if you broke up the job of emblend so that it output it's pieces to
intermediate tiles files which were then assembled into the final tiff
after? Then maybe you could distribute the non-overlapping tiles and
and input images to another machine. You kind of "lock" those tiles
until that section is done. The idea is to keep in memory just the
section you are working on, even it it's like 100meg or so, that's not
too large.
Unless you do something like this, I can't see how hugin is going to
be able to create a pano any greater than the available swap space.
How difficult would it be to rework emblend to do something like the
above?
Michael
Jeffrey Martin
I still use smartblend regularly, it's 5 years old, single core, 32-bit.... a dinosaur. but still superior in terms of blending to anything else out there, in my opinion.
Blending is the most expensive part of stitching.... PTStitcherNG is great, but the blending is not very "smart".
I heard Prof. Dersch will continue development of PTStitcherNG this winter, apparently (by the way)
Jeffrey
Jimmy
as an alternative to SGE (we use slurm quite a bit in work for our
compute clusters) and to top it off, there is a patch to gnumake which
does an "srun" (under an a slurm allocation) instead of a fork and
sh .. if would be quite easy to taskfarm out the jobs if hugin could
dump out a makefile with a bunch of commands to run for the stitch
jimmy.
Pablo d'Angelo
Am 22.01.2011 12:20, schrieb michael.grant:
> Unless you do something like this, I can't see how hugin is going to
> be able to create a pano any greater than the available swap space.
> How difficult would it be to rework emblend to do something like the
> above?
Enblend can use temporary files (if compiled without OpenMP), and then
can process very large images.
A good option for clustering would be to split the panorama in many
smaller parts (tiles, using the crop feature of hugin), and distribute
these. There might be some artefacts near the boundaries, so rendering
overlapping tiles is probably better. Then these tiles could be cropped
and easily assembled together into a big output file (if one output file
is required at all...).
ciao
Pablo
Bruno Postle
>
> A good option for clustering would be to split the panorama in
> many smaller parts (tiles, using the crop feature of hugin), and
> distribute these.
gigatile is an experimental tool for doing this, it splits the Hugin
.pto project into 4096x4096 regions for separate rendering and
manages the remapping and blending with a Makefile. This only uses
a small amount of memory whatever the panorama size and should work
well with distributed processing:
http://search.cpan.org/dist/Panotools-Script/bin/gigatile
> There might be some artefacts near the boundaries, so rendering
> overlapping tiles is probably better.
gigatile doesn't do this overlapping, I didn't have a need for it,
so you do get these artefacts at the tile boundaries which may or
may not be a problem depending on your subject.
--
Bruno
Rogier Wolff
On Sun, Jan 23, 2011 at 07:29:28AM +0100, Pablo d'Angelo wrote:
> Am 22.01.2011 12:20, schrieb michael.grant:
>
> >Unless you do something like this, I can't see how hugin is going to
> >be able to create a pano any greater than the available swap space.
> >How difficult would it be to rework emblend to do something like the
> >above?
>
> Enblend can use temporary files (if compiled without OpenMP), and then
> can process very large images.
I'm not entirely sure that the imagecache code is bugfree. There were
a lot of bugs reported with crashing and/or image corruption when the
image cache code was enabled. IIRC I only got my last large project
stitched by having a shitload of RAM and disabling the imagecache....
Roger.
--
** R.E....@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
** Delftechpark 26 2628 XH Delft, The Netherlands. KVK: 27239233 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement.
Does it sit on the couch all day? Is it unemployed? Please be specific!
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ
paul womack
> By needing the emblend to be done all in one shot like this you loose
> the advantage of make here. Especially since if you stop the job at
> this point, you have to start the emblend all over again.
> Distributing the other pieces is a small job in comparison to the
> emblend which seems to take the most time.
>
> So it seems that emblend is keeping the entire image in memory. What
> if you broke up the job of emblend so that it output it's pieces to
> intermediate tiles files which were then assembled into the final tiff
> after? Then maybe you could distribute the non-overlapping tiles and
> and input images to another machine.
enblend has this, but only "internally", via the -a flag.
BugBear