Poor python and/or Zope performance on Sparc
joa2212
I'm posting this message because I'm quiet frustrated.
We just bought a software from a small software vendor. In the
beginning he hosted our application on a small server at his office. I
think it was a Fujitsu-Siemens x86 running debian Linux. The
performance of the DSL-Line was very poor, so we decided to buy an own
machine to host the application ourselves.
The application is based on the Zope Application server (2.8.8-final
w/ python 2.3.6) along with many other packages like ghostview,
postgres, freetype, python imaging lib, etc... I once saw the
application running at the office of my software vendor and it was
running very well.
So I thought that - when the bottleneck of poor DSL performance has
disappeared - the software will run as fast as at his office. But I
erred. The performance is more than poor.
We have a Sun T1000 with 8 cores and 8 GB of RAM. First, I installed
Solaris 10 because I know this OS better than Debian Linux. Result:
poor Performance. After that I decided to migrate to debian:
root@carmanager > uname -a
Linux carmanager 2.6.18-5-sparc64-smp #1 SMP Wed Oct 3 04:16:38 UTC
2007 sparc64 GNU/Linux
Result: Almost even worse. The application is not scaling at all.
Every time you start a request it is hanging around on one cpu and is
devouring it at about 90-100% load. The other 31 CPUs which are shown
in "mpstat" are bored at 0% load.
If anybody needs further information about installed packages, I'll
post it here.
Any hints are appreciated!
Thanks, Joe.
PS: Fortunately the Sun is not bought yet. It's a "try&buy" from my
local dealer. So if there are any hints like "buy a new machine
because Sun is crap" I will _not_ refuse obediance.
George Sakkis
> Result: Almost even worse. The application is not scaling at all.
> Every time you start a request it is hanging around on one cpu and is
> devouring it at about 90-100% load. The other 31 CPUs which are shown
> in "mpstat" are bored at 0% load.
You are probably not aware of Python's Global Interpeter Lock:
http://docs.python.org/api/threads.html.
George
joa2212
Hi George,
yes, that's right. I wasn't aware of this. If I understand you
correctly then we have a problem of implementing threads in the
software we are using. But tell me one thing: Why does this software
almost runs like a fool on an Intel machine with a single cpu (perhaps
dual core?) and slow like a snail when it runs on Sparc? It's exactly
the same source code on both platforms. Certainly all relevant
packages (python + zope) were recompiled on the sparc.
Sorry for my questions, I'm really no software developer. I'm just a
little bit helpless because my software vendor can't tell my anything
about by concerns.
Joe.
Jean-Paul Calderone
The Sun machine you have has divided its hardware resources into 32 units.
That means that, roughly, each unit is only 1/32nd as fast as the overall
system. All of the units together are, roughly, about the speed of the
Intel machine you were using before, so one unit alone is 32 times slower.
Since Zope is only using 1 unit, it runs 32 times slower, and 31 out of 32
hardware units sit idle.
The T1000 isn't a very good machine for general server purposes. It has
advantages when running software with a lot of hardware-level parallelism,
but Zope isn't such a piece of software.
Jean-Paul
Duncan Booth
> The T1000 isn't a very good machine for general server purposes. It
> has advantages when running software with a lot of hardware-level
> parallelism, but Zope isn't such a piece of software.
Zope can scale well on multi-processor machines, but you have to configure
it correctly. You need to run multiple instances of Zope all talking to a
common Zeo backend.
In this case you could try running up to 32 single-threaded zope instances,
although you'll need to watch the memory consumption so depending on the
application you might not get away with that many.
Ivan Voras
> We have a Sun T1000 with 8 cores and 8 GB of RAM. First, I installed
> Solaris 10 because I know this OS better than Debian Linux. Result:
> poor Performance. After that I decided to migrate to debian:
Do you know the architecture of this machine? It's extremely streamlined
for data throughput (IO) at the expense of computational ability. In
particular:
- it runs at a relatively small clock speed (1 GHz - 1.4 GHz)
- it's terrible for floating-point calculations because there is only
one FPU shared by all 32 logical processors
While it will be screamingly fast for serving static content, and pretty
decent for light database jobs, this not an optimal platform for dynamic
web applications, especially for Python since the language's so dynamic
and it doesn't support SMP. Since it's so optimized for a certain
purpose, you can freely consider it a special-purpose machine, rather
than a general-purpose one.
Even if you manage to get Zope to spawn parallel request handlers
(probably via something like fastcgi), if the web application is
CPU-intensive you won't be happy with its performance (for CPU-intensive
tasks you probably don't want to spawn more than 8 handlers, since
that's the number of physical cores in the CPU).
python...@gmail.com
> Result: Almost even worse. The application is not scaling at all.
> Every time you start a request it is hanging around on one cpu and is
> devouring it at about 90-100% load. The other 31 CPUs which are shown
> in "mpstat" are bored at 0% load.
>
Like others already pointed out: Performance is going to suffer if
you
run something that's single-threaded on a T1000.
We've been using the T1000 for a while (but not for python) and it's
actually really fast. The performance is way better than the 8 cores
imply. In fact, it'll do work that's not too far from 32 times the
work
of a single thread.
Of course, it all depends on what your application is like, but try
experimenting a bit with parallelism. Try changing from 4 up to 64
parallel
working processes to see what kind of effects you'll get.
I'm guessing that whatever caching of persistent data you can do will
be a good thing as well as the disk might end up working really hard.