Mono on PS3 Linux

[stejin]

Hi Rasmus

Thank you for this project.

I have got a recent PS3 running Gentoo (kernel 2.6.24-ps3) and would
like to experiment with the Cell BE. Since .NET/Mono is my preferred
development platform I am very excited to try out CellDotNet.
Unfortunately, I am having difficulties getting Mono to build/run on
my system and I was wondering if you could give me a few details about
your PS3 Linux+Mono installation.

I am facing following problems:
On Gentoo/64bit userland Mono does not compile because some of the
exception handling code references stuff that is not defined or
defined differently in the 64bit version of ucontext.h. The errors I
got are basically identical to what is described in this bug report
<https://bugzilla.novell.com/show_bug.cgi?id=MONO58453> although that
seems to come from ppc32 and may have been fixed in the meantime (for
ppc32).
I made a few simple changes to mini-ppc.h to get exceptions-ppc
compiled. This worked well until I got to the part in the build
process when C# code had to be compiled to IL for the first time. I
think the C# compiler itself is written in C# so it is probably when
IL is JIT-compiled for the first time. The errors I get are
segmentation faults (bus errors). At the time the error occurs the
mono runtime has been built and mcs.exe and the mono-wrapper also
exist in my build directory. I get the same segmentation faults
whenever I try to use any of those, for example if I try to run mono
mcs.exe ... or mono test.exe (where test.exe has been built by mono on
an x86 system).
On Gentoo/32bit userland I did not have to make any changes to source
files (32bit ucontext.h contains what Mono expects) but again I get
segmentation faults and the build process stops at exactly the same
place (only difference being that the system usually freezes requiring
a hard reboot).

I have not found any statement on www.mono-project.com confirming that
Mono does support ppc64. In fact, yours is the only confirmed
installation of Mono on ppc64 Linux I found on the internet. I am
pretty sure that Mono works with ppc32, given this site:
http://www.mono-project.com/Mono:PowerPC (note that it explicitly
refers to the ppc32 ABI). However, I thought that programs were pretty
much binary compatible and did not expect these problems.

Sorry for the long introduction - just wanted to let you know where I
am coming from. Anyway, here are my questions:
Did you experience any problems when installing/running Mono on PS3?
How did you install (e.g. using an RPM)? Did you have to modify Mono
in any way to build/run it? Which Linux distros are you using
successfully on PS3 with Mono - you mentioned Red Hat and YDL, any
others?

Finally, could you post the content of /usr/include/sys/ucontext.h
from one of your PS3 Linux systems that has Mono installed?

Thank you very much,
Steffen

[Rasmus Halland]

Hi Stejin!

On Tue, Jul 22, 2008 at 7:03 PM, stejin <steffen...@web.de> wrote:
>
> Hi Rasmus
>
> Thank you for this project.

You're welcome ;).

I haven't had issues during install which required me to modify the
source - as far as I remember, the only problem was figuring out the
correct platform arguments to the build configuration script.

>
> I have not found any statement on www.mono-project.com confirming that
> Mono does support ppc64. In fact, yours is the only confirmed
> installation of Mono on ppc64 Linux I found on the internet. I am
> pretty sure that Mono works with ppc32, given this site:
> http://www.mono-project.com/Mono:PowerPC (note that it explicitly
> refers to the ppc32 ABI). However, I thought that programs were pretty
> much binary compatible and did not expect these problems.

I compiled mono as for ppc32, which seems to be the supported ppc variant.

>
> Sorry for the long introduction - just wanted to let you know where I
> am coming from. Anyway, here are my questions:
> Did you experience any problems when installing/running Mono on PS3?
> How did you install (e.g. using an RPM)? Did you have to modify Mono
> in any way to build/run it? Which Linux distros are you using
> successfully on PS3 with Mono - you mentioned Red Hat and YDL, any
> others?

I installed from a mono 1.9 tarball without modifying it, and FC and
YDL are the only ones I've tried with - both with success.

>
> Finally, could you post the content of /usr/include/sys/ucontext.h
> from one of your PS3 Linux systems that has Mono installed?

Sure, the attached ucontext.h is from my ps3 with mono 1.9 and the
following uname output:
2.6.16-20061110.ydl.2ps3 #1 SMP Tue Dec 5 12:35:42 EST 2006 ppc64
ppc64 ppc64 GNU/Linux

Let me know how it goes.

/rasmus

[stejin]

Hi Rasmus

Thanks for getting back so soon.
Your ucontext.h is identical to what I have got on my system.
Is there any way you could send me the Mono JIT compiler (mono) from
your PS3? I would like to have look inside.
Sorry for pestering you with this. I will not ask for anything else.
Thank you,
Steffen

[Rasmus Halland]

Hey Stejin

I don't mind sending you my mono installation, including source, but
it's rather large - around 185 MB as tar.gz. I think I've got a mail
server which will send it; do you think yours will receive it?

[stejin]

Hi Rasmus

Thanks for the offer. I got the JIT compiler to work over the weekend
and can compile and run C# code. I still get segmentation faults when
trying to compile mcs but should be able to resolve in the course of
this week.

Again, thanks for your help.
Steffen

[Koichi Senada]

Steffen, describe your success story when it's done, please?
:)

[stejin]

I managed to compile and install Mono 1.9.1 and Monodevelop on my PS3
running Gentoo.

Here is what I did:

- Downgraded to kernel 2.6.23 (probably not necessary and I have
upgraded to 2.6.24 in the meantime)
- Switched to 32bit userland (it could be possible on 64bit userland
since I was able to build the runtime but have not tried)
- Applied Gentoo specific patches to source (part of Gentoo Mono 1.9.1
ebuild)

After this I was able to build a working mono runtime.

Compiling .NET assemblies (MCS directory) still failed with
segmentation faults. I managed to work around this by "injecting" pre-
compiled assemblies into the build directories.
Whenever the build process failed, I copied the pre-compiled assembly
into the target directory and execute 'make' again. Apparently, the
affected assembly was not skipped but compiled the second time.
I will attach a script which shows all assemblies I had to treat this
way.
I obtained the pre-compiled files from one of the 1.9.1 RPM's
available at http://www.mono-project.com. I tried to get them from
http://mono.ximian.com/daily/ (monocharge) but those assemblies were
compiled against a newer version of mscorlib.dll.

After all assemblies were compiled, I executed 'make install' and
registered mono in Portage. Monodevelop installed ok from Portage. I
only had to manually build GTK-sharp due to a known bug. I have been
testing Mono and Monodevelop for some time and did not have any
further problems.

[stejin]

Added "precompiled.txt" to Files.

[Rasmus Halland]

That's great, Stejin! Let us know how it goes with celldotnet.

[stejin]

I managed to compile and run CellDotNet + SciMarkCell on my PS3 after
some initial difficulties.

I first compiled CellDotNet with the Microsoft C# compiler (via Visual
Studio 2005) and executed on PS3/Mono. This worked fine. SciMarkCell
also compiled against CellDotNet.exe and I was able to run the
benchmarks.

CellDotNet failed to compile on PS3/Mono (via Monodevelop). I got
following errors:

--------------------------------------------------------------------------------------------------------------------------------------

ILOpCodeExecutionTest.cs
private static void TestExecution<T>(ILWriter ilcode, T expectedValue)
where T : struct

Building Project: CellDotNet (Debug|Any CPU)
Performing main compilation...
Exception caught by the compiler while emitting:
Block that caused the problem begin at: /home/mono/CellDotNet/
ILOpCodeExecutionTest.cs(894,17):
System.Exception: method not found
Internal compiler error at /home/mono/CellDotNet/
ILOpCodeExecutionTest.cs(893,37):: exception caught while emitting
MethodBuilder [ILOpCodeExecutionTest::TestExecution]

/home/mono/CellDotNet/Class1.cs(55,30): warning CS0219: The variable
`b' is assigned but its value is never used
/home/mono/CellDotNet/Class1.cs(53,31): warning CS0219: The variable
`i16' is assigned but its value is never used

Unhandled Exception: System.Exception: method not found
at System.Reflection.Emit.TypeBuilder.GetMethod (System.Type
instanciated, System.Reflection.MethodInfo meth) [0x00000]
at Mono.CSharp.AnonymousMethodBody.Emit (Mono.CSharp.EmitContext ec)
[0x00000]
at Mono.CSharp.Argument.Emit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Invocation.EmitArguments (Mono.CSharp.EmitContext ec,
System.Collections.ArrayList arguments, Boolean dup_args,
Mono.CSharp.LocalTemporary this_arg) [0x00000]
at Mono.CSharp.Invocation.EmitCall (Mono.CSharp.EmitContext ec,
Boolean is_base, Mono.CSharp.Expression instance_expr,
System.Reflection.MethodBase method, System.Collections.ArrayList
Arguments, Location loc, Boolean dup_args, Boolean omit_args)
[0x00000]
at Mono.CSharp.Invocation.EmitCall (Mono.CSharp.EmitContext ec,
Boolean is_base, Mono.CSharp.Expression instance_expr,
System.Reflection.MethodBase method, System.Collections.ArrayList
Arguments, Location loc) [0x00000]
at Mono.CSharp.MethodGroupExpr.EmitCall (Mono.CSharp.EmitContext ec,
System.Collections.ArrayList arguments) [0x00000]
at Mono.CSharp.Invocation.Emit (Mono.CSharp.EmitContext ec)
[0x00000]
at Mono.CSharp.Invocation.EmitStatement (Mono.CSharp.EmitContext ec)
[0x00000]
at Mono.CSharp.StatementExpression.DoEmit (Mono.CSharp.EmitContext
ec) [0x00000]
at Mono.CSharp.Statement.Emit (Mono.CSharp.EmitContext ec)
[0x00000]
at Mono.CSharp.Block.DoEmit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.Emit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.ExplicitBlock.Emit (Mono.CSharp.EmitContext ec)
[0x00000]
at Mono.CSharp.Block.DoEmit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.Emit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.DoEmit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.Emit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.DoEmit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.Emit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.DoEmit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.Emit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.DoEmit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.Emit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.DoEmit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.Emit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.DoEmit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.Emit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.DoEmit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.Block.Emit (Mono.CSharp.EmitContext ec) [0x00000]
at Mono.CSharp.ExplicitBlock.Emit (Mono.CSharp.EmitContext ec)
[0x00000]
at Mono.CSharp.ToplevelBlock.Emit (Mono.CSharp.EmitContext ec)
[0x00000]
at Mono.CSharp.EmitContext.EmitResolvedTopBlock
(Mono.CSharp.ToplevelBlock block, Boolean unreachable) [0x00000]
at Mono.CSharp.EmitContext.EmitTopBlock (IMethodData md,
Mono.CSharp.ToplevelBlock block) [0x00000]


Build complete -- 1 error, 2 warnings

---------------------- Done ----------------------


I did not worry too much about the test cases so I commented out the
faulting method or excluded ILOpCodeExecutionTest


---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Next I got


CellDotNet/Spe/BitVectorTest.cs(67,28): error CS0121: The call is
ambiguous between the following methods or properties:
`CellDotNet.Spe.BitVector.AddAll(System.Collections.Generic.IEnumerable<int>)'
and
`CellDotNet.Spe.BitVector.AddAll(System.Collections.Generic.IEnumerable<uint>)'

CellDotNet/Spe/BitVectorTest.cs(110,28): error CS0121: The call is
ambiguous between the following methods or properties:
`CellDotNet.Spe.BitVector.RemoveAll(System.Collections.Generic.IEnumerable<int>)'
and
`CellDotNet.Spe.BitVector.RemoveAll(System.Collections.Generic.IEnumerable<uint>)'

CellDotNet/Spe/BitVectorTest.cs(124,28): error CS0121: The call is
ambiguous between the following methods or properties:
`CellDotNet.Spe.BitVector.RemoveAll(System.Collections.Generic.IEnumerable<int>)'
and
`CellDotNet.Spe.BitVector.RemoveAll(System.Collections.Generic.IEnumerable<uint>)'

Here are the calls in BitVectorTest

[Test]

public void TestBitVector_AddAllIEnumerable()

{

BitVector b1 = new BitVector();


uint[] array = new uint[] { 7, 9, 333 };



b1.AddAll(array);



IsFalse(b1.IsCountZero() || b1.Count != 3 || !b1.Contains(7) || !
b1.Contains(9) || !b1.Contains(333));

}




[Test]

public void TestBitVector_RemoveAllIEnumerableInt()

{

BitVector b1 = new BitVector();

b1.Add(7);

b1.Add(9);

b1.Add(333);



int[] array = new int[] { 7, 333 };

b1.RemoveAll(array);



IsFalse(b1.IsCountZero() || b1.Count != 1 || b1.Contains(7) || !
b1.Contains(9) || b1.Contains(333));

}


[Test]

public void TestBitVector_RemoveAllIEnumerableUint()

{

BitVector b1 = new BitVector();

b1.Add(7);

b1.Add(9);

b1.Add(333);


uint[] array = new uint[] { 7, 333 };

b1.RemoveAll(array);



IsFalse(b1.IsCountZero() || b1.Count != 1 || b1.Contains(7) || !
b1.Contains(9) || b1.Contains(333));

}

I worked around this problem by defining new AddAll and RemoveAll
methods in BitVector with unique names (e.g. AddAlluint) and fixing
the references in BitVectorTest.


-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Finally, mcs (or gmcs) produced an unhandled exception during
compilation. Apparently, it had difficulties verifying the enum types
defined in CellDotNet.


Unhandled Exception: System.ArgumentOutOfRangeException: Argument is
out of range.
Parameter name: 0
at Mono.CSharp.Enum.GetDefinition (System.Object value) [0x00000]
at Mono.CSharp.TypeManager.CSharpEnumValue (System.Type t,
System.Object value) [0x00000]
at Mono.CSharp.EnumConstant.AsString () [0x00000]
at Mono.CSharp.TypeContainer.VerifyMembers () [0x00000]
at Mono.CSharp.ClassOrStruct.VerifyMembers () [0x00000]
at Mono.CSharp.RootContext.EmitCode () [0x00000]
at Mono.CSharp.Driver.Compile () [0x00000]
at Mono.CSharp.Driver.Main (System.String[] args) [0x00000]

This took a while and after reviewing the mcs source code, I found
that this error can be avoided by reducing the compiler warning level
from 4 to 3 or lower.


This was the last hurdle. I am now able to compile both CellDotNet and
SciMarkCell on my PS3 using Monodevelop.

Next, I will study the API and start experimenting. I want to use it
for pricing financial derivative instruments mainly involving Monte
Carlo and binomial trees.


Could you share your views about Cell BE performance based on your
research? What are the results of your experiments with benchmarks?
Maybe you could upload your thesis/research paper?
If you have time, please also post SciMarkCell stats from your test
systems.

Cheers,
Steffen

[Rasmus Halland]

On Wed, Aug 13, 2008 at 7:43 PM, stejin <steffen...@web.de> wrote:
>
> I managed to compile and run CellDotNet + SciMarkCell on my PS3 after
> some initial difficulties.
>
> I first compiled CellDotNet with the Microsoft C# compiler (via Visual
> Studio 2005) and executed on PS3/Mono. This worked fine. SciMarkCell
> also compiled against CellDotNet.exe and I was able to run the
> benchmarks.

Cool.

I haven't tried to use mono/mcs to compile, but I'm sure that they
would appreciate a bug report :-).

>
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> Next I got
>
>
> CellDotNet/Spe/BitVectorTest.cs(67,28): error CS0121: The call is
> ambiguous between the following methods or properties:
> `CellDotNet.Spe.BitVector.AddAll(System.Collections.Generic.IEnumerable<int>)'
> and
> `CellDotNet.Spe.BitVector.AddAll(System.Collections.Generic.IEnumerable<uint>)'
>
> CellDotNet/Spe/BitVectorTest.cs(110,28): error CS0121: The call is
> ambiguous between the following methods or properties:
> `CellDotNet.Spe.BitVector.RemoveAll(System.Collections.Generic.IEnumerable<int>)'
> and
> `CellDotNet.Spe.BitVector.RemoveAll(System.Collections.Generic.IEnumerable<uint>)'
>
> CellDotNet/Spe/BitVectorTest.cs(124,28): error CS0121: The call is
> ambiguous between the following methods or properties:
> `CellDotNet.Spe.BitVector.RemoveAll(System.Collections.Generic.IEnumerable<int>)'
> and
> `CellDotNet.Spe.BitVector.RemoveAll(System.Collections.Generic.IEnumerable<uint>)'
>
> Here are the calls in BitVectorTest
>
> [Test]
>
> public void TestBitVector_AddAllIEnumerable()
>
> {
>
> BitVector b1 = new BitVector();
>
>
> uint[] array = new uint[] { 7, 9, 333 };
>
>
>
> b1.AddAll(array);

Seems like mcs thinks uint[] can be implicitly cast as
IEnumerable<int>, which doesn't sound right to me...

Yet another mcs bug report candidate...

>
>
> This was the last hurdle. I am now able to compile both CellDotNet and
> SciMarkCell on my PS3 using Monodevelop.

Great!

>
> Next, I will study the API and start experimenting. I want to use it
> for pricing financial derivative instruments mainly involving Monte
> Carlo and binomial trees.
>
>
> Could you share your views about Cell BE performance based on your
> research?

Sure. First of all it's important to be aware that the compiler is not
all that mature in terms of robustness and optimizations, so you need
to be prepared to use some time improving it and fixing problems.

There are articles out there about cell performance which goes into
great depth, but here is the short version of what we found for the
benchmarks that we did. Vectorization is in general probably the most
important optimization that the programmer can perform. For the
compiler:
- Instruction scheduling in order to increase dual issue and decrease
stalls is important. Unfortunately we never did get that really
working, but there is some code in there to do it, and the results
were promising.
- Lack of branch prediction takes its toll, but we never really
attacked that issue except by providing intrinsic methods like
SpuMath.CompareGreaterThanAndSelect which can be used to avoid some
small branches.
- There are other aspects of the compiler wich really could use some
work, it all depends on how much you want to dig in...

For inspection and debugging of the generated code, debugging.html has
some instruction. (the part about blob functions is outdated,
though...)

Of course, if you are bandwidth bound, these optimizations might not
make so much difference...

> What are the results of your experiments with benchmarks?

A couple of highlights:
- Vectorization makes a huge difference.
- Monte carlo PI benchmark on cell significantly outperforms the
.net/mono versions, while the other benchmarks are not so favorable,
but also not that bad.
- The compiler still needs love :-)

> Maybe you could upload your thesis/research paper?
> If you have time, please also post SciMarkCell stats from your test
> systems.

The thesis is in danish, but I've put it in the downloads section
(http://code.google.com/p/celldotnet/downloads/list) along with the
abstract. The performance figures/tables near the end hopefully makes
some sense, even if it's in danish. Here are a few works which might
come handy:

Løkkeudrulning = loop unrolling
Stall pga. data afhængighed = stalls caused by data dependency


If you feel like working on the compiler, let me know, because then we
should probably coordinate the efforts...


rasmus


>
> Cheers,
> Steffen
>
>
>
>
>
>
>
> >
>

[stejin]

That's brilliant. Thanks!

[stejin]

Hi Rasmus

I am considering to log bug reports as you suggested and prepared 2
sets of CellDotNet source files on an x86 OpenSUSE virtual machine to
reproduce the compiler errors. Not sure if they have a PS3 for
testing. Do you mind if I attach the source files to the bugreport?
That way they only have to download and compile to reproduce the
errors.

Kind regards,
Steffen

[Rasmus Halland]

I don't mind at all, on the contrary. But I think the chances of
getting the bugs fixed will go up if you prepare a small test case,
rather than send in the entire CND source :-).