"Zeus" 64 core / 512Gb ram machine

[Jasper Wallace]

Hi,

I've started a wiki page with some ideas for what we would use Zeus for,
and how it would be used:

http://wiki.london.hackspace.org.uk/view/Zeus

It's unlikely that this machine will be up and running before we have
finished ebaying the majority of the stuff we've got at the moment.

--
[http://pointless.net/] [0x2ECA0975]

[David Murphy]

It's amusing that this has more RAM than my old uni's Beowulf cluster.

anything planned for it? translation projects? bioinformatics? wether prediction?

[Tim Reynolds]

All of these ideas are awesome, especially giving OSS projects access for scalability testing and the rendering. 

[Big Will]

It would be useful for learning big data processing

[Russ Garrett]

On 7 October 2012 21:09, Jasper Wallace <jas...@pointless.net> wrote:
> I've started a wiki page with some ideas for what we would use Zeus for,
> and how it would be used:
>
> http://wiki.london.hackspace.org.uk/view/Zeus

At the risk of seeming like I'm on a vendetta here: as I mentioned
before, I'm fine with having this in the space for the short term, and
playing around with it. However, I think it would be a tremendous
waste (both of space, and of potential funds) if we didn't eBay them
afterwards. I suspect it'll be tremendously under-utilised, and I
don't want to get into the business of billing people for power.

If anyone needs computing power like this, you can get a cc2.8xl
instance on EC2 for $2.40/hour. This is commodity stuff.

--
Russ Garrett
ru...@garrett.co.uk

[Steff]

On 7 October 2012 22:31, Big Will <william.g...@gmail.com> wrote:
> It would be useful for learning big data processing

Not really. The whole thrust of the "big data" thing is using large
numbers of normal-sized commodity machines to do things you'd
previously have needed the big iron for. The utility of a really large
machine is precisely that it allows one to use a large dataset without
the need for any fancy map-reduce footwork.

S

[Simon]

Thank you Jasper, this is great.
I personally would like to see this hardware used in the space, even if we cannot justify leaving it powered on all the time there are plenty of projects that could utilise this hardware.

One problem with using the SCA ports to connect these servers together into a single BIOS image is that the resulting hardware platform is not SMP but NUMA.  If the installed operating system just sees 64 CPU cores and 512 G of RAM then the likelihood is that performance will not scale as expected as the scheduler schedules processes to run on a CPU core that is far from the memory used.  If the OS understands the NUMA nature of the platform then it should scale pretty well and it would certainly be really interesting to see.


Other options for using this hardware include running 'Zeus' as 4 separate partitions of a single x3950.  With the SCA hardware physically connected it is possible to configure the BIOSes to any combination of partitions of the 4 physical hosts without physically connecting/disconnecting anything.


I think that for running VMs or using as a render farm then 4 single host nodes would work best.
I would love to be able to let the BSD hackers loose on the hardware for them to test their scalability and I think it would be good for LHS to be associated with such efforts.


BTW - I have this weekend removed these 4 x3950 servers from the space to safe storage in my cellar in Clapton.  There are still 2 * x3850 and 2 * x3950 in the space.
I think that once the flack has calmed down from this hardware donation and a number of the other servers have been successfully auctioned on eBay then I can return them if there is enough interest and consensus among LHS members


Simon

[Jasper Wallace]

On Sun, 7 Oct 2012, Simon wrote:

> Thank you Jasper, this is great.
>
> I personally would like to see this hardware used in the space, even if
> we cannot justify leaving it powered on all the time there are plenty of
> projects that could utilise this hardware.
>
> One problem with using the SCA ports to connect these servers together
> into a single BIOS image is that the resulting hardware platform is not
> SMP but NUMA.  If the installed operating system just sees 64 CPU cores
> and 512 G of RAM then the likelihood is that performance will not scale
> as expected as the scheduler schedules processes to run on a CPU core
> that is far from the memory used.  If the OS understands the NUMA nature
> of the platform then it should scale pretty well and it would certainly
> be really interesting to see.

Oh yes, that reminds me, i know the guy that did this:

http://anil.recoil.org/talks/fosdem-io-2012.pdf

so he might want to play with it for his reserch.

> Other options for using this hardware include running 'Zeus' as 4
> separate partitions of a single x3950.  With the SCA hardware physically
> connected it is possible to configure the BIOSes to any combination of
> partitions of the 4 physical hosts without physically
> connecting/disconnecting anything.
>
>
> I think that for running VMs or using as a render farm then 4 single
> host nodes would work best.
>
> I would love to be able to let the BSD hackers loose on the hardware for
> them to test their scalability and I think it would be good for LHS to
> be associated with such efforts.
>
>
> BTW - I have this weekend removed these 4 x3950 servers from the space
> to safe storage in my cellar in Clapton.  There are still 2 * x3850 and
> 2 * x3950 in the space.
>
> I think that once the flack has calmed down from this hardware donation
> and a number of the other servers have been successfully auctioned on
> eBay then I can return them if there is enough interest and consensus
> among LHS members
>
>
> Simon
>
>
>
>
>
> On Sunday, 7 October 2012 22:31:06 UTC+1, Big Will wrote:
> It would be useful for learning big data  processing
>
>
>

--
[http://pointless.net/] [0x2ECA0975]

[Tim Reynolds]

Renting compute time isn't as cool as cobbling together it ourselves to play with, even if just for the short term. I had a chance to build and play with an enormous Myrinet/Rocks powered cluster a few years ago and it was a brilliant learning experience. It isn't something you get a chance to do very often.

I totally understand the power/space issues and I think Simons idea of it happening after all the current server hubbub has died down is a good one.

[Russ Garrett]

On 7 October 2012 22:57, Simon <skl...@gmail.com> wrote:
> Thank you Jasper, this is great.
> I personally would like to see this hardware used in the space, even if we
> cannot justify leaving it powered on all the time there are plenty of
> projects that could utilise this hardware.

Apologies for being blunt, but: I don't see any.

What's currently on the wiki page is a list of cool ideas, but with no
real plans behind them. I have yet to see any concrete reasons why we
need this hardware in the space.

This hardware is likely worth thousands of pounds on ebay. We
desperately need more cash to help with the upcoming space move. I
don't see how it's in the best interests of the space to keep hold of
this hardware for the long term, depreciating, just because some
people think it's cool.

In the end, Hackspace is about providing physical facilities which
people can't get elsewhere. Computing power is not a physical
facility, and so I don't think we have any place providing it on a
significant scale.

--
Russ Garrett
ru...@garrett.co.uk

[Billy]

Later is good, so the arguments are avoided.

IIRC, when the Stratsys was going to be used, the charge for the
printing was going to include the cost of the plastic and the cost of
the power to run it.

Do the same. Put a time booking system in place, work out the power
requirements, then set it up, so when you book the machine for a run,
you pay for the power used.

The PSU's are a known capacity, so it's easy to calculate how much
juice they're using.

We already have a version of the power-monitoring software designed.
Sol had a display running off the smart meter output during the first
server auction.. I don't know how much work it'll be to adapt it to
this use.

Add in a link to IRC, where you can log how long it's been running/how
long the current run is to go.

Time booking via the wiki or a google-calender.

If the member's who want to use it for the first couple of runs set
this up and organise the maintenance schedule between them, then it'll
be another facility for the space, and another sub-group that can
teach other people how to do this.

It'll also be another selling point for new members that want access
to bigger iron. We didn't have access to this sort of hardware when i
was at uni.

On Oct 7, 11:06 pm, Tim Reynolds <t...@christwithfries.net> wrote:
> Renting compute time isn't as cool as cobbling together it ourselves to play with, even if just for the short term. I had a chance to build and play with an enormous Myrinet/Rocks powered cluster a few years ago and it was a brilliant learning experience. It isn't something you get a chance to do very often.
>
> I totally understand the power/space issues and I think Simons idea of it happening after all the current server hubbub has died down is a good one.
>

> On 7 Oct 2012, at 22:42, Russ Garrett <r...@garrett.co.uk> wrote:
>
> > On 7 October 2012 21:09, Jasper Wallace <jas...@pointless.net> wrote:
> >> I've started a wiki page with some ideas for what we would use Zeus for,
> >> and how it would be used:
>
> >>http://wiki.london.hackspace.org.uk/view/Zeus
>
> > At the risk of seeming like I'm on a vendetta here: as I mentioned
> > before, I'm fine with having this in the space for the short term, and
> > playing around with it. However, I think it would be a tremendous
> > waste (both of space, and of potential funds) if we didn't eBay them
> > afterwards. I suspect it'll be tremendously under-utilised, and I
> > don't want to get into the business of billing people for power.
>
> > If anyone needs computing power like this, you can get a cc2.8xl
> > instance on EC2 for $2.40/hour. This is commodity stuff.
>
> > --
> > Russ Garrett

> > r...@garrett.co.uk

[Bacon Zombie]

What about running a Doom multiplayer Server on it?

[SamLR]

Can someone clarify how big this is and where it would go? 

There are a couple of projects that sound interesting but I agree with Russ that this seems to be something that will be used once or twice and then forgotten. Billy's idea is nice but I don't think it will work. We already have a similar system with the laser cutter and last time I checked we were getting a lot less that 75% of the time on it being paid for. 

Beyond the 7 people on this list that have expressed any (not obviously dis)interest is there and real call for this? The BSD stuff sounds like it might be of interest but:

a) do we have any existing members who fall into this category 

b) do they actually want to try this?

A render farm would be cool but is there anyone who wants to:

a) set the bloody thing up

b) use it

c) teach others to use it

We have a lot of computers, we a reasonable amount of computing power (for what, I think, the majority of us want to do) and whilst as a one of "fire it up and poke it" exercise we should definitely do it, keeping it around seems to be a very large, expensive albatross. 

tl;dr set it up once for a weekend of big iron fun then, unless people can actually show that they (not some hypothetical 'others') have some serious projects that they are committed to and will carry out that really do need this sort of machine: sell it,

</2 uninformed cents>

S

[Billy]

On Oct 8, 1:28 am, SamLR <sam.lindenrat...@gmail.com> wrote:
> Can someone clarify how big this is and where it would go?
>
> There are a couple of projects that sound interesting but I agree with Russ
> that this seems to be something that will be used once or twice and then
> forgotten. Billy's idea is nice but I don't think it will work. We already
> have a similar system with the laser cutter and last time I checked we were
> getting a lot less that 75% of the time on it being paid for.
>

Storage, might be an issue, but it's already been mooted that you
could install it high up on brackets in one of the toilets.

If it only gets used, once or twice, sure that's an issue, but if the
people that are interested in using it, are the people setting it up,
then what's the problem, it's their time.

If it's out of the way, and only accessed via the network, then it's
only one network socket, and we're not using all of the ones we've
got.

Book in advance, and pay in advance. It'll also mean that anyone using
it would have to be more organised about what they're doing, which is
no bad thing.

> Beyond the 7 people on this list that have expressed any (not obviously
> dis)interest is there and real call for this? The BSD stuff sounds like it
> might be of interest but:
> a) do we have any existing members who fall into this category
> b) do they actually want to try this?
>
> A render farm would be cool but is there anyone who wants to:
> a) set the bloody thing up
> b) use it
> c) teach others to use it
>

As i said, if it's their time at the space that they're using to set
this up, and it's stashed out of the way, and the power is factored
into the costs, then why worry?

> We have a lot of computers, we a reasonable amount of computing power (for
> what, I think, the majority of us want to do) and whilst as a one of "fire
> it up and poke it" exercise we should definitely do it, keeping it around
> seems to be a very large, expensive albatross.
>

If the people who want to use it, are actually interested in using it,
then they'll take the time to set it up and convince other members
that it's a worthwhile project, just like buying the laser-cutter or
the 3-in-1 lathe.

> *tl;dr* set it up once for a weekend of big iron fun then, unless people
> can actually show that *they* (not some hypothetical 'others') have some
> serious projects that they are committed to *and will carry out* that

> really do need this sort of machine: sell it,
>

If they don't get it set up then sure, flog it on ebay.

I'll happily help with installing it on the toilet ceiling. It's dead
space at the moment. We've got spare shelf-brackets and nailing a
server to a toilet ceiling won't be one of the strangest installation
jobs i've done.

If someone who knows the network can plumb it in, the rest is just
setting up the power monitoring software. and documenting it on the
wiki.

Then the people who are interested can get on with it.

And if no-one is interested in a few months, then we can flog it on
ebay.

I don't know if the limit on the number of sales per month is still
active on the hackspace ebay account, but if we sell it last, then
that should be enough time for anyone interested to get on with it.

> </2 uninformed cents>
>
> S

>
> On 8 October 2012 00:27, Bacon Zombie <baconzom...@gmail.com> wrote:
>
> > What about running a Doom multiplayer Server on it?

[Morris]

This is awesome. Some fascinating ideas on the wiki, I would never in a million years thought of but I am excited by.

On 7 October 2012 21:09, Jasper Wallace <jas...@pointless.net> wrote:

--
>
++++++++++[>+>+++>++
+++++>++++++++++<<<<
-]>>>+++++++.>++++++
+++++.+++..---------
.++++++++++.<<+++.<.

[Clare Greenhalgh]

I think Russ's point that the Space is planning to move (which costs money) and it is worth a great deal (relatively speaking) should over-rule the points about how it could be used. We need to look at the space a a community, which has the best interests of the majority at heart. I have used the laser cutter twice (but know other people use it a lot, but don't pay for the time they are on it, which is disgraceful, and think it is a brilliant piece of equipment). 

Zeus would take up space (which is limited), especially while we are working at getting it installed. 

If people need to book a time to use it before hand I think it will be a pink elephant. 

We need to look at the big picture of the Space rather than seeing a pretty new shiny!

Hugs,

Noko

[SamLR]

I think Russ's point that the Space is planning to move (which costs money) and it is worth a great deal (relatively speaking) should over-rule the points about how it could be used. We need to look at the space a a community, which has the best interests of the majority at heart. I have used the laser cutter twice (but know other people use it a lot, but don't pay for the time they are on it, which is disgraceful, and think it is a brilliant piece of equipment). 

I think if people are genuninely going to use it and it's not going to be a drain on resources (i.e. space and money, which by if it can be stored nicely in the toilets and have its 'leccy paid for it won't be) then we should keep it. 

By your argument we should also sell the laser cutter & 3-in-1 as they're worth a fair bit. 

I'm more worried that it will get used once or twice and become a drain on resources (by depreciating whilst taking up space or by being used and not paid for). 

In the interest of fairness I would be very interested to see how much various other items cost in terms of electricity.  

[Clare Greenhalgh]

Surely my point was that this is not going to be used as much as the laser cutter and 3-in-1? This is not yet up and running so surely it is a different argument.

Ask the list how many people think they will genuinely use it. 

Noko

[David Murphy]

few people do big data projects which can't be handled on a modern gaming rig.

I certainly see the point. but it's worth a try for a couple of months. perhaps someone will do something really interesting.

[Jaimal Chohan]

As somebody previously mentioned, the cost of a 88 core/60gb EC2 instance is $2.40 (~£1.60) per hour.  So it;s not like the cost or availability of such a machine is prohibitive (unlike say access to a Laser Cutter or 3-in-one).  

Not having 'Zeus' would also means not having the maintenance/management effort required (I'd imagine with such a system there's going to be a fair amount of tweaking required on a project by project basis depending on which resources are being pushed)

Finally, I was under the impression that the idea of 'big data' projects, from a technology standpoint, is supposed to be that you can distribute workload (and reduce bottlenecks) for improved performance, as long as you can develop the project on a less able machine, you should be able to execute it in an EC2 environment.

[Benjamin Blundell]

I originally spoke with Simon and Jasper about this and their thoughts
were that it should stay and we should at least play with it. There
are some cool ideas and I think that is also what the space is about.
I agreed with them that we should at least plug it in, get some hard
figures and then decide.

That said, why re-invent the wheel? Billing for time, looking at power
requirements... EC2 has that covered. I initially wanted a render farm
setup but I quickly realised thats all covered with EC2 3DSMax and
Blender instances anyway.

I think that a lot of our problems will be solved by a move. Its a
subtle and elegant solution and if this raises some capital, then so
much the better. I now agree that yes, we'd be wasting our time.

That said, its muggins here who has to shift the bugger as Im likely
to be driving it to storage ;)

B

[Nick Leaton]

One thing that is interesting is that EC2 has some instances that come with 2 TB of SSD, as well as massive Ram. 

--
Nick

[samthetechie]

Very interested in the Internet simulation aspect and modelling scenarios of lots of networked computers. I am preparing some material with some international friends from #cryptoparty to make a case for this.

sam

[Jasper Wallace]

On Sun, 7 Oct 2012, Billy wrote:

>
>
> On Oct 8, 1:28 am, SamLR <sam.lindenrat...@gmail.com> wrote:
> > Can someone clarify how big this is and where it would go?
> >
> > There are a couple of projects that sound interesting but I agree with Russ
> > that this seems to be something that will be used once or twice and then
> > forgotten. Billy's idea is nice but I don't think it will work. We already
> > have a similar system with the laser cutter and last time I checked we were
> > getting a lot less that 75% of the time on it being paid for.
> >
>
> Storage, might be an issue, but it's already been mooted that you
> could install it high up on brackets in one of the toilets.

No, the machine that may go in one of the toilets is 'Borg', the babbage
replacement and vm server.

Zeus will be 4 4U machines, so a stack about 3 foot high. I think it would
be too heavy to wall mount safely, and without a rack to put them in the
only option is stacking them on the floor somewhere.



--
[http://pointless.net/] [0x2ECA0975]

[Matthew Copperwaite]

If it will only be used for short periods of time and not permanently
on then it could be that it has a home when it's off and set up
wherever there is space available when it is needed. Of course as long
as the 'off location' is not inconvenient and it is not left in the
'on location' for too long.

Matt

>
>
>
> --
> [http://pointless.net/] [0x2ECA0975]

[Anthony Evans]

If the machine is to go, I'd really like to see a small programme of demonstations/workshops/lectures based around the machine and its capabilities before it gets moved on.

I know very little about these machines. And a little educational event to learn about them would be fun.  

[David Schneider]

A few people seem to be disregarding the opportunity cost involved in running this server. It's not a case of paying £x/hr to cover electricity and thereby breaking even. The space is losing out on a few thousand pounds by not selling the server.

I haven't been around long but I'm interested to know how the laser cutter was purchased. I assume a pledge was involved? Why not set up a pledge to cover a reasonable % of the market value of Zeus? This will prove we have members who plan to use it. If the pledge should fail, the machine is sold to help cover move costs.

Dave

[Martin Dittus]

I very much agree with this.

Usual rules apply. Just because the cost of acquisition was incredibly low (free, I assume) doesn't mean people don't need to make a case if they intend to _keep_ it. Atm that case has not been made, based on the discussion here. This is exacerbated by the expected loss in sales, and somewhat by its expected maintenance cost and space usage. (I don't think floor space is a suitable long-term storage option.)

It's unfortunate because I don't want to curb anyone's enthusiasm. But I agree with Russ (and Ben and others) in that it will generally be hard to make a good case for self-hosted computing hardware when space is one of our most costly and rare resources.

Atm the laser cutter is not a comparable project for the reasons David mentions below (there was a discussion and pledge _before_ it arrived.)

Please do experiment with Zeus for a while though. Nobody's in a massive hurry to get rid of it.

m.

[Russ Garrett]

I've discussed this matter with the other trustees. It's worth noting
that we are obliged to make sure that London Hackspace strives to do
the best we can with our assets to further our goals. We've concluded
it would be irresponsible for us not to try and raise as much revenue
as possible from these machines.

As previously mentioned, comparisons with the 3-in-1 and the laser
cutter are moot: these were paid for predominantly by pledges, and the
pledge system goes some way to validating the Hackspace's need for
these servers. We're not convinced that it's in the best interests of
the membership to keep hold of these servers rather than selling them.
So:

1) We will keep these machines for three months. This should give
people plenty of time to play around with the hardware.
2) Members can run a pledge to "buy" these machines for the Hackspace
at market value. (I'm more than happy to discuss what the market value
is.)
3) We will auction the remaining servers off. I'm happy to set a
sensible reserve amount, and we can perhaps keep them around if that
isn't met.

--
Russ Garrett
ru...@garrett.co.uk