A100 accelerator node

[Jean-Baptiste Denis]

Hello everybody,

I can't find any valuable information on the A100 accelerator node
besides the installation guide. Am I missing something ?

My understanding is that if you already have a bunch of nodes with 10Gb
network connection, there is no interest in using the external network
of the A100, correct ?

If I'm not using A100 the external network, will I (potentially) benefit
from it regarding reads and/or writes ?

Can I choose what part of the filesystem I want to be cached ?

Can I use multiple A100 node within the same cluster to augment my L1
cache accordingly ?

My use case would be heavy read access pattern of hundreds/gigabytes
files from a specific directory from a compute cluster. Today, if I've a
workload consisting of multiple processes requesting different large
files (through NFSv3) that do not fit in RAM, cache poisonning is
occuring locally on the compute node which implies network traffic and
(I think) hammering the isilon cluster. Multiply this behaviour with
dozens of compute nodes...

I'm not 100% sure how badly this pattern will affect my isilon cluster,
but I'm just looking for general answers to better understand the
behaviour of an accelerator node and see if this could be a potentially
good fit for my usage. I'm perfectly aware that this can't be the silver
bullet that will save me from fully understand and qualify what's going
on here.

Anyone could provide some general feedback about the A100 ?

Jean-Baptiste

[Rob Peglar]

Greetings J-B,

As it turns out, yes you are missing something :-)

You absolutely _do_ want to use the 10GE ports on the A100. By directing
your high-throughput clients to it, you give those clients the ability to
read the A100's cache, for the purpose you well stated below - reading a
large working set with maximum throughput.

Good practice is to create a SmartConnect zone for these clients, directed
specifically at the A100 10GE ports. You can have multiple A100s,
absolutely, to scale out.

As for 'choosing what part of the filesystem I want cached', one does that
indirectly. The A100 will cache those files which are opened and read via
the A100 ports. Some people call this 'pre-heating' the working set. Once
read initially, the A100 will cache file data up to its RAM limit, which is
256GB and is used as all level-1 cache.

As for getting documentation, ask your SE... he or she has plenty of that
:-)

Hope this helps

Rob

--
You received this message because you are subscribed to the Google Groups
"Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2014.0.4158 / Virus Database: 3629/6837 - Release Date: 11/14/13

[Cory Snavely]

Depending on the size of the data set you hope to cache, your
application might also be a candidate for a flash-based NAS cache -
flash gives you a bigger cache for the same $. Avere is a front-runner
with this technology, but there are other viable options.

As Rob says, though, you should definitely see what kind of performance
improvements your accelerator can provide.

[Matt Dey]

Just to add on to what Rob has to say.  We have been testing the A100's with some mixed results.  Were currently stalled in our testing which started back in September as Isilon seems to of let a few things slip by their QA process so we are eagerly awaiting some fixes.

Issues aside the initial results we are seeing seem to indicate we will receive a performance boost even over 10G connected X400 nodes.  If you aren’t using the external network interfaces on the A100’s you will receive zero benefit from them.   The A100’s will only cache the data that is flowing thru them as they service client requests so if they aren’t handling any client requests their large cache will not benefit you at all.   You can use multiple A100’s per a cluster we are currently testing with 2.

The large cache on the A100’s should help you.  To target things to be cached you will need to configure your smart-connect zones and clients properly. 

[Jean-Baptiste Denis]

On 11/15/2013 04:14 PM, Rob Peglar wrote:
> Greetings J-B,

Hi,

> As it turns out, yes you are missing something :-)

Ehe,

> You absolutely _do_ want to use the 10GE ports on the A100. By directing
> your high-throughput clients to it, you give those clients the ability to
> read the A100's cache, for the purpose you well stated below - reading a
> large working set with maximum throughput.

Ok. But I "only" have two 10GE ports I can use, which could be quite
limiting if I already have multiple 10 GB connection on my isilon
cluster. But using the port on the A100, I also introduce a SPOF if I
have only one.

Are you saying that I HAVE to use the external port to be able to read
from A100 cache ? Is the A100 still interesting without using them ?

> Good practice is to create a SmartConnect zone for these clients, directed
> specifically at the A100 10GE ports. You can have multiple A100s,
> absolutely, to scale out.

Ok, I see. But I'm still worried with the introduced SPOF.

> As for 'choosing what part of the filesystem I want cached', one does that
> indirectly. The A100 will cache those files which are opened and read via
> the A100 ports. Some people call this 'pre-heating' the working set. Once
> read initially, the A100 will cache file data up to its RAM limit, which is
> 256GB and is used as all level-1 cache.

It make sense if I use the external port of the A100, indeed.

> As for getting documentation, ask your SE... he or she has plenty of that
> :-)

I've asked the support I they have any technical public documentation on
it, which does not seem to be the case. I'll contact my SE !

> Hope this helps

Very much, thank you !

[Jean-Baptiste Denis]

On 11/15/2013 04:32 PM, Cory Snavely wrote:
> Depending on the size of the data set you hope to cache, your
> application might also be a candidate for a flash-based NAS cache -
> flash gives you a bigger cache for the same $. Avere is a front-runner
> with this technology, but there are other viable options.

I should look into that, thank you for the suggestion.

> As Rob says, though, you should definitely see what kind of performance
> improvements your accelerator can provide.

Yep !

Jean-Baptiste

[Jean-Baptiste Denis]

On 11/15/2013 04:37 PM, Matt Dey wrote:

> Issues aside the initial results we are seeing seem to indicate we will
> receive a performance boost even over 10G connected X400 nodes. If you

> aren�t using the external network interfaces on the A100�s you will

> receive zero benefit from them.

Ok.

> The A100�s will only cache the data

> that is flowing thru them as they service client requests so if they

> aren�t handling any client requests their large cache will not benefit
> you at all.

This answers my previous question to Rob perfectly, thank you.

[Peter Serocka]

Hey,

let’s do not misunderestimate the L2 cache…

The L1 caches only the data sent from a specific node
to its own clients, so the same file data might get replicated
in different nodes, kind of wasting memory,
even more so with multiple A100s (when all are serving
the same data set).

The L2 cache is truly shared as it caches the
blocks read from one nodes’s disks for being fed
to other nodes.

Say you have 10 X200 nodes with 6 GB RAM each (minimum offered),
and your data set is 100G — that won’t fit into those 60GB.
Will fit into an A100's 256GB, but then you waste
your 10GE ports on the X200s.

X200/X400 should be able to have their RAM upgraded,
10 x 12 GB or 10 x 24 GB (as an example) would really
help -- through L2 cache!

Cheers

— Peter

[Jason Davis]

Yeah, fully speced X400s with the maximum amount of RAM available do make for an interesting source of L1 and L2 cache for the cluster. 

In our deployments, we leverage the accelerator nodes to point specific, non-customer workloads and use them for running things like ad-hoc SmartPool migrations. Unfortunately, we're still on 6.5.5.x so I can't directly comment on the A100s.

On Fri, Nov 15, 2013 at 9:49 AM, Jean-Baptiste Denis <jbd...@pasteur.fr> wrote:

On 11/15/2013 04:37 PM, Matt Dey wrote:

> Issues aside the initial results we are seeing seem to indicate we will
> receive a performance boost even over 10G connected X400 nodes.  If you

> aren’t using the external network interfaces on the A100’s you will

> receive zero benefit from them.

Ok.

> The A100’s will only cache the data

> that is flowing thru them as they service client requests so if they

> aren’t handling any client requests their large cache will not benefit
> you at all.

This answers my previous question to Rob perfectly, thank you.

[Jason Davis]

Agreed, L2 cache can't be underestimated... L2 cache is still faster than hitting a disk, SSD or HDD. 

This is partially why Isilon leverages IB for the back end, to make these cache hits less latency heavy :)

[Matthew Dey]

Just curious the workloads you moved to the accelerators nodes were they large or abusive loads?  Why did you chose to move them to the the accelerators instead of customer work loads?  I would of thought you would want to put the customer work loads on the accelerators to speed them up and isolate less critical workloads to a subset of some storage nodes?

Matt Dey
203-428-4166

[Saker Klippsten]

We have 4 A100's 

We use them to solely server out our applications. Being a visual effects company we have a lot of software beta's and we are always updating and testing so it makes sense for use to server it off our Isilon cluster to save time on deploying to everyones workstations and our render farm. One of the issues we started to have was that the application load/launch time began to increase because of the heavy load/burden put onto the storage nodes themselves by our render farm ( ~2000 Six and Eight Core Procs ) We are all CIFS, and that loves CPU cycles. Our entire "Tools" directory for the various applications and python code base ~ 200GB so it fits nicely within the specs of the A100's L1 Cache. Though we are ALWAYS hungry for more we have tried to keep it as lean as possible to stay within the spec and give some overhead room. 

With this in place our users no longer feel the "big slow downs" associated with launching/interacting with the applications when large renders are kicked off hitting the storage nodes.  We created another smartzone  \\apps.zoicstudios.com which point to the 4 A100's soley. We are thinking of getting more to separate the workstations and render nodes to reduce the burden even more.

I will add that yes adding as much L2 to each storage node helps as well which is what we did and maxed those out. 

We also have some Avere nodes but we use those for caching  over the WAN between clusters/facility locations. They work nicely but it would be very nice to have an Isilon solution for that issue which I hope is coming soon ;) just to keep managing everything under one roof.. along with the other obvious benefits like permissions etc.. 

@ Matt Dey do you mind sharing ( even offline ) what issues you were having with your A100 testing? What bugs issues did you run into?

Thanks

-S

[Jason Davis]

Yeah, the work being done on these accelerator nodes is somewhat abusive IO wise, so the thinking is that we could better isolate any cluster wide impact by pointing the jobs at these nodes.

[Jeff Young]

I work for Matt....

 

We have been working hard with Isilon trying to help them improve the quality of the A100s. From a performance standpoint, we found significant benefit as a result of putting ALL of our IOs through a pair of A100s instead of through the storage nodes alone. Variability went down and steady state perf was as good always and better often. We tested both with and without the A100s extensively.

 

But the A100s have monitoring and stability challenges.

 

Are you an InsightIQ user? How do your A100s look there? Do you have any way to sanity check your InsightIQ data? Throughput, for example - are you able to look at your switch port traffic with any other monitoring tool to compare that to InsightIQ's perspective of A100 throughput?

 

 

[Jean-Baptiste Denis]

On 11/15/2013 04:57 PM, Peter Serocka wrote:

> let�s do not misunderestimate the L2 cache�

Thank you for pointing this out.

Jean-Baptiste

[Daniel Cornel]

Just curiosity here.

Would the performance you get from adding nodes and their subsequent cache not be as direct of an alleviation? also, the clients that benefit from the accelerators are not databases, correct? Im curious how accelerators would assist with vmdk files, with some scaling up to 6tb.