Issues with Mac Finder on Isilon

[Dan Hatton]

Hi all,

I am having a problem with one customer who is using Mac's connecting to an Isilon.  The Isilon is running the latest version of One FS.  (7.2.X)  I've made the recommended settings from the Isilon Mac White paper, and we are getting the kind of performance we expect from an Isilon with a 10G connection, but when the Mac first connects to the shares (SMB or NFS) it takes a long time for the Finder to open the folders.

 I've tried connecting using SMB and NFS with the same slowness opening the finder.  I also tried swapping out the flash drives to use as L3 cache, and/or GNA, with no noticable difference.  

There are two folders that are causing most of the problems.  One folder has about 40K files in it, and the other one only has 25 folders, but the files are all 250G+.  In the folder with all the smaller files, they have the "Apple double" files (also known as dot-bar files) the folder with the large files does not have the extra apple meta-data files.  When you look at the processes on the mac, they just seem to sit there with the memory and CPU working, but not doing much on the network connection.  It's like the Isilon is handing contents of every file, and the Mac is ignoring almost everything.  Once the finder manages to open the folders everything works three to four times faster than the Windows server they have been using.  

Unfortunately, I have no experience with Macintosh computers, and this problem doesn't exist on any of the Windows or Linux clients, just the Mac.

Any chance anyone here has experience with the Mac?  I need to find a way to speed up the seeding of the finder cache.

Thanks 

Dan

[Peter Serocka]

The Finder is a bit crazy....

Have you tried switching off  "Show item info" 

and "Show icon preview" under "View Options" ?

If you want to further analyze what is going on,

these commands can help:

(on the Mac)  

sudo fs_usage -f pathname          

Note: pathname is literal, not a placeholder

(on the Isilon)

  

isi statistics client --remote_name CLIENTNAME --top   

Note: CLIENTNAME is a placeholder, e.g. joes-imac.example.com

isi statistics protocol --node NODENUM --top  

Note: NODENUM is a placeholder, e.g. 8

The latter gives you more details on

the actually NFS/SMB operations, but not on client level.

Connect the client to a less busy node and query against that node.

To my experience the Finder can scan about 1000 small files per

second from the Isilon (with metadata reads from SSD via GNA),

so an initial scan of 40K files is somewhere in the minute range

rather than say 10 minutes -- what are your observations, quantitatively?

hth

-- Peter

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Youssef Ghorbal]

Hello Dan,

We've spent a lot of time investigating this issue (a very known
issue by the way)
It's their since Apple ditched SAMBA to replace it with its own stack
(SMBX I think)
Unfortunately, there is nothing you can do about it :(
With every OSX release we hope things get fixed but it's not.

The problem is simple, by design Finder will have to gather ALL the
metadata of ALL files inside the directory before starting showing
anything (an SMB request per file...). While the Windows Explorer (on
a Windows Workstation) will simply get the listing in one SMB request,
show the result in a matter of seconds (but will lack some metadata
which is fine in the majority of use cases)

On a LAN with low latency, you can have something that kinda work,
but as soon as you have more latency, it's a nightmare (SMB over a VPN
for exemple)

For the 40k files in one directory specifically, it's so huge that
anything will eventually fail on the workstation side. IMHO systems
are not built with these kind of uses cases in mind. On the MAC you
will never go faster than a CLI access on NFS.

Youssef

[Dan Pritts]

To my experience the Finder can scan about 1000 small files per

second from the Isilon (with metadata reads from SSD via GNA),

so an initial scan of 40K files is somewhere in the minute range

rather than say 10 minutes -- what are your observations, quantitatively?

We had finder failures with a directory with 2300 items in it when we were using a Celerra.  The finder would just time out and give up after a couple minutes.  This was about 2-3 years ago, with then-current MacOS.

The celerra was much slower than an Isilon with SSD metadata acceleration, of course.

--
Dan Pritts
ICPSR Computing & Network Services
University of Michigan
+1 (734) 615-9529

[Dan Hatton]

Right now we are doing testing, so we only have a single Mac connected to the Isilon on a 10G ethernet connection.  We did lots of testing with file copies using DD from a terminal, so I know we are getting the throughput we are expecting to see.  Once the finder get's cached, we are able to stream data and work about 3 times faster than they were able to on the Windows server.  The only issue we are seeing is the finder, that and the customer really misses the trash can on the network drive.

Is there a way to get the finder to keep it's cache when it reboots or re-mounts the share?  

[J. Lasser]

Everyone's comments about per-file metadata are spot on, and probably the cause. That said, your contention that the network looks idle mitigates against the "loading data for every file" answer.

You might make sure you're capturing traffic to your auth servers and see if your authentication is slowing you down. With Macs on OS X 10.10 (Yosemite), I saw (not on Isilons) lots of slowdowns related to authentication. In particular, I'd see the auth request sent and replied to, then the Mac did nothing for up to 30 seconds, after which it sent a new auth request, got a new reply, and kept going. (The pcaps were Mac-side, so I know the original reply was received.) This seems to have gone away under El Capitan (10.11), at least on my systems.

--

You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jon Lasser                     j...@lasser.org                      206-326-0614
. . . and the walls became the world all around . . .  (Maurice Sendak)

[Dan Pritts]

The obvious solution is to make a different directory structure with fewer items in the directory.  I'm sure that's difficult or you would have already done it.

Less obvious would be to make a parallel directory structure with a bunch of symbolic links to the big directory.  That way the finder will only have to read a relatively few files. 

I can't imagine the finder can be told to keep its cache like you're hoping.  Seems very dangerous. 

Another thing you might try is running a bunch of 'ls' commands in parallel in a shell script.  Presumably (I can't say for sure) the finder is just using the OS-level file caching, so once the ls's have freshened the cache the finder will be faster.

I think someone already suggested this but make sure to disable as much as you can in the finder's view options.  "show view options" under the view menu.  Icon preview in particular. 

Hope this helps.  good luck.

danno

Dan Hatton

November 13, 2015 at 10:39 AM

Right now we are doing testing, so we only have a single Mac connected to the Isilon on a 10G ethernet connection.  We did lots of testing with file copies using DD from a terminal, so I know we are getting the throughput we are expecting to see.  Once the finder get's cached, we are able to stream data and work about 3 times faster than they were able to on the Windows server.  The only issue we are seeing is the finder, that and the customer really misses the trash can on the network drive.

Is there a way to get the finder to keep it's cache when it reboots or re-mounts the share?  

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dan Hatton

November 12, 2015 at 6:48 PM

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[Bernie Case]

On Thursday, November 12, 2015 at 11:48:07 PM UTC, Dan Hatton wrote:

There are two folders that are causing most of the problems.  One folder has about 40K files in it, and the other one only has 25 folders, but the files are all 250G+.  In the folder with all the smaller files, they have the "Apple double" files (also known as dot-bar files) the folder with the large files does not have the extra apple meta-data files.  When you look at the processes on the mac, they just seem to sit there with the memory and CPU working, but not doing much on the network connection.  It's like the Isilon is handing contents of every file, and the Mac is ignoring almost everything.  Once the finder manages to open the folders everything works three to four times faster than the Windows server they have been using.  

40K files in it is quite a lot. To enumerate any directory, the Mac has to do the following:

• Get a list of files in the directory (for both NFS and SMB)
• Query the storage server for the resource fork data
• On SMB this ends up being a query for an Alternate Data Stream
• On NFS this ends up being a query for the AppleDouble (dotbar, or ._) file.

Only after the Mac has received all of the file listing and extra metadata will it display all the files. That's an interesting design from Apple (I filed a bug years ago; it was shut down) and this is why the guide recommends making things simple on the Mac via things like List view and turning off icon preview. That's just less for the Mac to ask for from the server.

Even in the best of network environments with sub-ms latency enumerating 40K files you're looking at many more than just 40K network calls to get a Finder to draw its window. Add disk latency on that and you can see why your users sit and wait and wait. The network won't seem busy, but if you take a pcap you will see a lot of calls back and forth (the isi statistics suggestion is a good one to see it without a pcap). This is why the Mac guide recommends enabling readdirplus on the Mac (for NFS). For SMB there hasn't been much you can do to make it better other than SSD metadata and low network latency.

El Capitan changed that and if you 'man nsmb.conf' you'll see that there is an option now called dir_cache_async_cnt (the default is 10) which sends multiple SMB queries to fill the Mac's directory cache. Finally someone at Apple got the message that SMB wasn't nearly as good at metadata as AFP and they improved their El Capitan client. I still haven't really done much to figure out just how high you can tune it before you hit the limit, though, but I have seen faster metadata on El Capitan as a result of this.

So if you've got low latency network and metadata on SSD and El Capitan works with your workflows and applications it may provide the relief you need. If that's not possible, then I would suggest looking at NFS for Mac-only workflows as that will have the benefit of readdirplus and directory readahead. Mixing Macs on NFS with Windows on SMB is going to create a bunch of dotbar files for the Windows clients; that may not be a good experience for your users.

Finally, just a general suggestion, where possible, test your applications and workflows with an Isilon virtual node to make sure you don't see any sort of odd errors like Finder's -36 error.

[Bernie Case]

On Saturday, November 14, 2015 at 5:44:40 PM UTC, Bernie Case wrote:

Mixing Macs on NFS with Windows on SMB is going to create a bunch of dotbar files for the Windows clients; that may not be a good experience for your users.

Of course, this is also a more challenging setup when it comes to authentication, too (UID/GID coherency, user mapping, etc.). 

[Dan Hatton]

OK, so I opened an SR with EMC.  We set the directory access to streaming, turned off L3 (By the way, you can't manage L3 or GNA from the GUI in 7.2.0.3.  The GUI reports it as being off, but the CLI shows it running)  Ran the Smart Pool rebalance, ran the SmartPool job to reclaim the SSD's and when we were done, there wasn't any change at all with the finder.  The following is the response from the Isilon engineer...

 

I have gone over the packet captures personally, and I am not seeing any errors in the NFS communication that would explain the slow initial enumeration from Finder.  That being said, I did want to point out two different items from the packet capture that can help us.  Below is an excerpt from Wireshark showing the NFSv3 service response times, in other words how quickly the Isilon sent a reply back to the client for each request packet that it received:

 

If you look at the average Service Response time, for the majority of these calls the Isilon is responding back within 0.1-0.2 milliseconds.  Furthermore, over the course of 57,814 ACCESS responses the longest it ever took to respond was still 3.3 milliseconds.  In other words, there is no indication here that the Isilon ever gets stuck, either once or multiple times, to such a degree that it produces a 20-30 second response because of it.

 

I also want to point out though just how many calls there were in this NFS conversation.  Remember that this was a very short packet capture that we gathered, where we initiated a conversation and opened a folder to enumerate its contents three times.  In order for the Finder application to list those files three times it sent the Isilon cluster over 148,000 requests.  This is consistent with documented Finder application behavior where it asks for the metadata and other information for all objects within a directory before it displays anything within that directory.  Because of this, even if the requests and responses travel the wire quickly, and even if the Isilon responds fast as lightning, those are still about 50,000 calls that need responses before Finder will display anything. 

 

So based on my personal analysis of the situation, I just do not see a way that we can get faster on the Isilon than we are already seeing in this capture.  The solution to getting faster enumeration of directory contents will have to come from changing the behavior of the Finder application to where it does not need to send 50,000 calls for a single directory listing rather than trying to squeeze more speed out of the environment to handle those calls when we are already seeing 0.1 millisecond response times.

 

[Peter Serocka]

Thanks for sharing this, and kudos to the Isilon Engineer who

took the efforts in testing this and writing the comprehensible report!

There is one more "trick" to speed up

Finder operations, in limited scenarios though:

It can only be applied if either

- multiple clients access the data READ-ONLY 

- or a there is a single client, who then may read&write

Create all folders in questions natively on a Mac,

pack everything into a dmg image file(s) and load these

onto the Isilon NFS folder. For the first case, use

a special NFS export with the READ-ONLY (ro) option set.

Then clients "open" or "mount"  the image(s) and Finder will fly...

Cheers

-- Peter