Isilon-Users CIFS SMB take a long time to respond

[LaurentMap]

Hello,

I experience bad performances when I try to use a CIFS share on my
Isilon cluster. Each time I try to open a CIFS share from a Windows
client it take 10 to 20 seconds to appears.

I capture packet of a session with Wireshark to show on which step of
the SMB negotiation is long and it's from the first step : "the
Negotiate Request".

Negotiate Protocol Response
[...]
SMB (Server Message Block Protocol)
SMB Header
Server Component: SMB
[Response to: 391]
[Time from request: 13.096380000 seconds]
NT status: STATUS_SUCCESS (0x00000000)
[...]

I search on MSDN but nowhere MS tell that it's normal to take thoses
long seconds (http://msdn.microsoft.com/en-us/library/
cc246792%28v=prot.13%29.aspx).

I try to install a SAMBA to another Linux machine with AD authent
(same as the Isilon node) and I don't experience this long waiting
time. Otherwise I observe that the SMB version on Isilon is a custom
one :
isilon-8# smbd -V
Version 3.4.0-Isilon OneFS v6.0.3.10

And on my Linux server:
puppet:~# smbd -V
Version 3.2.5

For the clients I try and experience it both on Windows 2k3 or 2k8.

Another thing to note is that when I open an explorer on the CIFS
Isilon share, the first one take the 10 to 20 seconds to open, and
after, if I keep it open and try to open a new one, I don't experience
this long time. On the other side if I close this explorer.exe
windows, wait few seconds and retry to open a new one, I experience
again this long waiting time.

Here is the interesting lines on the Isilon logfile /var/log/samba/
samba.log :


[Intiating the explorer request 20 seconds before :]
2012-02-15T10:52:47+01:00 <30.4> isilon-7(id7) smbd[98614]:
[2012/02/15 10:52:47.421558, 1, effective(0, 10007), real(0, 0)] smbd/
service.c:1078(make_connection_snum)
2012-02-15T10:52:47+01:00 <30.4> isilon-7(id7) smbd[98614]:
MyComputerName (10.0.1.181) connect to service VA initially as user
MydomainName\myname(uid=0, gid=10007) (pid 98614)

[Here I close my explorer aproximatively 5 seconds before this line :]
2012-02-15T10:53:58+01:00 <30.4> isilon-7(id7) smbd[98614]:
[2012/02/15 10:53:58.792818, 1, effective(0, 0), real(0, 0)] smbd/
service.c:1257(close_cnum)
2012-02-15T10:53:58+01:00 <30.4> isilon-7(id7) smbd[98614]:
MyComputerName (10.0.1.181) closed connection to service VA
2012-02-15T10:54:28+01:00 <30.3> isilon-7(id7) smbd[98614]:
[2012/02/15 10:54:28.778685, 0, effective(0, 0), real(0, 0)] lib/
util_sock.c:537(read_socket_with_timeout)
2012-02-15T10:54:28+01:00 <30.3> isilon-7(id7) smbd[98614]:
[2012/02/15 10:54:28.778855, 0, effective(0, 0), real(0, 0)] lib/
util_sock.c:1483(get_peer_addr_internal)
2012-02-15T10:54:28+01:00 <30.3> isilon-7(id7) smbd[98614]:
getpeername failed. Error was Socket is not connected
2012-02-15T10:54:28+01:00 <30.3> isilon-7(id7) smbd[98614]:
read_socket_with_timeout: client 0.0.0.0 read error = Socket is not
connected.

My configuration on Isilon is very "simple" :

My SMB.conf:

[global]
include = /usr/local/etc/isilon.smb.conf
dos charset = ASCII
workgroup = MydomainName
realm = MydomainName.ext
server string = Isilon Server
security = ads
password server =
hosts allow =
dedicated keytab file = /etc/krb5.keytab
kerberos method = dedicated keytab
name resolve order =
interfaces = em0 em1 fec0
netbios name = isilon
netbios aliases = isilon-8
guest ok = no

[VA]
comment = VA data
path = /ifs/test-VA-processed
admin users = S-1-1-0

My isilon.smb.conf :

# DO NOT EDIT. Please edit /etc/mcp/override/smbd.xml instead.
syslog only = Yes
unix extensions = No
passdb backend = wbc_sam
idmap uid = 10000 - 20000
security = SHARE
debug uid = Yes
dos charset = ascii
store dos attributes = Yes
force create mode = 100
use unexpected tdb = No
local master = No
workgroup = WORKGROUP
posix locking = No
perfcount module = pc_onefs
max log size = 0
min receivefile size = 1
read only = No
kernel oplocks = Yes
hide dot files = No
create mask = 700
smb passwd file = /usr/local/private/smbpasswd
socket options = TCP_NODELAY
enable privileges = No
delete readonly = Yes
syslog = 11
auth methods = guest wbc
directory mask = 700
disable spoolss = No
strict locking = No
dns proxy = No
force unknown acl user = Yes
map archive = No
use sendfile = Yes
host msdfs = No
stat cache = No
debug hires timestamp = Yes
idmap gid = 10000 - 20000
server string = Isilon Server
vfs objects = onefs_shadow_copy onefs
veto files = /.ifsvar/
smbd:search ask sharemode = No

My /etc/mcp/override/smbd.xml :

<?xml version="1.0" encoding="utf-8"?><isi-data file="smbd">
<add-tag id="smbd1">
<propagation-timestamp>1320077566.69</propagation-timestamp>
</add-tag>
<add-tag id="smbdglobal">
<field name="hosts allow" value=""></field>
</add-tag>
<add-tag id="smbdglobal">
<field name="name resolve order" value=""></field>
</add-tag>
<add-tag id="smbdglobal">
<field name="password server" value=""></field>
</add-tag>
<add-tag id="smbdglobal">
<field name="security" value="ads"></field>
</add-tag>
<add-tag id="smbdglobal">
<field name="server string" value="Isilon Server"></field>
</add-tag>
<add-tag id="smbdglobal">
<field name="dos charset" value="ASCII"></field>
</add-tag>
<add-tag id="smbdglobal">
<field name="workgroup" value="MydomainName"></field>
</add-tag>
<add-tag id="smbdglobal">
<field name="realm" value="MydomainName.prod"></field>
</add-tag>
<add-tag id="smbdglobal">
<field name="dedicated keytab file" value="/etc/krb5.keytab"></
field>
</add-tag>
<add-tag id="smbdglobal">
<field name="kerberos method" value="dedicated keytab"></
field>
</add-tag>
<add-tag id="smbd1">
<version value="5.5"></version>
</add-tag>
</isi-data>

How can I increase this bad performance ?
Any ideas or debug test ?

Thanks

[J. Lasser]

This can often occur if you're not using Kerberos for authentication.
Are you using the shortname for the server? Try using the FQDN to
connect and see if that has better performance.

Other common causes of this include negotiation issues with the AD
server and problems with reverse DNS. From the client side, you might
grab a packet capture that includes DNS and AD traffic to all hosts,
as well as SMB traffic to the Isilon, and see what's occurring in the
time window. (Combine this with a cluster-side packet capture
including DNS and AD, and you'll have a complete picture of the
relevant traffic.)

Jon

--
Jon Lasser                     j...@lasser.org                      206-326-0614
. . . and the walls became the world all around . . .  (Maurice Sendak)

[LaurentMap]

Thanks it was it. After various tests I find that one of our AD has
bad respond time for AD authentication. I try without and it was
instantanely.

Many thanks
Laurent

[saurabh chaudhary]

Hi Laurent,

We are also facing the same issue in our Isilon.

Can you please guide us or let us know what set of steps you performed to resolve or narrow down the issue.

Regards,

Saurabh

[Laurent Tupin]

Hi Saurabh,

To troubleshoot this kind of troubles you can use Windows tools to check everything on your AD and DNS Windows domain side.

Rgds

[Christopher Kingett]

Laurent,

we have exactly the same problem you where seeing. Can  I ask you just to confirm that to fix this you pointed your DNS settings to another server(s), or did you have to change the way you where authenticating against Windows. Isilon support are baffled by this.

Any advice appreciated,

Thanks,

Chris Kingett

[Laurent Tupin]

Hi Christopher,

That was a loooooooong time ago and if I think a lot to remember this case we had two AD servers and, has explained one of them was "sort of down". So I set the IP in Isilon to the better one (and not the dns lb address) and revert to a correct confiuration when the second bad one was repaired (that was a sort of update troubles...) everything were right. So, from my point of view and this experience, be sure that you're DNS and AD are correctly configured (nslookup dnscmd and other sort of dcdiag tools should be your friends).

Rgds

Laurent TUPIN

[Christopher Kingett]

Laurent,

That sounds like our issue. I have a ticket in with emc so I'll see where they take it.

Many thanks for your reply,

Chris

Sent from my iPad

[saurabh chaudhary]

Hello All,

Even we face the issue in our Isilon Cluster.

Problem as: 

While accessing the files over the network shares first time user faces slowness while opening the file (almost 2-3 min. to open a file like excel or PDFs), but slowness removes after first access. But as the session remain idle problem repeat again.

We opened the case with EMC and worked with our AD Team, below is the root cause:

Due to multiple nested AD groups member in the user profile causes Kerberos token value for user to exceed the value causing authentication delays while accessing files from Isilon shares. Due to that first time access cause latency to open file but then second time file opens quickly as the user was authenticated session established. But as the session remain idle problem repeat again.

Solution to the problem:

Long Term Solution - Continuous Improvement - Perform Nested Group cleanup on AD groups and users profile.

Short Term Solution - Perform Code upgrade from 7.2.0.1 to 7.2.1.0 for the Isilon on DR first and then on production Isilon (Risk: Code upgrade is irreversible\no-rollback).

Hope this may help you in your case too.

Thanks and Regards,

Saurabh Chaudhary
+91-8527508366

--
You received this message because you are subscribed to a topic in the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/isilon-user-group/BzEE0D4FOXs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sanghvi, Arun R. (GE Power)]

We resolved similar issue  by using   FQN.

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

[Alistair Stewart]

If you don't have a SPN in AD for the name you're using, authentication has to fall back and uses NTLM instead of Kerberos to authenticate with no caching of authorisation. Connecting to the share using the ip address ALWAYS uses NTLM.
SPNs are created for all SMB shares when the cluster is joined to AD, but any shares added later will need SPNs created for them.

Al...