MS15-027, Kerberos, NTLM - warning, you might still be using NTLM

[Dan Pritts]

You've probably all seen the technote about this, if you haven't, see

https://support.emc.com/kb/199379

summary, if you are using NTLM authentication, AD auth of your cluster will break after the AD servers install patch MS15-027

So you're thinking, NBD, I'm sure we use kerberos.  After all, this is 2015, and the isilon even warned me that I had to put  service principal names for my access zones into AD - all should be good.

In our case, it wasn't.  Our campus AD still supports NTLM to support some legacy something or another. And, our win7 clients were using NTLM to authenticate to the cluster. 

Turns out that our clients access \\foo.umich.edu.  Meanwhile, the Isilon told me to install an SPN for HOST/foo.    Installing an SPN for HOST/foo.umich.edu seems to have fixed the problem, although I'm not 100% sure. 

So, before you patch your AD, it might behoove you to do some testing.

And yes, I opened a ticket with EMC complaining that the cluster gave me a false sense of security by complaining about the HOST/foo SPN.  Not sure exactly what I think they should do about it, but something.

we're on 7.0.2.12, but i have no reason to believe this is version-specific. 

hope this helps someone. 

danno
--

Dan Pritts
ICPSR Computing & Network Services
University of Michigan
+1 (734)615-7362

[Jamie Ivanov]

Dan,

I'm sorry if this is duplicate information but I can't see that KB article. That is in response to CVE-2015-1637 (http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-1637) in which Microsoft is changing their SSL/TLS ciphers (https://technet.microsoft.com/en-us/library/security/3046015.aspx). While that affects all OneFS releases, there are patches being developed that may go back as far as the latest 6.5.5.x (6.5.5.29?) release. This should also not affect Kerberos nor should it affect NTLM authentication to the local authentication provider (on the cluster, not AD).

In terms of SPNs, there should generally be a SPN for the shortname and FQDN.

The best source of information would be the Windows Protocols team within Isilon support.

Jamie Ivanov
Mobile: 608.399.4252
http://www.linkedin.com/in/jamieivanov
-- -- -- -- -- -- -- -- -- -- -- --
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Pritts]

"isi smb sessions list" provides a valuable clue about how users are authenticating. 

[root@ICPSRIsilon-1 ~]# isi_for_array isi smb sessions list | grep danno
ICPSRIsilon-5: 192.168.146.42  UMROOT\danno
ICPSRIsilon-1: 192.168.146.55  da...@ADSROOT.ITCS.UMICH.EDU

.55 is a windows machine in AD, .42 is a mac that's not in AD. 

danno

Dan Pritts

March 13, 2015 at 10:19 AM

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Pritts]

Jamie Ivanov wrote:
> In terms of SPNs, there should generally be a SPN for the shortname
> and FQDN.

So I've learned. And if I'd thought about it, I'd have realized that
the FQDN would be a good idea.

However, the cluster specifically complained when i didn't put in the
SPN for the shortname. But it makes no such complaint about missing
FQDN's.

It even has "isi auth ads spn check" to confirm that i've got everything
"right".

anyway, no skin off my nose. I take it from the lack of other response
that nobody onthe list got bitten by this, good to hear. Based on my
conversation with support, there are lots of people out there who did
get hit.

[Ozen]

Yesterday EMC published a patch for this issue only targeting the latest 7.2 family code 7.2.0.1. Patch number is 145051. Rgiht after installation we experienced some auth difficulties that resolved once auth services refresh on cluster side.

As of today there is no released patch targeting other OneFS versions for this issue.

[Erik Weiman]

Patches are available for other versions of OneFS and they were downloadable as of yesterday late afternoon. Search for MS15-027 and you should see all of the available options. Also I'm fairly certain that the final step in the read me for installation is to run a service refresh command. 

--

Erik Weiman 

Sent from my iPhone 6+

[Ozen]

Hello Erik,

There is no service refresh step for patch i mention before, but it would very nice to had it.

[Peter Serocka]

It would be also helpful to have both
the ETA and the Current Patches document
updated to announce these patches...

fwiw

— Peter

[John Beranek - PA]

Here we held off installing the Microsoft patch, as our current production cluster is running an older OneFS version which is unlikely to get a patch, and we're not currently in a position to do a service-impacting upgrade.

Talk of the SPNs made me check ours, but we're covered on that point.

The "isi smb sessions list" trick is interesting, it does show a few Mac users on our site.

John

[John Beranek - PA]

Our other cluster has been updated to 7.1.1.2, but already has patch-140284 installed. The notes for patch-145050 don't mention whether it includes all the fixes in patch-140284. Off to EMC Support I go...

John

On Friday, 13 March 2015 14:19:50 UTC, Daniel Pritts wrote:

[Dan Pritts]

John Beranek - PA wrote:
> The "isi smb sessions list" trick is interesting, it does show a few
> Mac users on our site.

Just to be clear, the FOO\user doesn't specifically mean it's a Mac, it
means it's using NTLM. I think.

[John Beranek - PA]

Confirmed, a Linux client using smbclient without Kerberos shows as "DOMAIN\user" too.

John

[John Beranek - PA]

Additionally I can't do SMB with Kerberos on that AD-authenticated Linux client, apparently due to some SPN issue. Need to get my head around that.

John

[John Beranek - PA]

Aha, apparently it was because I was using a DFS path to the Isilon. If I use the smartpool name (which is in the cluster's SPNs) SMB+Kerberos works.

Now I wonder if we should be adding the DFS domain into the Isilon's SPNs...

John

[John Beranek - PA]

Hmm, not totally convinced that having @ in the user does mean Kerberos. If I add "-v" into the smb sessions command, I can see the following for all of our users connecting via DFS:

   Computer: 10.20.30.40

       User: jo...@DOMAIN.EXAMPLE.COM

Client Type: DOS LM 2.0

Now what does the Isilon mean by "DOS LM 2.0"!?

John

[Jamie Ivanov]

"DOS LM 2.0" is the negotiated SMB dialect. During the SMB connection setup, the client and server (in this case, OneFS node running the Likewise SMB daemon) negotiates the dialect so both client and server have a common set of features that the protocol supports.

For additional reading, please see:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365235%28v=vs.85%29.aspx
https://msdn.microsoft.com/en-us/library/dd541643.aspx

Jamie Ivanov
Mobile: 608.399.4252
http://www.linkedin.com/in/jamieivanov
-- -- -- -- -- -- -- -- -- -- -- --
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

[karan reddy]

So, does that mean @ is kerberos and domain/user is NTLM. I see both on our clusters.

We haven't applied the microsoft patch too, as our clusters are on older OneFS version and not ready to do the disruptive upgrade.

Thanks,

Karan

[Dan Pritts]

I think your interpretation is correct, Karan.

I don't know for sure, but I do know that as soon as I added FQDN SPN's to my isilon's AD object, most users went from DOMAIN\foo to f...@DOMAIN.UMICH.EDU.

Does anyone have any word from inside EMC as to when a patch will be ready?  My campus AD folks are getting a bit itchy about running unpatched.

danno

karan reddy

March 26, 2015 at 1:33 PM

Jamie Ivanov

March 25, 2015 at 5:16 PM

"DOS LM 2.0" is the negotiated SMB dialect. During the SMB connection setup, the client and server (in this case, OneFS node running the Likewise SMB daemon) negotiates the dialect so both client and server have a common set of features that the protocol supports.

For additional reading, please see:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365235%28v=vs.85%29.aspx
https://msdn.microsoft.com/en-us/library/dd541643.aspx

Jamie Ivanov
Mobile: 608.399.4252
http://www.linkedin.com/in/jamieivanov
-- -- -- -- -- -- -- -- -- -- -- --
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

John Beranek - PA

March 25, 2015 at 10:05 AM

Hmm, not totally convinced that having @ in the user does mean Kerberos. If I add "-v" into the smb sessions command, I can see the following for all of our users connecting via DFS:

   Computer: 10.20.30.40

       User: jo...@DOMAIN.EXAMPLE.COM

Client Type: DOS LM 2.0

Now what does the Isilon mean by "DOS LM 2.0"!?

John

On Wednesday, 25 March 2015 12:36:23 UTC, John Beranek - PA wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

John Beranek - PA

March 25, 2015 at 8:36 AM

Aha, apparently it was because I was using a DFS path to the Isilon. If I use the smartpool name (which is in the cluster's SPNs) SMB+Kerberos works.

Now I wonder if we should be adding the DFS domain into the Isilon's SPNs...

John

On Wednesday, 25 March 2015 12:30:55 UTC, John Beranek - PA wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

John Beranek - PA

March 25, 2015 at 8:30 AM

Additionally I can't do SMB with Kerberos on that AD-authenticated Linux client, apparently due to some SPN issue. Need to get my head around that.

John

On Wednesday, 25 March 2015 12:28:58 UTC, John Beranek - PA wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Dan Pritts
ICPSR Computing & Network Services
University of Michigan

+1 (734)615-7362

[Erik Weiman]

Patches are available already. Search the EMC download site for the Microsoft Security Bulletin number. 

If you really want to know if NTLM is being used the best way would be to enable NTLM usage auditing in your domain. 

--

Erik Weiman 

Sent from my iPhone 6+

On Mar 26, 2015, at 12:47 PM, Dan Pritts <da...@umich.edu> wrote:

I think your interpretation is correct, Karan.

I don't know for sure, but I do know that as soon as I added FQDN SPN's to my isilon's AD object, most users went from DOMAIN\foo to f...@DOMAIN.UMICH.EDU.

Does anyone have any word from inside EMC as to when a patch will be ready?  My campus AD folks are getting a bit itchy about running unpatched.

danno

<compose-unknown-contact.jpg>

karan reddy

March 26, 2015 at 1:33 PM

So, does that mean @ is kerberos and domain/user is NTLM. I see both on our clusters.

We haven't applied the microsoft patch too, as our clusters are on older OneFS version and not ready to do the disruptive upgrade.

Thanks,

Karan

On Wednesday, March 25, 2015 at 5:16:04 PM UTC-4, Neproshennie wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<compose-unknown-contact.jpg>

Jamie Ivanov

March 25, 2015 at 5:16 PM

"DOS LM 2.0" is the negotiated SMB dialect. During the SMB connection setup, the client and server (in this case, OneFS node running the Likewise SMB daemon) negotiates the dialect so both client and server have a common set of features that the protocol supports.

For additional reading, please see:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365235%28v=vs.85%29.aspx
https://msdn.microsoft.com/en-us/library/dd541643.aspx

Jamie Ivanov
Mobile: 608.399.4252
http://www.linkedin.com/in/jamieivanov
-- -- -- -- -- -- -- -- -- -- -- --
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<postbox-contact.jpg>

John Beranek - PA

March 25, 2015 at 10:05 AM

Hmm, not totally convinced that having @ in the user does mean Kerberos. If I add "-v" into the smb sessions command, I can see the following for all of our users connecting via DFS:

   Computer: 10.20.30.40

       User: jo...@DOMAIN.EXAMPLE.COM

Client Type: DOS LM 2.0

Now what does the Isilon mean by "DOS LM 2.0"!?

John

On Wednesday, 25 March 2015 12:36:23 UTC, John Beranek - PA wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<postbox-contact.jpg>

John Beranek - PA

March 25, 2015 at 8:36 AM

Aha, apparently it was because I was using a DFS path to the Isilon. If I use the smartpool name (which is in the cluster's SPNs) SMB+Kerberos works.

Now I wonder if we should be adding the DFS domain into the Isilon's SPNs...

John

On Wednesday, 25 March 2015 12:30:55 UTC, John Beranek - PA wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<postbox-contact.jpg>

John Beranek - PA

March 25, 2015 at 8:30 AM

Additionally I can't do SMB with Kerberos on that AD-authenticated Linux client, apparently due to some SPN issue. Need to get my head around that.

John

On Wednesday, 25 March 2015 12:28:58 UTC, John Beranek - PA wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Dan Pritts
ICPSR Computing & Network Services
University of Michigan
+1 (734)615-7362

[Jamie Ivanov]

After some digging around, I think this may help clarify some things:

640 A Kerberos "principal" is a unique identity to which Kerberos can
641 assign tickets.  Principals can have an arbitrary number of components.
642 Each component is separated by a component separator, generally `/'.
643 The last component is the realm, separated from the rest of the
644 principal by the realm separator, generally `@'.  If there is no realm
645 component in the principal, then it will be assumed that the principal
646 is in the default realm for the context in which it is being used.
647
648 Traditionally, a principal is divided into three parts:  the "primary",
649 the "instance", and the "realm".  The format of a typical Kerberos V5
650 principal is `primary/instance@REALM'.
651
652   * The "primary" is the first part of the principal.  In the case of
653     a user, it's the same as your username.  For a host, the primary is
654     the word `host'.
655
656   * The "instance" is an optional string that qualifies the primary.
657     The instance is separated from the primary by a slash (`/').  In
658     the case of a user, the instance is usually null, but a user might
659     also have an additional principal, with an instance called
660     `admin', which he/she uses to administrate a database.  The
661     principal `jenn...@ATHENA.MIT.EDU' is completely separate from
662     the principal `jennifer/ad...@ATHENA.MIT.EDU', with a separate
663     password, and separate permissions.  In the case of a host, the
664     instance is the fully qualified hostname, e.g.,
665     `daffodil.mit.edu'.
666
667   * The "realm" is your Kerberos realm.  In most cases, your Kerberos
668     realm is your domain name, in upper-case letters.  For example,
669     the machine `daffodil.example.com' would be in the realm
670     `EXAMPLE.COM'.

So when you see:

> User: jo...@DOMAIN.EXAMPLE.COM

That would then imply:

jo...(/<default kerberos instance)@<kerberos realm>.

The readers digest version: yes, that should indicate the user has a kerberos ticket and is not using NTLM to authenticate.

Jamie Ivanov
Mobile: 608.399.4252
http://www.linkedin.com/in/jamieivanov
-- -- -- -- -- -- -- -- -- -- -- --
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

[Jamie Ivanov]

+1

Jamie Ivanov
Mobile: 608.399.4252
http://www.linkedin.com/in/jamieivanov
-- -- -- -- -- -- -- -- -- -- -- --
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

[Bob]

We have been trying to get our head around this issue for the last few weeks to gauge the impact to our environment. We also had SPN issues that have since been straightened out and now our AD bound systems using AD logons are authenticating properly with kerberos. One thing to note, AD bound systems that use local logons still do use NTLM (we have a good number of shared systems that use local logon ID's). 

We have two clusters on different code versions and provide different output to the isi smb sessions list command. Our primary cluster is 7.1.1.1 and it only provides user name for the User info, no domain or realm information with it. Our replication cluster is running 7.0.2.8 and it does provide User info in the user@domain format. It uses the "@" regardless of kerberos or NTLM authentication in our case. We determined this by enabling NTLM auditing on the client side and we can track the authentication traffic to the cluster along with monitoring the kerberos tickets issued, or lack there of, by the cluster. 

One last thing to note: we received word from our TAM that we should be coordinating the cluster patching with the domain controller patching. We should not have patched domain controllers with unpatched clusters or patch clusters with unpatched domain controllers. Fortunately our infrastructure team has the MS15-027 patch in a holding pattern while all the different factions here assess the impact to their environments. I do not look forward to the scheduling of these patches when the time comes. 

[Dan Pritts]

We have been trying to get our head around this issue for the last few weeks to gauge the impact to our environment. We also had SPN issues that have since been straightened out and now our AD bound systems using AD logons are authenticating properly with kerberos. One thing to note, AD bound systems that use local logons still do use NTLM (we have a good number of shared systems that use local logon ID's).

Presumably, these would be unaffected by AD patches, right? 

One last thing to note: we received word from our TAM that we should be coordinating the cluster patching with the domain controller patching. We should not have patched domain controllers with unpatched clusters or patch clusters with unpatched domain controllers. Fortunately our infrastructure team has the MS15-027 patch in a holding pattern while all the different factions here assess the impact to their environments. I do not look forward to the scheduling of these patches when the time comes.

That's how I was interpreting the tech note, I was hoping I was wrong.  what a PITA.

thanks
danno

[Jamie Ivanov]

Bob,

Excellent information! Unfortunately everything that I've seen in the source code references formatting from the platform API so it would take more time to come up with a definitive answer or something more in line with Ockham's Razor.

Have you considered setting up one of the OneFS virtual nodes for testing against an AD environment for testing both the Microsoft and OneFS patches?

Jamie Ivanov
Mobile: 608.399.4252
http://www.linkedin.com/in/jamieivanov
-- -- -- -- -- -- -- -- -- -- -- --
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

[Bob]

We have been trying to get our head around this issue for the last few weeks to gauge the impact to our environment. We also had SPN issues that have since been straightened out and now our AD bound systems using AD logons are authenticating properly with kerberos. One thing to note, AD bound systems that use local logons still do use NTLM (we have a good number of shared systems that use local logon ID's).

Presumably, these would be unaffected by AD patches, right? 

 

The users use the local logon to get into the workstation and then use AD credentials to connect to the Isilon share so I think they would still be affected since they are still hitting the domain controllers for authentication. 

what a PITA.

Yes. This. 

[karan reddy]

Thanks Bob! That clears it up. We have SPN for both FQDN & short names and I see both @ & /. We definitely use NTLM, there are quite a few Mac's.

Karan