requests-credssp and credentials encryprion at first hop.

[Павел Полушин]

Hello. 

I have security-related question.

In our environment we use ansible for application deployment. Ansible playbooks running by jenkins.

Scope for deployment contains Windows-based servers (2008R2+).

In some cases we are facing with "double-hop" problem when passing credentials is needed.

CredSSP is intended to solve problems like this but it's insecure (http://www.powershellmagazine.com/2014/03/06/accidental-sabotage-beware-of-credssp/).

In common cases, credentials are being sent in clear text. Here is picture http://www.powershellmagazine.com/wp-content/uploads/2014/03/image001.png/

I'm interested, is this problem solved in requests-credssp module? (Credentials are stored from jenkins. Ansible connects to servers using https.)

Does anyone tried to investigate it?

[Varun Chopra]

There was a recent post on Reddit about this...

Here it is: https://www.reddit.com/r/PowerShell/comments/7qra9r/double_hop_solvers_and_resourcebased_kerberos/

CredSSP isn't really the best way to go about this. And I think this post should go on Git as Ansible needs a better way to cover double-hops.

[Павел Полушин]

I see this topic but it not answers to my querstion. Common scenarioo when all servers are on windows server is covered in article.

вторник, 30 января 2018 г., 12:43:46 UTC+3 пользователь Павел Полушин написал:

[Jordan Borean]

TLDR: for Windows to authenticate with another server (double-hop), it will either need a Kerberos ticket with the delegation flag or the user's actual credentials (CredSSP/become) there is no way around that. If there is that is a security issue with Windows and should be fixed by Microsoft.

CredSSP isn't really the best way to go about this. And I think this post should go on Git as Ansible needs a better way to cover double-hops.

While I agree, it really isn't the best way to go about this and people should use Kerberos with delegation instead, CredSSP is still viable in some situations and in total Ansible covers 3 ways of allowing this

• Kerberos with delegation
• CredSSP
• Become (Ansible specific implementation)
  

All 3 allow you to cover double-hops while the 3rd fix even more limitations of a network process, I'll go into more detail below.

Background

The way that Windows works is that on a normal local logon, the username and password is supplied to LSA who then authenticates the user. This password is then available in that logon session and can be used by the client to authenticate with a remote server, i.e. connecting to a fileshare with the logged on credentials and so on. When authenticated with a server over the network (like WinRM), it will use an auth protocol like NTLM, Kerberos, CredSSP to authenticate the user and in the majority of cases the password of the user is not sent to the server. When LSA creates a logon session without a password (from an NTLM hash or Kerberos ticket like WinRM), it is unable to do things like calculate an NTLM hash or get a Kerberos ticket from the KDC in order to authenticate with another server.

This is usually fine for things like SMB but with WinRM users may want to interact with a third server to copy files like they would do locally but without that local password this is impossible for Windows to do, i.e. it can't calculate the NTLM hash if it doesn't have the password. In order to get over this problem, you can either use CredSSP to send the password to the remote server that is accessible by LSA or you can use a Kerberos ticket with the delegation flag set. With either of these methods, the remote server is able to authenticate with another server like you would be able to do locally but is unable to delegate to a 4th server from there, e.g. local -> remote -> double hop remote -> another hop remote is not allowed.

CredSSP

The first option CredSSP is a protocol designed by Microsoft to allow an application (A) to delegate the user's credentials to another server (B) by sending the user's password as "plaintext" and not a hash. This delegation means that the LSA of the remote host (B) has a copy of the plaintext username and password of the user which it can then use to authenticate with a third remote server (C). In general the protocol works like this

• The initial server response returns a HTTP 401 error with CredSSP in the WWW-Authenticate header
• The client sets up a TLS connection and starts the TLS Handshake which includes things like cipher suite negotiation
• Once the handshake is complete, the client will send either an NTLM or Kerberos token to authenticate the user
• After authenticating the user, the client will encrypt the server’s CredSSP public key with the authentication wrap function and send that to the server
• The server validates that the correct public key was used and there is no middle man in between the client and the server
• The server then sends it’s public key again with the first bit set to 1 also encrypted with the authentication wrap function
• The client will verify the public key and verify the first bit was set to 1
• Once both the client and server have verified each other, the client will then encrypt the username and password with the authentication wrap function and send that to the server

As well as the steps that use the authentication wrap function to encrypt the data, each step after the TLS handshake is also encrypted with the TLS protocol itself, e.g. when sending the username and password it is doubly encrypted (auth wrap and then TLS wrap). So the fact that the password is sent over the wire is troubling (and rightly unacceptable for some) but anyone snooping over the network would have a really hard time to get the credentials as it would have to break both the TLS protocol encryption and the auth protocol encryption (+1 more TLS encryption if running on the HTTPS WinRM listener).

Kerberos

Kerberos with delegation is a slight modification of the normal Kerberos process, normally the flow works like (I may be slightly wrong in some steps)

1. User contacts the KDC (Domain Controller) to get a Ticket Granting Ticket (TGT) for their logon session (for Windows this happens on logon time)
2. When authenticating with a server, the client uses the TGT and sends that to the KDC alongside the server's SPN where the KDC then verifies everything is correct and sends back a service ticket and session key
3. The client then sends this ticket to the service and uses the session key to encrypt the data
4. The service then talks to the KDC to verify the client's ticket and then sends back another ticket based on the KDC response
5. The client then verifies this same ticket with the KDC in order to verify that the service is who they say it is
   

Once this process is complete, the service has authenticated the user and the user has also authenticated the server itself but this is still not enough for the server to authenticate to another remote server as it does not have the user's password, just a short lived Kerberos ticket. The Kerberos protocol has an extension that defines a delegation flag which is attached to the request in step 2 that tells the KDC the service can delegate the user's credentials to another service. So when the service in step 4 talks to the KDC it is able to get a TGT and use it to run the same process for a remote host.

Become

The third option (with Ansible) is to use become, as the username and password are sent to the server and used to create a new logon process. As this means LSA then has the credentials of that account it works like it would when running processes locally. This has it's own problems in itself but solves more issues than what CredSSP or Kerberos with credential delegation fix. I won't go into too much detail about this but become pretty much changes the Ansible process to run like you would run it locally.

Summary

Ultimately all 3 options allow you to authenticate with a further remote host on the remote session but where Kerberos with delegation is best is that you can define constrains on what servers it is able to authenticate with. With constrained delegation, the remote server (B) can then use that ticket to authenticate with another server (C) as long as it has been allowed and not constrained in the Domain Controller.

So in the end, if you need to authenticate with another server (double hop) you are better off using Kerberos with delegation as;

• You can restrict the server's the credentials can be used to auth with to stop untrusted server's from getting a hash/ticket of your credentials
• The password itself is still never sent to the server, a special Kerberos ticket is sent instead
• The number of requests is less than CredSSP so the setup time can be quicker

If you cannot use Kerberos (for whatever reason) then CredSSP can also be done to achieve the same result but it does have some security implications you should be aware off beforehand. When people say CredSSP is insecure, that is definitely not right, you just need to be aware of the risks involved and mitigate them if necessary.

Thanks

Jordan

[Jeremie Levy]

Thank you Jordan,

So i'm using Kerberos and i'm missing the "with delegation" part.

Can you point me to how and where to configure this option ?

Thank you!

[Jordan Borean]

By default, the Kerberos ticket does not have delegation enabled, you can set

ansible_winrm_kerberos_delegation: True

in your host vars and Ansible will set the delegation flag required to get a kerb ticket with delegation. If this doesn't work you may need to configure your AD environment to properly allow it.

Thanks

Jordan

[Jeremie Levy]

Thanks Jordan

So i have this option activated since the beginning but the double hop is not resolved.

What extra configuration is needed here?

<hasgappqba2303> WINRM CLOSE SHELL: 22A19915-A7B0-4AFB-B840-263A9980023A

<hasgappqba2302> WINRM RESULT u'<Response code 1, out "", err "Exception calling "R">'

<hasgappqba2302> WINRM STDOUT                                                         

<hasgappqba2302> WINRM STDERR Exception calling "Run" with "1" argument(s): "Exception calling "Invoke" with "0" argument(s): "The running command                                                                                                                                                   

stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: Access is denied""                                      

At line:47 char:5                                                                                                                                           

+     $output = $entrypoint.Run($payload)                                                                                                                   

+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                                                                                   

    + CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException                                                                      

    + FullyQualifiedErrorId : ScriptMethodRuntimeException                                                                                                  

                                                                                                                                                            

[J Hawkesworth]

Can you share your playbook or tasks to reproduce this, the error message here isn't enough on its own to work out what is needed?

Many thanks,

Jon

[Jeremie Levy]

Of course

The inventory file is just hostnames so i don't think there is a need to post it.

# cat group_vars/windows.yml

# it is suggested that these be encrypted with ansible-vault:

# ansible-vault edit group_vars/windows.yml

ansible_user: ansible

#password goes here when you don't want to use -k option.

ansible_password: "PASSWORD_HERE"

ansible_port: 5986

ansible_connection: winrm

ansible_winrm_transport: kerberos

ansible_winrm_scheme: https

ansible_winrm_server_cert_validation: ignore

ansible_winrm_kerberos_delegation: true

# cat vscode.yaml

---

- hosts: windows

tasks:

- name: Install Visual studio Code

win_package:

path: \\some\dfs\path\VSCodeSetup-1.8.1.exe

creates_path: "C:\\Program Files\\Microsoft VS Code\\Code.exe"

arguments: /verysilent

state: present

Thanks.

[Jordan Borean]

What I would try so that we can narrow down the issue

* Run win_package but set the path to a local path and see if that works
* Run a win_stat of the network path executable and see if it sees the file (exists is True)
* Use the username/password options and set it to "{{ ansible_user }}" and "{{ ansible_password }}" respectively as that will be used when copying the file locally
* If you are on Ansible 2.5 (or the devel branch), see if become works it should run the module as you would when logged on locally

Thanks

Jordan

[Jeremie Levy]

Win_package is working fine locally. I'm downloading all the SW i need and install them locally

win_stat:

TASK [Check Visual studio Code file] ***************************************************************************************************************************************************************************************************************************************************************************************

task path: /ansible/scripts/win_stat.yaml:4

Using module file /usr/lib/python2.7/dist-packages/ansible/modules/windows/win_stat.ps1

Using module file /usr/lib/python2.7/dist-packages/ansible/modules/windows/win_stat.ps1

<hasgappqba2302.domain> ESTABLISH WINRM CONNECTION FOR USER: ansible@domain on PORT 5986 TO hasgappqba2302.domain

<hasgappqba2303.domain> ESTABLISH WINRM CONNECTION FOR USER: ansible@domain on PORT 5986 TO hasgappqba2303.domain

creating Kerberos CC at /tmp/tmpMi3utN

creating Kerberos CC at /tmp/tmpUIFzBS

calling kinit for principal ansible@domain

calling kinit for principal ansible@domain

kinit succeeded for principal ansible@domain

<hasgappqba2302.domain> WINRM CONNECT: transport=kerberos endpoint=https://hasgappqba2302.domain:5986/wsman

kinit succeeded for principal ansible@domain

<hasgappqba2303.domain> WINRM CONNECT: transport=kerberos endpoint=https://hasgappqba2303.domain:5986/wsman

<hasgappqba2302.domain> WINRM OPEN SHELL: FA267E6D-94C1-4BF1-AD41-75900BCC25DC

<hasgappqba2303.domain> WINRM OPEN SHELL: 47C7EA5A-202C-4560-B54B-B614A3C8B9C7

EXEC (via pipeline wrapper)

EXEC (via pipeline wrapper)

<hasgappqba2302.domain> WINRM EXEC 'PowerShell' ['-NoProfile', '-NonInteractive', '-ExecutionPolicy', 'Unrestricted', '-']

<hasgappqba2303.domain> WINRM EXEC 'PowerShell' ['-NoProfile', '-NonInteractive', '-ExecutionPolicy', 'Unrestricted', '-']

<hasgappqba2303.domain> WINRM RESULT u'<Response code 1, out "{"changed":false,"st", err "Test-Path : Access i">'

<hasgappqba2303.domain> WINRM STDOUT {"changed":false,"stat":{"exists":false}}

<hasgappqba2303.domain> WINRM STDERR Test-Path : Access is denied

At line:91 char:5

+ If (Test-Path -Path $path)

+     ~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : PermissionDenied: (\\some\dfs\path\j...Setup-1.8.1.exe:String) [Test-Path], UnauthorizedAc

   cessException

    + FullyQualifiedErrorId : ItemExistsUnauthorizedAccessError,Microsoft.PowerShell.Commands.TestPathCommand

<hasgappqba2303.domain> WINRM CLOSE SHELL: 47C7EA5A-202C-4560-B54B-B614A3C8B9C7

ok: [hasgappqba2303.domain] => {

    "changed": false,

    "stat": {

        "exists": false

    }

}

<hasgappqba2302.domain> WINRM RESULT u'<Response code 1, out "{"changed":false,"st", err "Test-Path : Access i">'

<hasgappqba2302.domain> WINRM STDOUT {"changed":false,"stat":{"exists":false}}

<hasgappqba2302.domain> WINRM STDERR Test-Path : Access is denied

At line:91 char:5

+ If (Test-Path -Path $path)

+     ~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : PermissionDenied: (\\some\dfs\path\j...Setup-1.8.1.exe:String) [Test-Path], UnauthorizedAc

   cessException

    + FullyQualifiedErrorId : ItemExistsUnauthorizedAccessError,Microsoft.PowerShell.Commands.TestPathCommand

<hasgappqba2302.domain> WINRM CLOSE SHELL: FA267E6D-94C1-4BF1-AD41-75900BCC25DC

ok: [hasgappqba2302.domain] => {

    "changed": false,

    "stat": {

        "exists": false

    }

}

META: ran handlers

META: ran handlers

PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************************************************************************

hasgappqba2302.domain : ok=2    changed=0    unreachable=0    failed=0

hasgappqba2303.domain : ok=2    changed=0    unreachable=0    failed=0

I am not sure how to use ansible_user variable, in the inventory file or the playbook ? 

fatal: [hasgappqba2303.domain]: FAILED! => {

    "failed": true,

    "msg": "Internal Error: this connection module does not support running commands via sudo"

}

With:

tasks:

- name: Check Visual studio Code file

win_stat:

path: \\some\dfs\path\VSCodeSetup-1.8.1.exe

register: result

state: present

become: yes

become_user: ansible

I've downloaded version 2.5 and this is the result i got...

Any direction?

[Jeremie Levy]

Also Tried

---

- hosts: windows

gather_facts: no

vars:

- username: ansible@DOMAIN

- password: password

become_method: runas

tasks:

- name: Check Visual studio Code file

win_stat:

path: \\some\dfs\path\VSCodeSetup-1.8.1.exe

register: result

state: present

become: yes

And got:

ansible-playbook 2.5.0 (devel 575e9e9294) last updated 2018/02/06 08:39:18 (GMT +300)                                       

  config file = /ansible/scripts/ansible.cfg                                                              

  configured module search path = [u'/home/ansible/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']

  ansible python module location = /ansible/scripts/ansible/lib/ansible                                        

  executable location = /ansible/scripts/ansible/bin/ansible-playbook                                          

  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]                                                  

Using /ansible/scripts/ansible.cfg as config file                                                              

setting up inventory plugins                                                                                                     

Parsed /scripts/windows-inventory.yml inventory source with yaml plugin                                

Loading callback plugin default of type stdout, v2.0 from /ansible/scripts/ansible/lib/ansible/plugins/callback/default.pyc

<hasgappqba2303.domain> WINRM STDERR

<hasgappqba2303.domain> WINRM CLOSE SHELL: AC3CF2B5-A044-43DF-A24B-53DB23624226

The full traceback is:

Exception calling "RunAsUser" with "7" argument(s): "LogonUser failed (The user name or password is incorrect,

Win32ErrorCode 1326)"

At line:1062 char:9

+         $result = [Ansible.BecomeUtil]::RunAsUser($username, $passwor ...

+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : Win32Exception

fatal: [hasgappqba2303.domain]: FAILED! => {

    "changed": false,

    "failed": true,

    "msg": "Exception calling \"RunAsUser\" with \"7\" argument(s): \"LogonUser failed (The user name or password is incorrect, Win32ErrorCode 1326)\""

}

[Jeremie Levy]

I've checked the eventlog and it appears that ansible was trying to become root instead of the username from the vars.

I've dig again here to find some examples of become and i found one of your answers: https://groups.google.com/forum/#!searchin/ansible-project/windows$20become%7Csort:date/ansible-project/g205HMIEjws/tYYMEzlSBQAJ

Bottom line, it's working like that - and using the -K flag @ command line.

---

- hosts: windows

gather_facts: no

tasks:

- name: Check Visual studio Code file

win_stat:

path: \\some\dfs\path\VSCodeSetup-1.8.1.exe

register: result

state: present

become: yes

become_method: runas

become_user: ansible@domain

Thank you Jordan!