[JIRA] [ec2-plugin] (JENKINS-25385) Jenkins EC2 plugin is not able to launch Windows Slaves in AWS

462 views
Skip to first unread message

araza@splunk.com (JIRA)

unread,
Oct 30, 2014, 9:07:07 PM10/30/14
to jenkinsc...@googlegroups.com
Issue Type: Bug Bug
Assignee: Francis Upton
Components: ec2-plugin
Created: 31/Oct/14 1:06 AM
Description:

https://wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+Plugin

I followed the guidelines for setting up Windows AMI. While I can see that the EC2 is launched in AWS, the instance is not able to connect to master as slave. logs show something like

'waiting for Windows RM ... going to sleep ..'

Notes for Windows AMI:
EC2 Windows slaves are accessed with CIFS (to send the initial Jenkins slave.jar) and WinRM to launch and connect to the slave afterward. This windows AMI must be configured with:

a security group allowing SMB over TCP (incoming TCP port 445) and WinRM (incoming TCP port 5985)
windows firewall should allow incoming SMB over TCP
java should be installed and available in the %PATH%
WinRM should be enabled with the following commands (for more information see: Microsoft article 555966):
winrm quickconfig
winrm set winrm/config/service/Auth @{Basic="true"}
winrm set winrm/config/service @{AllowUnencrypted="true"}
winrm set winrm/config/winrs @{MaxMemoryPerShellMB="1024"}

Project: Jenkins
Labels: plugin ec2-plugin
Priority: Critical Critical
Reporter: Ali Raza
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

thorsten.3.moellers@here.com (JIRA)

unread,
Nov 17, 2014, 5:32:08 AM11/17/14
to jenkinsc...@googlegroups.com

tcpdump output

Change By: Thorsten Möllers (17/Nov/14 10:30 AM)
Attachment: win-agent.txt

thorsten.3.moellers@here.com (JIRA)

unread,
Nov 17, 2014, 5:38:08 AM11/17/14
to jenkinsc...@googlegroups.com

I see the same problem. Tried to debug it with tcpdump, and have the following findings:

1. The TCP connection is set up, so there are no firewall issues
2. The initial http request jenkins-server ==> raised windows-node does not have any username/password for authentication
3. The windows-node replies with HTTP/1.1 401 (authentication failed)

The node logfile logs like this to me:
Node Windows build machine (i-7665db97)(i-7665db97) is still stopping, waiting 5s
[above line repeated while node boots up]
Node Windows build machine (i-7665db97)(i-7665db97) is ready
Windows build machine (i-7665db97) booted at 1415975164000
Connecting to ip-10-124-9-246.release.in.here.com(10.124.9.246) with WinRM as Administrator
Waiting for WinRM to come up. Sleeping 10s.
[above two lines repeated indefinitely]

thorsten.3.moellers@here.com (JIRA)

unread,
Nov 17, 2014, 5:44:09 AM11/17/14
to jenkinsc...@googlegroups.com
 
Thorsten Möllers edited a comment on Bug JENKINS-25385

I see the same problem. Tried to debug it with tcpdump (see attached file, win-dump.txt), and have the following findings:

1. The TCP connection is set up, so there are no firewall issues
2. The initial http request jenkins-server ==> raised windows-node does not have any username/password for authentication
3. The windows-node replies with HTTP/1.1 401 (authentication failed)

The node logfile logs like this to me:
Node Windows build machine (i-7665db97)(i-7665db97) is still stopping, waiting 5s
[above line repeated while node boots up]
Node Windows build machine (i-7665db97)(i-7665db97) is ready
Windows build machine (i-7665db97) booted at 1415975164000
Connecting to ip-10-124-9-246.release.in.here.com(10.124.9.246) with WinRM as Administrator
Waiting for WinRM to come up. Sleeping 10s.
[above two lines repeated indefinitely]

This message is automatically generated by JIRA.

kk@kohsuke.org (JIRA)

unread,
Dec 4, 2014, 8:15:08 AM12/4/14
to jenkinsc...@googlegroups.com

For people seeing this issue, please enable a custom logger for hudson.plugins.ec2.win.winrm so that we can see how it is failing to connect.

tomwar@syncron.com (JIRA)

unread,
Dec 5, 2014, 7:34:08 AM12/5/14
to jenkinsc...@googlegroups.com
 
Tomasz Warchoł edited a comment on Bug JENKINS-25385

I was getting this issue too, and just a sec ago I managed to successfully start windows slave (without https support). I did two things:
1) Help states that winrm should configure /winrm/config/service entries, but I had to run

winrm set winrm/config/client @{AllowUnencrypted="true"}
winrm set winrm/config/client/Auth @{Basic="true"}
to allow external connections. Before that I wasn't able to telnet to port 5985, after it was possible.

2) In WinRMClient.buildHTTPClient() I've removed unregister for AuthPolicy.SPNEGO and added unregister for KERBEROS, DIGEST and NTLM. Before this change there was exception about unsupported auth method, but it was getting lost somewhere in the code flow.

NOTE: I do not know what I am doing. I have very little idea about how exactly winrm works, I am just gathering info from random pages and trying to apply it. But I hope it will help to fix this properly

tomwar@syncron.com (JIRA)

unread,
Dec 5, 2014, 7:34:08 AM12/5/14
to jenkinsc...@googlegroups.com

I was getting this issue too, and just a sec ago I managed to successfully start windows slave (without https support). I did two things:


1) Help states that winrm should configure /winrm/config/service entries, but I had to run

winrm set winrm/config/client @{AllowUnencrypted="true"}
winrm set winrm/config/client/Auth @{Basic="true"}
to allow external connections. Before that I wasn't able to telnet to port 5985, after it was possible.

2) In WinRMClient.buildHTTPClient I've removed unregister for AuthPolicy.SPNEGO and added unregister for KERBEROS, DIGEST and NTLM. Before this change there was exception about unsupported auth method, but it was getting lost somewhere in the code flow.

NOTE: I do not know what I am doing. I have very little idea about how exactly winrm works, I am just gathering info from random pages and trying to apply it. But I hope it will help to fix this properly

This message is automatically generated by JIRA.

arazauci@java.net (JIRA)

unread,
Dec 5, 2014, 11:51:09 PM12/5/14
to jenkinsc...@googlegroups.com
arazauci commented on Bug JENKINS-25385

@kohsuke
I can manually telnet into the windows ec2 instance. However, Jenkins cannot seem to add it as a slave. I see the following error in the winrm logger

Request:
POST http://xxx.xxx.xx.xx:5985/wsman
<?xml version="1.0" encoding="UTF-8"?>
<env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:rsp="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xmlns="http://schemas.microsoft.com/wbem/wsman/1/wsman.xsd"><env:Header><a:To>http://xxx.xxx.xx.xx:5985/wsman</a:To><a:ReplyTo><a:Address mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</a:Address></a:ReplyTo><w:MaxEnvelopeSize mustUnderstand="true">153600</w:MaxEnvelopeSize><a:MessageID>uuid:AC60C672-A7F9-4283-B161-17B5A37A9F63</a:MessageID><w:Locale mustUnderstand="false" xml:lang="en-US"/><p:DataLocale mustUnderstand="false" xml:lang="en-US"/><w:OperationTimeout>PT60S</w:OperationTimeout><a:Action mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/09/transfer/Create</a:Action><w:ResourceURI mustUnderstand="true">http://schemas.microsoft.com/wbem/wsman/1/windows/shell/cmd</w:ResourceURI><w:OptionSet><w:Option Name="WINRS_NOPROFILE">FALSE</w:Option><w:Option Name="WINRS_CODEPAGE">437</w:Option></w:OptionSet></env:Header><env:Body><rsp:Shell><rsp:InputStreams>stdin</rsp:InputStreams><rsp:OutputStreams>stdout stderr</rsp:OutputStreams></rsp:Shell></env:Body></env:Envelope>

Dec 05, 2014 8:46:40 PM SEVERE hudson.plugins.ec2.win.winrm.WinRMClient sendRequest

I/O Exception in HTTP POST
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:244)
at hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest(WinRMClient.java:215)
at hudson.plugins.ec2.win.winrm.WinRMClient.openShell(WinRMClient.java:94)
at hudson.plugins.ec2.win.winrm.WinRM.ping(WinRM.java:29)
at hudson.plugins.ec2.win.WinConnection.ping(WinConnection.java:117)
at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:118)
at hudson.plugins.ec2.win.EC2WindowsLauncher.launch(EC2WindowsLauncher.java:29)
at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:101)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:241)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

I did what Tomasz suggested
2) In WinRMClient.buildHTTPClient() I've removed unregister for AuthPolicy.SPNEGO and added unregister for KERBEROS, DIGEST and NTLM. Before this change there was exception about unsupported auth method, but it was getting lost somewhere in the code flow.

private DefaultHttpClient buildHTTPClient()
    {
        DefaultHttpClient httpclient = new DefaultHttpClient();
        //httpclient.getAuthSchemes().unregister(AuthPolicy.SPNEGO);
        httpclient.getAuthSchemes().unregister(AuthPolicy.KERBEROS);
        httpclient.getAuthSchemes().unregister(AuthPolicy.DIGEST);
        httpclient.getAuthSchemes().unregister(AuthPolicy.NTLM);

scm_issue_link@java.net (JIRA)

unread,
Apr 11, 2015, 3:18:06 PM4/11/15
to jenkinsc...@googlegroups.com

Code changed in jenkins
User: Jason Mittertreiner
Path:
src/main/java/hudson/plugins/ec2/win/EC2WindowsLauncher.java
src/main/java/hudson/plugins/ec2/win/winrm/WinRMClient.java
http://jenkins-ci.org/commit/ec2-plugin/0e840f7129b91af5101cb8f08f938743dc188ff9
Log:
JENKINS-27260 SPNEGO for Windows in EC2 Plugin

Fixed the Windows temp directory getting set to ""
Enabled SPNEGO authentication

JENKINS-25385 and JENKINS-4995 both have comments complaining about
infinite loops when creating Windows slaves. Because SPNEGO is
unregistered for, the httpclient throws and exception that is silently
caught and causes the infinite loop.

Reply all
Reply to author
Forward
0 new messages