One Agent conflicting with itself

[Colin Mackay]

I have exactly one agent, yet a couple of times a week I get a message like this:

[Agent located at [ZZ26301IT, 10.1.2.109, C:\Program Files (x86)\Go Agent]] has duplicate unique identifier which conflicts with [Agent located at [ZZ26301IT, 10.1.2.109, C:\Program Files (x86)\Go Agent]] [Sep-23 09:11:00]

Previously I managed to resolve the issue by uninstalling the agent, wiping out its directory completely then reinstalling it. However, this is not a long term solution. It usually happens if the service gets restarted for some reason. e.g. Reboot of the machine. (We're still evaluating the software so it is running on a laptop - but it seems to me that it should still handle reboots in a reasonable manner)

I don't know if it is relevant, but the server log contains lots of entries like this:

2015-09-23 09:12:20,990  WARN [qtp26566101-24] AgentService:312 - Found agent [Agent [ZZ26301IT, 10.1.2.109, d74e0d8d-1c6f-4a38-b7b2-7dbd2f00585e, 88787027-dbf3-477c-95a3-618e0ffa5bd1]] with duplicate uuid. Please check the agent installation.

2015-09-23 09:12:20,990  WARN [qtp26566101-24] RemoteInvocationTraceInterceptor:87 - Processing of HttpInvokerServiceExporter remote call resulted in fatal exception: com.thoughtworks.go.remote.BuildRepositoryRemote.ping

org.springframework.remoting.RemoteAccessException: Agent [Agent [ZZ26301IT, 10.1.2.109, d74e0d8d-1c6f-4a38-b7b2-7dbd2f00585e, 88787027-dbf3-477c-95a3-618e0ffa5bd1]] has invalid cookie; nested exception is com.thoughtworks.go.server.service.AgentWithDuplicateUUIDException: Agent [Agent [ZZ26301IT, 10.1.2.109, d74e0d8d-1c6f-4a38-b7b2-7dbd2f00585e, 88787027-dbf3-477c-95a3-618e0ffa5bd1]] has invalid cookie

at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.wrappedException(BuildRepositoryRemoteImpl.java:125)

at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.ping(BuildRepositoryRemoteImpl.java:56)

at com.thoughtworks.go.server.messaging.BuildRepositoryMessageProducer.ping(BuildRepositoryMessageProducer.java:45)

at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at com.sun.proxy.$Proxy56.ping(Unknown Source)

at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:205)

at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:38)

at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)

at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114)

at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:73)

at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)

at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)

at com.thoughtworks.go.server.web.i18n.LocaleResolver.doFilter(LocaleResolver.java:41)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:359)

at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)

at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)

at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)

at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:381)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.web.FlashLoadingFilter.doFilter(FlashLoadingFilter.java:43)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)

at org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)

at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.ui.x509.X509ProcessingFilter.doFilter(X509ProcessingFilter.java:137)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.web.i18n.LocaleResolver.doFilter(LocaleResolver.java:41)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.security.ArtifactSizeEnforcementFilter.doFilter(ArtifactSizeEnforcementFilter.java:76)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.security.ModeAwareFilter.doFilter(ModeAwareFilter.java:48)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.security.PerformanceLoggingFilter.doFilter(PerformanceLoggingFilter.java:50)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:174)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:499)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Unknown Source)

Caused by: com.thoughtworks.go.server.service.AgentWithDuplicateUUIDException: Agent [Agent [ZZ26301IT, 10.1.2.109, d74e0d8d-1c6f-4a38-b7b2-7dbd2f00585e, 88787027-dbf3-477c-95a3-618e0ffa5bd1]] has invalid cookie

at com.thoughtworks.go.server.service.AgentService.updateRuntimeInfo(AgentService.java:317)

at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.ping(BuildRepositoryRemoteImpl.java:53)

... 79 more

The Agent log similarly has lots of entries like this:

2015-09-23 09:14:00,992 [pingThread] ERROR thoughtworks.go.agent.AgentController:116 - Error occurred when agent tried to ping server: 

org.springframework.remoting.RemoteAccessException: Agent [Agent [ZZ26301IT, 10.1.2.109, d74e0d8d-1c6f-4a38-b7b2-7dbd2f00585e, 88787027-dbf3-477c-95a3-618e0ffa5bd1]] has invalid cookie; nested exception is com.thoughtworks.go.server.service.AgentWithDuplicateUUIDException: Agent [Agent [ZZ26301IT, 10.1.2.109, d74e0d8d-1c6f-4a38-b7b2-7dbd2f00585e, 88787027-dbf3-477c-95a3-618e0ffa5bd1]] has invalid cookie

at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.wrappedException(BuildRepositoryRemoteImpl.java:125)

at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.ping(BuildRepositoryRemoteImpl.java:56)

at com.thoughtworks.go.server.messaging.BuildRepositoryMessageProducer.ping(BuildRepositoryMessageProducer.java:45)

at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at com.sun.proxy.$Proxy56.ping(Unknown Source)

at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:205)

at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:38)

at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)

at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114)

at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:73)

at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)

at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)

at com.thoughtworks.go.server.web.i18n.LocaleResolver.doFilter(LocaleResolver.java:41)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:359)

at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)

at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)

at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)

at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:381)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.web.FlashLoadingFilter.doFilter(FlashLoadingFilter.java:43)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)

at org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)

at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.ui.x509.X509ProcessingFilter.doFilter(X509ProcessingFilter.java:137)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.web.i18n.LocaleResolver.doFilter(LocaleResolver.java:41)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.security.ArtifactSizeEnforcementFilter.doFilter(ArtifactSizeEnforcementFilter.java:76)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.security.ModeAwareFilter.doFilter(ModeAwareFilter.java:48)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.security.PerformanceLoggingFilter.doFilter(PerformanceLoggingFilter.java:50)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:174)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:499)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

at java.lang.Thread.run(Unknown Source)

at org.springframework.remoting.support.RemoteInvocationUtils.fillInClientStackTraceIfPossible(RemoteInvocationUtils.java:47)

at org.springframework.remoting.support.RemoteInvocationResult.recreate(RemoteInvocationResult.java:115)

at org.springframework.remoting.support.RemoteInvocationBasedAccessor.recreateRemoteInvocationResult(RemoteInvocationBasedAccessor.java:85)

at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:148)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at com.sun.proxy.$Proxy5.ping(Unknown Source)

at com.thoughtworks.go.agent.AgentController.ping(AgentController.java:112)

at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)

at org.springframework.scheduling.support.MethodInvokingRunnable.run(MethodInvokingRunnable.java:65)

at org.springframework.scheduling.timer.DelegatingTimerTask.run(DelegatingTimerTask.java:70)

at java.util.TimerThread.mainLoop(Unknown Source)

at java.util.TimerThread.run(Unknown Source)

Caused by: com.thoughtworks.go.server.service.AgentWithDuplicateUUIDException: Agent [Agent [ZZ26301IT, 10.1.2.109, d74e0d8d-1c6f-4a38-b7b2-7dbd2f00585e, 88787027-dbf3-477c-95a3-618e0ffa5bd1]] has invalid cookie

at com.thoughtworks.go.server.service.AgentService.updateRuntimeInfo(AgentService.java:317)

at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.ping(BuildRepositoryRemoteImpl.java:53)

... 95 more

[Zabil C M]

Hi Colin,

  A similar problem was discussed here

  https://github.com/gocd/gocd/issues/1134

  Are you running more than one agent of the same box? 

 

 

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Zabil C M]

I meant

"Are you running more than one agent on the same box? "

[Zabil C M]

Sorry you mentioned that you have exactly one agent. My bad. 

Gotta get some coffee.

[Colin Mackay]

I am running exactly one agent in total. So, yes, I'm only running one agent on the box.

I looked at your link and it suggested that deleting the config/guid.txt file and restarting the service would work. So I 

* Stopped the Windows Go Agent Service

* renamed the guid.txt (just in case, but I suspect deleting it outright will be fine)

* Started the Windows Go Agent Service.

My agent now is now building pipelines again. Many thanks for your help - This is much easier than uninstalling and reinstalling an agent.

I'm not sure what the implications of just deleting the guid.txt on other things we've not looked at yet - we're still at the evaluation phase.

Regards,
Colin.

[Colin Mackay]

Well, this is new. I get the same error but it started in the middle of running through a pipeline. The first stage worked out okay, but it got stuck on the second stage. I don't exactly know when, I only noticed after about 30 minutes by which time it had truncated the log for that stage so I cannot see the first 15 minutes or so.

I restated the agent with a new guid (as per the previous resolution), and the pipeline restarted a few minutes after it picked up the "new" agent.

I'll try to monitor it more closely next time to see exactly when things to awry.

[Sriram Narayanan]

Hi:

I've been using Go for years, but I have not yet faced this problem. But your case interests me quite a bit since it looks like a "My Car doesn't like Vanilla Ice-Cream" problem to me.

Some questions:

1. Is there any other service between the Go agent and the server, like an HTTP proxy?

The Go agent contacts the server once in a while. At least at startup, it contacts the server via HTTP, and then switches to HTTPS. I have not yet checked the code, nor used tcpdump, not checked firewall logs, nor talked to the devs to check if there are frequent connects on HTTP as well.

2. Do you have any sort of NAT configuration?

I once ran a setup where we have multiple environments all with identical host names and IPs for the VMs within them, connecting to a single shared Go server outside of them.

I have since then worked with clients where "servers" are kept on one network, and "agents or workstations" are kept on another Network with a firewall in between. Do you have any such setup?

3. Are you certain that there isn't another Go Agent running? (e.g. an install attempt may have been cancelled but the agent may have already got installed, OR you may have tried a command line instance of this)

Alternatively, was there another computer that was initially used for a Go Agent, and then you moved to this one?

Are there other teams also trial-ing Go and pointing to the same server, per chance? At one client place, they had a bit of success with Go, and the second team wanting to try this out copied over the Go Agent folder (and the guid.txt as well!) and pointing to the same Go server :)

4. Are there multiple network paths to the server?
At my current workplace, there were some network level changes recently that have given me two different paths to the Go server from my workstation.  I'm not running a Go Agent on my workstation, though.

5. Are you exploring Auto Registration?

I'm wondering if a combination of the above could lead to the interesting that you're reporting.

Some things to try just to eliminate the possibility of a phantom agent:

1. Stop the Go agent service, disable it, reboot the computer, and check the Agent's tabl on the Go server.

Is there still an active agent in "idle" state there?

2. Change the ports the Go server is listening on.

These are 8153 and 8154 by default.

Configure the Go Agent to point to the new ports now.

Observe for some time to see if you still face the duplicate agent problem.

This may not be a fool proof check since it's possible for multiple agents to refer to the same config file due to configuration errors, for e.g.

Some things to try on the Go server when you see this problem. Please don't mind me spelling things out a bit - this mail may be useful for others too.

1. Using the Go Agent from the command line

Would it be possible for you to stop the Windows service, mark it as "disabled" in Control Panel, reboot the computer, and then start the Go agent from the command line?

2. On the Go server, please check how many agents are actually registered.

a. Use netstat to see which IPs are connecting to the Go port of 8153 and 8154.

You should turn off all the web browsers connected to the Go server (literally all of them - we've had a mis diagnosis once), and check just which computers are connected to these ports. On the Windows server hosting Go, "netstat -ban > connections.txt" run as administrator would give a list of all connections (established, listening, closing). Search for 8153 and 8154.

Do you know all the computers that are connecting to these ports? Are they supposed to be connecting to the Go service?

-- Ram

[Aravind SV]

> Some things to try just to eliminate the possibility of a phantom agent:

I'm suspecting a phantom agent as well. I'd get procexp from Microsoft sysinternals and bring down the agent and look carefully for any java processes which are alive. Run procexp as administrator, for good measure.

--

[Carl Reid]

This problem troubles us on occasion, today being one of those occasions.

We are seeing this error message:

[Agent located at [WIN-KANQRBSIE3G, 10.132.44.244, C:\GoAgent]] has duplicate unique identifier which conflicts with [Agent located at [Infrastructure - GO Build Server - WIN-KANQRBSIE3G, 10.132.44.244, C:\GoAgent]] [Sep-12 15:56:38]

So an agent is conflicting with itself. There are definitely no other agents on this machine. The servers that provide these build agents are created and destroyed on a daily basis and there are 6 of them - all built using cloud formation in exactly the same way and none of the other servers are reporting any errors.

I suspect some kind of bug in GO but I am not sure how to further diagnose the issue. If anyone can shed any light on this problem I would be very grateful.

Thanks

Carl

[Carl Reid]

This error has happened every day this week now and it's becoming frustrating

Looking through the logs it appears that Windows update is causing the servers to reboot when they are first starting and this is somehow causing errors with the agent registrations.

For now, I am having to manually fix the servers by: (I will fix this soon with a new EC2 AMI that doesn't have a pending reboot)

1. Getting the current guid from the agent
   
2. Stopping the service
3. Deleting the guid.txt file
4. Starting the service
5. Getting the new guid.txt
6. Replacing the old guid with the new one in the config.xml

This works most of the time however one server keeps throwing the following error:

[Agent located at [WIN-FT4UM0OAS63, 10.132.44.46, C:\GoAgent]] has duplicate unique identifier which conflicts with [Agent located at [, Unknown, ]] [Sep-14 11:26:52]

I am not sure what is going on here and would appreciate any thoughts.

Thanks

Carl

[Carl Reid]

This issue is now officially driving me crazy.

Can someone please offer some help as to how to fix this issue.

We seem to get this problem every day now. At least one, often more agents end up conflicting with themselves and pipelines will not run on the agent.

Sometimes I can "fix" the issue by carrying out the steps above but often this does not work and I get the conflicting with "unknown" issue 

[Agent located at [WIN-NG150RLHV9R, 10.132.44.160, C:\GoAgent]] has duplicate unique identifier which conflicts with [Agent located at [, Unknown, ]] [Sep-19 09:44:59]

Clearly there must be another place where GO holds it's agent registrations other than the xml file - I assume this is in the database somewhere. Is there a way of purging this information to fix this issue?

Thanks

[Ketan Padegaonkar]

I've looked at the code and I don't see this as a problem on the server end. It likely looks like you have multiple agents running on the agent VM. As Arvind mentions, could you run procexp on such agents as to eliminate if the issue is really multiple agent processes running on the same VM which such a problem happens again?

[Carl Reid]

Ketan, thanks for looking at this. For reference I managed to resolve this issue.

As I said we use Amazon EC2 instances created on a daily basis from an AMI. When the machines come up the agent is stopped. 

The machines run a bootstrapping PowerShell script that set the correct agent resources, host name and environment using the AutoRegister feature then starts the GO Agent service.

There was however also a scheduled task that checked the state of the agent and if it was not running it started it. This was written to deal with problem we have seen in the past with machines shutting down non-gracefully and the lockfile not being deleted. When this happens the GO Agent does not start automatically hence we created this scheduled task to check the status of the agent, delete the lock file and start the service.

It seems that the agent bootstrapping script and this scheduled task were at times causing the GO Agent service to register twice with the server. Once the scheduled task was removed we no longer see the duplicate uuid errors.

Carl