Duplicate agent identifier detected

[Carl Reid]

I have 3 agents installed on a single machine.

I keep getting a warning that there is a duplicate agent uuid detected:

The warning is

[Agent located at [sd-trcd-srv, 10.129.0.7, C:\GoAgent 2]] has duplicate unique identifier which conflicts with [Agent located at [, Unknown, ]] [Jun-08 14:49:24]

If I look in the log I see more detail as shown below.

I have checked the guid.txt files on the machine and confirm that each of the UUIDs are different.

I have also tried 

1. deleting all the machines from GO and letting them re-register themselves
   
2. stopping the services and removing the guid.txt files and restarting the service

None of the actions fixes the problem.

It seems that the duplicate uuid is stored somewhere else, perhaps in the internal database that I don;t seem to have any way of looking at?

Any ideas welcome

Carl

Log file from the GO Server:

2015-06-08 12:39:19,031  WARN [16430402@qtp-17709542-5541] RemoteInvocationTraceInterceptor:87 - Processing of HttpInvokerServiceExporter remote call resulted in fatal exception: com.thoughtworks.go.remote.BuildRepositoryRemote.ping

org.springframework.remoting.RemoteAccessException: Agent [Agent [sd-trcd-srv, 10.129.0.7, ed78123c-3f70-424d-9b3c-097a9e0f752b, 12597443-e5bb-4b1e-9c87-14408d59e9c1]] has invalid cookie; nested exception is com.thoughtworks.go.server.service.AgentWithDuplicateUUIDException: Agent [Agent [sd-trcd-srv, 10.129.0.7, ed78123c-3f70-424d-9b3c-097a9e0f752b, 12597443-e5bb-4b1e-9c87-14408d59e9c1]] has invalid cookie

at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.wrappedException(BuildRepositoryRemoteImpl.java:125)

at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.ping(BuildRepositoryRemoteImpl.java:56)

at com.thoughtworks.go.server.messaging.BuildRepositoryMessageProducer.ping(BuildRepositoryMessageProducer.java:45)

at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)

at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

at $Proxy55.ping(Unknown Source)

at sun.reflect.GeneratedMethodAccessor1109.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:205)

at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:38)

at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)

at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114)

at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:73)

at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)

at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)

at com.thoughtworks.go.server.web.i18n.LocaleResolver.doFilter(LocaleResolver.java:41)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:359)

at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)

at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)

at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)

at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:381)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.web.FlashLoadingFilter.doFilter(FlashLoadingFilter.java:43)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)

at org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101)

at org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.ui.x509.X509ProcessingFilter.doFilter(X509ProcessingFilter.java:137)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.web.i18n.LocaleResolver.doFilter(LocaleResolver.java:41)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.security.ArtifactSizeEnforcementFilter.doFilter(ArtifactSizeEnforcementFilter.java:76)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.security.ModeAwareFilter.doFilter(ModeAwareFilter.java:48)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at com.thoughtworks.go.server.security.PerformanceLoggingFilter.doFilter(PerformanceLoggingFilter.java:50)

at org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:371)

at org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:174)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)

at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)

at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)

at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)

at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:939)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)

at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Caused by: com.thoughtworks.go.server.service.AgentWithDuplicateUUIDException: Agent [Agent [sd-trcd-srv, 10.129.0.7, ed78123c-3f70-424d-9b3c-097a9e0f752b, 12597443-e5bb-4b1e-9c87-14408d59e9c1]] has invalid cookie

at com.thoughtworks.go.server.service.AgentService.updateRuntimeInfo(AgentService.java:293)

at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.ping(BuildRepositoryRemoteImpl.java:53)

... 73 more

[Ketan Padegaonkar]

Normally, this happens if you forget to restart the bootstrapper, or the agent bootstrapper refuses to restart.

Can you kill all agents and start them after verifying the guid?

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ketan Padegaonkar]

Also, what might help is checking the working directory from where each agent process runs from.

[Gergely Brautigam]

Hi,

When you created the new agents, did you define a proper working directory, and an Agent NAME for the new service which you created? 
Did you delete the txt file and also the .agent-bootstrapper.running file while creating the new service?

And, as Ketan said, restarting helps. ;)

Gergely. 

[Aravind SV]

In 10.129.0.7, are there any rogue java processes? I don't know if this is Windows or Linux, but if it is linux, a pgrep -lf java might help. If on Windows, using Microsoft procexp to see whether there is a rogue java process might help.

If you stop everything on that IP and you still see this, then maybe some other machine thinks it is 10.129.0.7?

[Carl Reid]

I have many examples of multiple agents working on other machines so I really can't explain what is happening with this one. 

All 3 agents are running as LOCAL SYSTEM and were created by copying the agent directory for the first install and removing the guid.txt files and the .agent-bootstrapper.running file. I then created new services for each of them specifying the new copied directories as the location of the exe and config. Each version has it's own JRE directory.

I have just tried to fix this again with the same result.

Here is what I did: (this is a Windows 2012 R2 server)

1. Stopped all three services on the machine
2. Deleted the guid.txt file from the config directory
3. Reset the permissions on all files under the sandbox location so the LOCAL SYSTEM has full control
4. Restarted the services and observed that 3 new guid.txt files were created with new unique guids
   
5. Disabled the three old agents on the GO Server UI

After approx 2 mins the warning has re-appeared.

This is proving a tricky and frustrating problem! Where does GO keep the details of the agents? I know they are stored in the database somewhere as well as in the config.xml file.

The entries in the config xml all show the correct, unique UUIDs for the agents.

Thanks

Carl

[Gergely Brautigam]

On Monday, 8 June 2015 17:56:47 UTC+2, Carl Reid wrote:

I have many examples of multiple agents working on other machines so I really can't explain what is happening with this one. 

All 3 agents are running as LOCAL SYSTEM and were created by copying the agent directory for the first install and removing the guid.txt files and the .agent-bootstrapper.running file. I then created new services for each of them specifying the new copied directories as the location of the exe and config. Each version has it's own JRE directory.

I have just tried to fix this again with the same result.

Here is what I did: (this is a Windows 2012 R2 server)

1. Stopped all three services on the machine
2. Deleted the guid.txt file from the config directory
3. Reset the permissions on all files under the sandbox location so the LOCAL SYSTEM has full control
4. Restarted the services and observed that 3 new guid.txt files were created with new unique guids
   
5. Disabled the three old agents on the GO Server UI

Did you edit the wrapper-agent.conf file to specify the names of the agents and such?

G.
 

[Carl Reid]

I do modify this file for changing the ports used however I have never modified it when setting up multiple agents on a machine.

I use multiple agents on most of our servers and workstations and this has never caused an issue before.

Looking at the file, I see a comment block that says:

#********************************************************************
# WARNING - Do not modify any of these properties when an application
#  using this configuration file has been installed as a service.
#  Please uninstall the service before modifying this section.  The
#  service can then be reinstalled.

This makes me believe that the file is only used when the service is initially created. Is that true?

It would be very useful to know more about this file and what its purpose is. Is there any reference documentation for this file?

I assume the propertiees you are referring to are these:

# Name of the service
wrapper.ntservice.name=Go Agent


# Display name of the service
wrapper.ntservice.displayname=Go Agent


# Description of the service
wrapper.ntservice.description=Go Agent

I can rename the agents (2 and 3) in the file however I can't see this fixing it as I have never done this before on any other machine. What do these settings actually do?

Thanks for your help with this.

Carl

[Gergely Brautigam]

On Tuesday, 9 June 2015 14:15:14 UTC+2, Carl Reid wrote:

I do modify this file for changing the ports used however I have never modified it when setting up multiple agents on a machine.

I use multiple agents on most of our servers and workstations and this has never caused an issue before.

Looking at the file, I see a comment block that says:

#********************************************************************
# WARNING - Do not modify any of these properties when an application
#  using this configuration file has been installed as a service.
#  Please uninstall the service before modifying this section.  The
#  service can then be reinstalled.

This makes me believe that the file is only used when the service is initially created. Is that true?

It would be very useful to know more about this file and what its purpose is. Is there any reference documentation for this file?

I assume the propertiees you are referring to are these:

# Name of the service
wrapper.ntservice.name=Go Agent


# Display name of the service
wrapper.ntservice.displayname=Go Agent


# Description of the service
wrapper.ntservice.description=Go Agent

I can rename the agents (2 and 3) in the file however I can't see this fixing it as I have never done this before on any other machine. What do these settings actually do?

Well, they influence the service which you are creating, the name of that service. I though it might influence the uid as well, but since you have it running on other servers and there seem to be no problem like on this one, I don't think it would help.

Some other component of that particular machine must be influencing the service. Are you using any logon feature of the service by specifying an explicit user which the service should use to run with?

G.
 

[Gergely Brautigam]

I just encountered your problem!!!

But for me, it was, because my previous agent exited by interrupt and is / was hanging and gocd still sees it.

So look for something like that. Hanging Agent processes and the likes.

G.

[Carl Reid]

All the GO Agents run as LOCAL SYSTEM with "interact with desktop" set. This is the same behaviour as running a single agent.

On other machines we do run some of the agents as domain users so they can work with Windows authentication but that doesn't apply in this case.

Can anyone else offer any suggestions here?

Thanks

On Monday, 8 June 2015 14:59:42 UTC+1, Carl Reid wrote:

[Aravind SV]

Hi Carl,

Here is some documentation about installing multiple agents as services, on Windows. I believe that works.

You'd asked about where Go stores information about agents. It does store it in the database, in the AGENTS table and in memory. However, you shouldn't need to delve into it, to figure out this problem. I've never had to (there's a first time for everything, I know). If you really need to, you should bring down the server, take a copy of the database file (db/h2db/cruise.h2.db) and work with that, instead of using the original file. You can find the h2-1.3.168.jar file inside the "work" folder, and then run it with "java -jar" and tell it to open up the DB file, once it brings up its interface.

However, I'd start with this instead:

1. Isolate the problem to one agent. Instead of starting 3 agents, start one. Wait for the 2 minutes you waited earlier.

2. Does it come back and cause the same issue? If so, it's easier to check with one moving part than 3.

3. If not, bring up another agent. See if the problem comes up. If so, somehow the agents are messing each other up.

If the problem happens with one agent, I'd install procexp and check which java processes are running, where they are running and which user they're running as. If Go says there's a conflict, then there should be another agent process which has the same directory, somehow.

I'd then check the service created, to see what it calls. I guess the service is created by you, since Go Agent does not install multiple services, without external help.

I hope you get to the root of the issue. As you said, you're running it yourself in other machines. So, this should work. I'm sure there's an explanation.

Regards,

Aravind

PS: The documentation I linked to does mention changing wrapper-agent.conf.

[Carl Reid]

Thank you for the detailed reply Aravind.

I will take another look at this in more detail as soon as time allows.

In respect of the points raised by Gergely Brautigam about changing the service name in changing wrapper-agent.conf, the documentation you linked to does not mention this at all.

I have a script to install multiple agents which I have ran successfully (I believe!) on many other machines and I have never changed the following values:

# Name of the service
wrapper.ntservice.name=Go Agent


# Display name of the service
wrapper.ntservice.displayname=Go Agent


# Description of the service
wrapper.ntservice.description=Go Agent

What are these values used for and should they be changed?

If I can get more information on this I will happily update the documentation on this myself along with some other issues I have found when working with Window services.

Unfortunately the wrapper aspects of how the GO service works seem to be a mystery with little in the way of documentation.

Thanks

Carl

[Aravind SV]

On Thu, Jun 11, 2015 at 5:36 AM, Carl Reid <carland...@gmail.com> wrote:

In respect of the points raised by Gergely Brautigam about changing the service name in changing wrapper-agent.conf, the documentation you linked to does not mention this at all.

I don't know if you were taking about the documentation I linked to, but I think it does mention wrapper-agent.conf. Here is a screenshot:

It doesn't say anything about changing the name. I don't think the name needs to be changed.

[Carl Reid]

Yes that is the documentation I mentioned.

 

I am curious to know why those properties exist in the wrapper-agent.conf if they are not used at all.

 

If they do not do anything then perhaps we can remove them from the file?

I have never changed them in previous changes I have made however it does seem strange that it is specifically asking for the Windows service name which, when you have multiple agents is different.

 

How can we get some clarity on what those properties are used for?

 

Thanks

 

Carl

--
You received this message because you are subscribed to a topic in the Google Groups "go-cd" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/go-cd/u8frKFxYBYg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to go-cd+un...@googlegroups.com.

[Aravind SV]

On Thu, Jun 11, 2015 at 10:02 AM, Carl Reid <carland...@gmail.com> wrote:

I am curious to know why those properties exist in the wrapper-agent.conf if they are not used at all.

[snip] 

How can we get some clarity on what those properties are used for?

The wrapper is actually the "Java Service Wrapper" from Tanuki. There's a lot of documentation about it. See this, for example, and this. You won't see any code for it in the Go codebase, because it's the wrapper whose config file it is.

[Darly Senecal-Baptiste]

What would happen for those agents installed at Linux server via zip file?

[Darly Senecal-Baptiste]

In case of Linux servers, you can perform by following the information here