When do new nodes show up?

[H4nd0]

OK - so I created the Manager and added two nodes.

Then later on in the day I added two more nodes to the cluster.

They never appeared in the Manager.

Checking the agent.log file all is good. I've rebooted everything.
Still no sign of the two additional nodes:

2011-06-03 05:14:55+1000 [-] Log opened.
2011-06-03 05:14:55+1000 [-] twistd 9.0.0 (/opt/enomalism2/dist/py
2.4.3) starting up.
2011-06-03 05:14:55+1000 [-] reactor class:
twisted.internet.selectreactor.SelectReactor.
2011-06-03 05:14:55+1000 [-] e2_pdaemon.agent.http.AgentSite starting
on 8082
2011-06-03 05:14:55+1000 [-] Starting factory
<e2_pdaemon.agent.http.AgentSite instance at 0x95d0128>
2011-06-03 05:14:55+1000 [-] twisted.web2.channel.http.HTTPFactory
starting on 8083
2011-06-03 05:14:55+1000 [-] Starting factory
<twisted.web2.channel.http.HTTPFactory instance at 0x9817518>
2011-06-03 05:14:55+1000 [-] Starting factory
<twisted.words.protocols.jabber.xmlstream.XmlStreamFactory object at
0x95c9750>
2011-06-03 05:14:55+1000 [XmlStream,client] [WARNING -
e2_pdaemon.agent.xmpp.XMPPAgent] Pingback is disabled; agent will not
detect zombie xmpp connections.
2011-06-03 05:14:55+1000 [XmlStream,client] [WARNING -
e2_pdaemon.agent.xmpp.XMPPAgent] Sending stanza on agent_info of

Lots of errors in log files on the manager:

manager.log:

2011-06-02 21:47:37+1000 [HTTPChannel,150,127.0.0.1] [WARNING -
turbogears.identity.soprovider] No such user:

info.log

2011-06-02 21:47:37,508 cherrypy.msg INFO : Error getting csrf token
cannot concatenate 'str' and 'NoneType' objects

debug.log

2011-06-02 18:36:00,534 enomalism2.exceptions ERROR Machine
24bbd30c-8cef-11e0-bf41-005056b50032 not found.
2011-06-02 18:58:02,679 enomalism2.modules.usercontrol.controllers
INFO new group values: {}
2011-06-02 21:45:37,903 enomalism2.modules.usercontrol.controllers
INFO new group values: {}

spotcloud.log

File "turbogears/startup.py", line 350, in start_server
File "cherrypy/_cpserver.py", line 72, in start
File "cherrypy/_cpengine.py", line 106, in start
File "cherrypy/_cpserver.py", line 81, in _start
File "cherrypy/_cpengine.py", line 117, in block
File "cherrypy/_cpengine.py", line 24, in SIGTERM
AttributeError: 'module' object has no attribute 'engine'
18:14:23.972: warning : virProxyReadClientSocket:274 : Failed to read
socket 12
18:14:24.984: warning : virProxyReadClientSocket:274 : Failed to read
socket 12

What gives? Why is this product so bad? 3.5 worked fine. 3.5.1 is
junk.

[Derek Anderson]

Without the agent.log on the two machines that you have added, I won't be able to diagnose the problem. If I had to guess at a cause I would check:

 1. That each new agent has a fully qualified domain name that is reachable from both the internet and the manager

 2. That the new agent can ping the manager, and vice versa, using that hostname, as well as it's IP (make sure the hostname points at the same IP)

 3. That there is no firewalling that prevents the new agent from reaching the manager on port 5222

If those don't point out the problem, I'll need to see the contents of the agent.log on each of these new agents, including an agent restart.

I'll address these log entries all inline.

> manager.log:

> 

> 2011-06-02 21:47:37+1000 [HTTPChannel,150,127.0.0.1] [WARNING -

> turbogears.identity.soprovider] No such user:

This is probably either a mismatch between your spotcloud and ecp credentials, and the manager/ui ones. This is not an error, this is a normal output when the credentials don't match.

> 

> info.log

> 

> 2011-06-02 21:47:37,508 cherrypy.msg INFO : Error getting csrf token

> cannot concatenate 'str' and 'NoneType' objects

This is an information message that shows when you have logging turned up, and there is no current login when calculating a CSRF token. Again, not an error, just saying it cannot merge the token and "NO USER". Turning error logging back to WARN, or fixing your login will remedy this issue.

> 

> debug.log

> 

> 2011-06-02 18:36:00,534 enomalism2.exceptions ERROR Machine

> 24bbd30c-8cef-11e0-bf41-005056b50032 not found.

This is just the output when ECP scans for a machine that is off. I don't think this is a problem either.

> 

> spotcloud.log

> 

>   File "cherrypy/_cpengine.py", line 24, in SIGTERM

> AttributeError: 'module' object has no attribute 'engine'

> 18:14:24.984: warning : virProxyReadClientSocket:274 : Failed to read

> socket 12

This is two separate messages. One is from terminating the agent (the no attribute 'engine' one) that shows that the http engine has been turned off, and the other (failed to read socket 12) is a side effect of our probing the system for all potential hypervisor connections. This probing raises a warning in libvirt, which is part of the Redhat/Centos stack, and is harmless.

 - Derek