[sheepdog-ng] unhandled error No object found

[Valerio Pachera]

Hi, I have a production cluster with 3 nodes.
One of the node went off because of power failure.
As of now, it's still not part of the cluster.

The reason is I noticed error messages in sheep.log.

Notice these message are not related to the power loss of today.

Jul 28 11:38:21   INFO [main] recover_object_main(864) object recovery progress  97%
Jul 28 11:38:21   INFO [main] recover_object_main(864) object recovery progress  98%
Jul 28 11:38:21   INFO [main] recover_object_main(864) object recovery progress  99%
Oct 24 09:46:07  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Oct 24 10:43:06  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Oct 24 11:19:12  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Oct 24 11:26:09  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Oct 24 21:35:30  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c

<cut>

Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c
Nov 01 21:27:34  ERROR [main] gateway_op_done(131) unhandled error No object found, oid 80a86f0000a62c

The other two node completed the recovery succesfully and there are no error reported.

What do you suggest to do?

Shall I run vdi check before add back the node?

dog node info -v
dog: invalid option -- 'v'
Try 'dog --help' for more information.
dog cluster info -v
Cluster status: running, auto-recovery enabled
Cluster store: plain with 2 redundancy policy
Cluster vnode mode: disk
Cluster created at Tue Mar  8 17:46:48 2016
Epoch Time           Version
2016-11-03 10:28:10      7 [192.168.6.111:7000(1), 192.168.6.112:7000(1)]
2016-07-25 15:40:45      6 [192.168.6.80:7000(1), 192.168.6.111:7000(1), 192.168.6.112:7000(1)]
2016-07-19 22:33:22      5 [192.168.6.80:7000(1), 192.168.6.111:7000(1)]
2016-03-08 19:14:52      4 [192.168.6.80:7000(1), 192.168.6.111:7000(1), 192.168.6.112:7000(1)]
2016-03-08 19:13:55      3 [192.168.6.111:7000(1), 192.168.6.112:7000(1)]
2016-03-08 19:13:31      2 [192.168.6.80:7000(1), 192.168.6.111:7000(1), 192.168.6.112:7000(1)]
2016-03-08 17:46:49      1 [192.168.6.111:7000(1), 192.168.6.112:7000(1)]

dog node info
Id      Size    Used    Avail   Use%
 0      3.4 TB  211 GB  3.2 TB    6%
 1      3.4 TB  211 GB  3.2 TB    6%
Total   6.8 TB  423 GB  6.4 TB    6%                                                                                                                                                            
Total virtual image size        2.0 TB        
                                                                                      
dog node list                                                                                                                                                                 
  Id   Host:Port         V-Nodes       Zone                                                                                                                                                    
   0   192.168.6.111:7000       881 1862707392                                                                                                                                                 
   1   192.168.6.112:7000       867 1879484608

[Liu Yuan]

for the error message, I'd suggest grep the two remaining nodes for 80a86f0000a62c manually and see if it exists on the local disks firstly . But either yes or no, I think you can simply add the node back. 

--
You received this message because you are subscribed to the Google Groups "sheepdog-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sheepdog-ng+unsubscribe@googlegroups.com.
To post to this group, send email to sheep...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sheepdog-ng/CAHS0cb-7aGY7maSnXZJGpaQyVEwLh90SF7KbY-noP%2BoJs-5BdQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Valerio Pachera]

2016-11-03 12:05 GMT+01:00 Liu Yuan <namei...@gmail.com>:

for the error message, I'd suggest grep the two remaining nodes for 80a86f0000a62c manually and see if it exists on the local disks firstly . But either yes or no, I think you can simply add the node back.

The object doesn't exist on any of the nodes.

I'll add the node back probably this evening/night.

Thank you.