ZODB fix

[Satay Epic]

One of my Zenoss 4.2.5 instances, I can't  delete devices. I get this error

File "/opt/zenoss/lib/python/Products/PluginIndexes/PathIndex/PathIndex.py", line 103, in index_object raise TypeError('path value must be string or tuple of strings') TypeError: path value must be string or tuple of strings​

Apparently , a team member created a ZenPack or something in that area. Since then, I've been seeing this error.  Even Modeler is not working. There maybe a damage somewhere else I think.

I suspect there is something wrong in the ZoDB.  I understand it is Python Object based DB. It does look a bit complicated from structure point of view.

So the question is there a way I track down the cause of the trouble and fix it?

Please advise,

Thanks

[Jane Curry]

Hmmm - could be nasty.  Do you know how many devices are affected by this problem??

Almost certainly you do have a ZODB corruption and it may well be caused by the zenpack - what is it supposed to do??  Does the ZenPack have a modeler plugin? 

Is your error line above all that you see or is there more of a stacktrace that might point us more closely at your ZenPack? 

You might try picking an affected device and try running the zenpack modeler against it in debug mode - if your Zenpack modeler is called xyzzy and your test device is fred.skills-1st.co.uk, then try:

zenmodeler run -v 10 -d fred.skills-1st.co.uk --collect xyzzy > /tmp/fred 2>&1

That should put the modeler output into /tmp/fred and might offer more clues.

Cheers,

Jane

[Satay Epic]

The error is at the bottom of a long trace seen while running modeler or trying to delete a device in the logs. In UI I see the "yellow" band of the error "TypeError: path value must be string or tuple of strings​".

I think all devices are affected with this problem. Since the class is PathIndex, I suspect the ZODB corruption happened at some device path object level.  The device path is split into its components and the level of each component which is supposed to be a "string" or "tuple of strings"

Apparently the zenpack was written for setting up email delivery and it doesn't seem using a modeler plugin. Not sure if the zenpack caused the problem as I saw around that time, "zodbscan" was run with fix=yes  and  as per the log had reported broken paths for 2 devices prior the fix.

Those 2 devices are no longer available in UI search.

I wonder if there a way to look into ZODB via dmd or sql queries to find out the corrupted paths and fix it...   One of them would be required to fix the trouble anyway, I think.

Thanks

[Jane Curry]

So we still haven't got the Zenoss wiki back - still a WIP.  But you can get to a copy of the RemoveDevices Troubleshooting tip at the wayback machine here -

https://web.archive.org/web/20190904175729/http://wiki.zenoss.org/RemoveDevices

Some of the graphics are missing but the zendmd hints are there and you might try some of those little scripts to examine devices in the ZODB database.

Cheers,

Jane

[Satay Epic]

This is what I see in the event.log when I try to maneuver to the OS Components ( Interface, Filesystem ). Going to check out the scripts and see if I can find out the distorted "path". Thanks

/opt/zenoss/log# tailf event.log
  File "/opt/zenoss/Products/ZenModel/Linkable.py", line 27, in index_links
    cat.catalog_object(self, self.getPrimaryId())
  File "/opt/zenoss/lib/python/Products/ZCatalog/ZCatalog.py", line 476, in catalog_object
    update_metadata=update_metadata)
  File "/opt/zenoss/lib/python/Products/ZCatalog/Catalog.py", line 339, in catalogObject
    blah = x.index_object(index, object, threshold)
  File "/opt/zenoss/lib/python/Products/PluginIndexes/PathIndex/PathIndex.py", line 102, in index_object

    raise TypeError('path value must be string or tuple of strings')
TypeError: path value must be string or tuple of strings

------
2022-12-16T10:51:44 INFO extdirect Direct request failed: path value must be string or tuple of strings: DeviceRouter.getComponents [{'sort': 'name', 'uid': '/zport/dmd/Devices/Server/Linux/VM/test/DC1/devices/test-03', 'keys': ['uid', 'name', 'status', 'severity', 'usesMonitorAttribute', 'monitor', 'monitored', 'locking', 'mount', 'totalBytes', 'availableBytes', 'usedBytes', 'capacityBytes', 'uuid', 'uid', 'meta_type', 'monitor'], 'start': 0, 'meta_type': 'FileSystem', 'limit': 50, 'page': 1, 'dir': 'ASC', 'name': None}]
------
2022-12-16T10:51:44 INFO extdirect DirectRouter suppressed the following exception (Response 582833b3-a1d4-42c6-a203-f07db4e7629a):
Traceback (most recent call last):
  File "/opt/zenoss/Products/ZenUtils/extdirect/router.py", line 179, in _processDirectRequest
    response.result = _targetfn(**data)
  File "<string>", line 2, in getComponents
  File "/opt/zenoss/Products/Zuul/decorators.py", line 121, in serviceConnectionError
    return func(*args, **kwargs)
  File "/opt/zenoss/Products/Zuul/routers/device.py", line 145, in getComponents
    limit=limit, sort=sort, dir=dir, name=name, keys=keys)
  File "/opt/zenoss/Products/Zuul/facades/devicefacade.py", line 202, in getComponents
    sort, dir, name=name, keys=keys)
  File "/opt/zenoss/Products/Zuul/facades/devicefacade.py", line 172, in _componentSearch
    obj.device()._createComponentSearchPathIndex()
  File "/opt/zenoss/Products/ZenModel/Device.py", line 531, in _createComponentSearchPathIndex
    c.index_object(idxs=[indexName])
  File "/opt/zenoss/Products/ZenModel/IpInterface.py", line 189, in index_object
    ip.index_object()
  File "/opt/zenoss/Products/ZenModel/IpAddress.py", line 239, in index_object
    self.index_links()
  File "/opt/zenoss/Products/ZenModel/Linkable.py", line 27, in index_links
    cat.catalog_object(self, self.getPrimaryId())
  File "/opt/zenoss/lib/python/Products/ZCatalog/ZCatalog.py", line 476, in catalog_object
    update_metadata=update_metadata)
  File "/opt/zenoss/lib/python/Products/ZCatalog/Catalog.py", line 339, in catalogObject
    blah = x.index_object(index, object, threshold)
  File "/opt/zenoss/lib/python/Products/PluginIndexes/PathIndex/PathIndex.py", line 102, in index_object

    raise TypeError('path value must be string or tuple of strings')
TypeError: path value must be string or tuple of strings

[Jane Curry]

Have you tried running the toolbox tools to correct this???  As per the recent thread here, if you are installing the tools on a 4.2.5 Zenoss, make sure you get the 2.0.0 version of the tools which you can get from here -   https://github.com/jcurry/Zenoss-Toolbox  .

Cheers,

Jane