problematic upgrade from onefs 7.2.1.4 to 8.0.1.2

[hbdhd...@gmail.com]

I had a semi successful update; 3 out of 4 nodes went through fine, the 4th one gave me:

From email: 

Cluster Name: XXXX

Cluster GUID: XXXXX

Sender OneFS Version: Isilon OneFS v8.0.1.2 B_MR_8_0_1_2_160(RELEASE)

Sender Serial Number: SN400-XXXXXX-XXX

Node Buttons-4 (devid 4) Eventgroups

------------------------------------------------------------------------

OneFS Version: Isilon OneFS v8.0.1.2 B_MR_8_0_1_2_160(RELEASE)

Serial Number: SN400-XXXXXX-XXX

------------------------------------------------------------------------

  ID           Started        Sev  Message

------------------------------------------------------------------------

1 06/26 11:47    C    Auth upgrade encountered critical errors that prevent the Auth service from being upgraded.

Please consult log at /ifs/.ifsvar/tmp/moby-auth-upgrade.log for additional context.

The following specific error occurred: 'Input/output error

    from gci_disk_file_load (/b/mnt/src/isilon/lib/isi_gconfig/gconfig_disk.c:2706)

    from gci_disk_ns_cache_load (/b/mnt/src/isilon/lib/isi_gconfig/gconfig_disk.c:4694): error while loading context from cache file Gconfig.main_config

    from gci_disk_ctx_load (/b/mnt/src/isilon/lib/isi_gconfig/gconfig_disk.c:3033)

    from _gci_ctx_new_flags_filter_cb (/b/mnt/src/isilon/lib/isi_gconfig/gconfig_context.c:926)

    from auth_upgrade_registry (/b/mnt/src/isilon/lib/isi_auth_upgrade/auth_upgrade.cpp:433)'.

Attachment Manifest:

Attached:

Knowledge base URL(s) for event 700010005 : 

Internal link: https://xx.xx.xx.xx:8080/help/index.html#ifs_r_event_700010005.html

External link: http://doc.isilon.com/onefs/8.0.1/help/en-us/#ifs_r_event_700010005.html

The referenced log file contains nothing of interest:

STARTED: Tue Jun 26 11:48:02 2018

lsass ConfigVersion = '7.1.1'

[    DELE] registry.Services.flt_audit.Enabled

[    DELE] registry.Services.flt_audit_nfs.Enabled

[  CREATE] registry.Services.lsass.Parameters.ConfigVersion ==> '8.0'

commit ISILON-REGISTRY

STOPPED: Tue Jun 26 11:48:03 2018

STARTED: Tue Jun 26 11:48:15 2018

Skipped ISILON-REGISTRY update to 8.0 -- already updated.

STOPPED: Tue Jun 26 11:48:15 2018

STARTED: Tue Jun 26 11:48:16 2018

Skipped ISILON-REGISTRY update to 8.0 -- already updated.

STOPPED: Tue Jun 26 11:48:16 2018

STARTED: Tue Jun 26 11:48:47 2018

Skipped ISILON-REGISTRY update to 8.0 -- already updated.

STOPPED: Tue Jun 26 11:48:48 2018

STARTED: Tue Jun 26 11:49:00 2018

Idmap version already set to 2 ('zone-specific keys') - Doing nothing

STOPPED: Tue Jun 26 11:49:00 2018

Event, as described in the web interface:

Event Group Causes

WINNET_AUTH_UPGRADE_ALERT: Auth Upgrade Failure

Event Group ID 1

Severity Critical

Alert Channels No value

Event Count 1

Time Noticed 2018-06-26 13:47:46

Marked Resolved No

Ignored No

ID 4.2

Event 700010005

Time 2018-06-26 13:47:46

Node 4

Severity Critical

Message

Auth upgrade encountered critical errors that prevent the Auth service from being upgraded. Please consult log at /ifs/.ifsvar/tmp/moby-auth-upgrade.log for additional context. The following specific error occurred: 'Input/output error from gci_disk_file_load (/b/mnt/src/isilon/lib/isi_gconfig/gconfig_disk.c:2706) from gci_disk_ns_cache_load (/b/mnt/src/isilon/lib/isi_gconfig/gconfig_disk.c:4694): error while loading context from cache file Gconfig.main_config from gci_disk_ctx_load (/b/mnt/src/isilon/lib/isi_gconfig/gconfig_disk.c:3033) from _gci_ctx_new_flags_filter_cb (/b/mnt/src/isilon/lib/isi_gconfig/gconfig_context.c:926) from auth_upgrade_registry (/b/mnt/src/isilon/lib/isi_auth_upgrade/auth_upgrade.cpp:433)'.

The problem is, dell/EMC has completely ignored my Support request when I planned this upgrade and just closed it when the support contract expired (a few days ago). It seems silly to renew a contract that proved to be completely useless. They have so far either ignored or totally messed up all of our SRs. And now I'm stuck with no info on what to do about this error. Can any of you share some thoughts/recommendations/wise words? We don't use SMB, only NFS, so I'm hoping it won't really affect us but I can't be sure and don't want to have an outstanding error on the cluster. 

Thank you all in advance

[YF]

I would check another upgrade file to ensure all is good:

cat /ifs/.ifsvar/tmp/riptide-auth-upgrade.log | grep AUTH | grep complete

Example of what a successful upgrade looks like:

Node 10 -2017-07-11 16:09:50,143 - INFO - AUTH upgrade part 1 complete

Node 10 -2017-07-11 16:09:50,331 - INFO - AUTH upgrade part 2 complete

Node 10 -2017-07-11 16:09:50,345 - INFO - AUTH upgrade part 3 complete

Node 10 -2017-07-11 16:09:50,362 - INFO - AUTH upgrade part 4 complete

[hbdhd...@gmail.com]

I do have almost exactly that

# grep AUTH /ifs/.ifsvar/tmp/riptide-auth-upgrade.log | grep complete

Node 4 -2018-06-26 11:47:54,237 - INFO - AUTH upgrade part 1 complete

Node 4 -2018-06-26 11:47:54,509 - INFO - AUTH upgrade part 2 complete

Node 4 -2018-06-26 11:47:54,514 - INFO - AUTH upgrade part 3 complete

Node 4 -2018-06-26 11:47:54,574 - INFO - AUTH upgrade part 4 complete

How does it relate to my node 4's problem? 

# isi status --node=4 | grep Health

Node Health:          -A--