Isilon Unbalanced Storage

[sc...@tacomadata.com]

This year I was asked to help with an Isilon project for our storage team.  We were planning a datacenter move and wanted to eject the oldest tech (first 20 nodes) from a 40 node Isilon cluster.  

To make the smartfail of n1-20  complete more quickly, our techsupport had us enable a trial license for 'smartpools'  -  then configure it to move data from nodes 1-20 to nodes 21-40.  This all went well. 

The cluster has been operating on 20 nodes, the move is still pending, but here's where it gets interesting:

Last month I got a 'pool is getting full' warning.  Our smartpools license expired in July as expected. I'd noticed Autobalance and FSanalyze had failed 2x, so I started them manually (they did complete) 

I realized I'd overlooked my networked nodes 33-40 from  pool "a2000_200tb_800gb-ssd-sed_64gb" were all filled to >99% while nodes 21-32 from pool 'a40_200tb_800gb-ssd-sed_16gb'  are only filled to 36%.

I made sure AutoBalance completed correctly but I think that might have only balanced data within each nodepool.  

Here is what I wonder:  Did we inadvertently turn-off a setting that keeps whole cluster balanced when we enabled the temporary Smartpools license?  When I look here I see 'balanced' set to 'no' between nodepools.  Maybe there is a way to set this to 'yes?' :-)

# isi storagepool nodepools list -v                  ID: 19                Name: a40_200tb_800gb-ssd-sed_16gb               Nodes: 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32       Node Type IDs: 2   Protection Policy: +3d:1n              Manual: No          L3 Enabled: Yes L3 Migration Status: l3                Tier: Archive               Usage                 Avail Bytes: 1.35P             Avail SSD Bytes: 0.00                    Balanced: No                  Free Bytes: 1.36P              Free SSD Bytes: 0.00                 Total Bytes: 2.11P             Total SSD Bytes: 0.00     Virtual Hot Spare Bytes: 19.12T --------------------------------------------------------------------------------                  ID: 50                Name: a2000_200tb_800gb-ssd-sed_64gb               Nodes: 33, 34, 35, 36, 37, 38, 39, 40       Node Type IDs: 3   Protection Policy: +2d:1n              Manual: No          L3 Enabled: Yes L3 Migration Status: l3                Tier: Archive               Usage                 Avail Bytes: 390.98G             Avail SSD Bytes: 0.00                    Balanced: Yes                  Free Bytes: 19.41T              Free SSD Bytes: 0.00                 Total Bytes: 1.41P             Total SSD Bytes: 0.00     Virtual Hot Spare Bytes: 19.02T

Hope someone here can help, or give me ideas on what I can do.