[Lustre-discuss] how to reuse OST indices (EADDRINUSE)
Craig Prescott
Hello list,
We recently evacuated several OSTs on a single OSS, replaced RAID
controllers, re-initialized RAIDs for new OSTs, and made new lustre
filesystems for them, using the same OST indices as we had before.
The filesystem and all its clients have been up and running the whole
time. We disabled the OSTs we were working on on all clients and our
MGS/MDS (lctl dl shows them as "IN" everywhere).
Now we want to bring the newly-formatted OSTs back online. When we try
to mount the "new" OSTs, we get this for each one in this syslog of the
OSS that has been under maintenance:
> Lustre: MGC10.13.28.210@o2ib: Reactivating import
> LustreError: 11-0: an error occurred while communicating with 10.13.28.210@o2ib. The mgs_target_reg operation failed with -98
> LustreError: 6065:0:(obd_mount.c:1097:server_start_targets()) Required registration failed for cms-OST0006: -98
> LustreError: 6065:0:(obd_mount.c:1655:server_fill_super()) Unable to start targets: -98
> LustreError: 6065:0:(obd_mount.c:1438:server_put_super()) no obd cms-OST0006
> LustreError: 6065:0:(obd_mount.c:147:server_deregister_mount()) cms-OST0006 not registered
What do we need to do to get these OSTs back into the filesystem?
We really want to reuse the original indices.
This is Lustre 1.8.4, btw.
Thanks,
Craig Prescott
UF HPC Center
_______________________________________________
Lustre-discuss mailing list
Lustre-...@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss
Wang Yibin
Did you backup old magic files (last_rcvd, LAST_ID, CONFIG/*) from the original OSTs and put them back before trying to mount them?
You probably didn't do that. So when you remount the OSTs with existing index, the MGS will refuse to add them without being told to writeconf, hence -EADDRINUSE.
The proper ways to replace an OST are described in bug 24128.
Craig Prescott
Thanks for this. You are right - we didn't back up and replace those
files. We did this once before, and I don't recall doing a writeconf or
anything with magic files, but I guess we must have.
We no longer have magic files (last_rcvd, LAST_ID, CONFIG/*) from the
old OSTs. According to bug 24128, this puts us in the "cold replace"
scenario. Is there anything we can do to avoid quiescing the entire
filesystem? If we can avoid unmounting all the clients, we'd prefer it.
Since our combo MGT/MDT is going to have to be unmounted for the
writeconf, we'd have the opportunity to mount it as ldiskfs and muck around.
Thanks again,
Craig Prescott
UF HPC Center
Wang Yibin wrote:
> Hello,
>
> Did you backup old magic files (last_rcvd, LAST_ID, CONFIG/*) from the original OSTs and put them back before trying to mount them?
> You probably didn't do that. So when you remount the OSTs with existing index, the MGS will refuse to add them without being told to writeconf, hence -EADDRINUSE.
> The proper ways to replace an OST are described in bug 24128.
>
Charles Taylor
A little more info here....
We have 30 OSTs hosted on 5 OSSs where we are using Areca 1680ix PCI-E
RAID controllers. The long and short of it is that the Areca
1680ix's have proven completely buggy and unreliable - not what you
want in a RAID card. So we are evacuating all the OSTs, replacing
the Areca 1680ix cards with Adaptec 51645s, re-initializing the LUNs,
reformatting the LUNs as OSTs (using the same OST index as before) and
remounting them. That is the plan anyway.
We've already reformatted (mkfs.lustre) one set of 6 OSTs and did not
save the "magic" files and so are getting the "Address in Use" error
for those OSTs. That being the case, I assume we must
1. Unmount the file system from all clients
2. Unmount the OSTs
3 Unmount the MDT
4. tunefs.lustre --writeconf /dev/mdt
5. remount the MDT
6. remount the OSTs (including the reformatted ones)
7. remount the file system on clients.
1. Is this the correct sequence?
2. Will this leave all our data intact?
3. Must we do a writeconf on the OSTs too or just the MDT?
Also, for the remaining OSTs we will save the "magic" files and
restore them after reformatting which should eliminate the need for
the procedure above. With some of the OSTs mounted as ldiskfs, I
see the last_rcvd file and the CONFIG directory but no LAST_ID
file. Should the LAST_ID file be there?
Regards,
Charlie Taylor
UF HPC Center
Andreas Dilger
> So we are evacuating all the OSTs, replacing
> the Areca 1680ix cards with Adaptec 51645s, re-initializing the LUNs,
> reformatting the LUNs as OSTs (using the same OST index as before) and
> remounting them. That is the plan anyway.
It's unfortunate that you didn't see the thread from a few weeks ago that discussed this exact topic of OST replacement. It should get a section in the manual I think.
> We've already reformatted (mkfs.lustre) one set of 6 OSTs and did not
> save the "magic" files and so are getting the "Address in Use" error
> for those OSTs. That being the case, I assume we must
>
> 1. Unmount the file system from all clients
> 2. Unmount the OSTs
> 3 Unmount the MDT
> 4. tunefs.lustre --writeconf /dev/mdt
> 5. remount the MDT
> 6. remount the OSTs (including the reformatted ones)
> 7. remount the file system on clients.
>
> 1. Is this the correct sequence?
> 2. Will this leave all our data intact?
> 3. Must we do a writeconf on the OSTs too or just the MDT?
There is actually only a flag on the OSTs that needs to be changed to have it stop trying to register with the MGS, and just pretend to be the OST index that you formatted it as.
The flag is in the binary CONFIGS/mountdata file (struct lustre_disk_data) in the ldd_flags field (offset 20). It should only have the LDD_F_SV_TYPE_OST (2) set.
> Also, for the remaining OSTs we will save the "magic" files and
> restore them after reformatting which should eliminate the need for
> the procedure above. With some of the OSTs mounted as ldiskfs, I
> see the last_rcvd file and the CONFIG directory but no LAST_ID
> file. Should the LAST_ID file be there?
This file is at /O/0/LAST_ID (capital 'o' then zero) and should be copied for OSTs you haven't replaced yet, along with the other files. It can be recreated with a binary editor from the value on the MDS (lctl get_param osc.*.prealloc_next_id) for the 6 OSTs that have already been replaced. Search the list or bugzilla for "LAST_ID" for a detailed procedure.
Charles Taylor
On Dec 21, 2010, at 12:39 PM, Andreas Dilger wrote:
> It's unfortunate that you didn't see the thread from a few weeks ago
> that discussed this exact topic of OST replacement.
Agreed. :(
> It should get a section in the manual I think.
Agreed.
> This file is at /O/0/LAST_ID (capital 'o' then zero) and should be
> copied for OSTs you haven't replaced yet, along with the other
> files. It can be recreated with a binary editor from the value on
> the MDS (lctl get_param osc.*.prealloc_next_id) for the 6 OSTs that
> have already been replaced. Search the list or bugzilla for
> "LAST_ID" for a detailed procedure.
This seems to do the trick. Thank you!. One important
clarification though...on the mds, should we getting the value of
prealloc_next_id or prealloc_last_id? Section 23.3.9 of the 2.0 Ops
manual for "How to fix a Bad LAST_ID on an OST" seems to use
prealloc_last_id. Which should we be using?
Thank you again,
Charlie Taylor
Andreas Dilger
> On Dec 21, 2010, at 12:39 PM, Andreas Dilger wrote:
>> This file is at /O/0/LAST_ID (capital 'o' then zero) and should be copied for OSTs you haven't replaced yet, along with the other files. It can be recreated with a binary editor from the value on the MDS (lctl get_param osc.*.prealloc_next_id) for the 6 OSTs that have already been replaced. Search the list or bugzilla for "LAST_ID" for a detailed procedure.
>
> This seems to do the trick. Thank you!. One important clarification though...on the mds, should we getting the value of prealloc_next_id or prealloc_last_id? Section 23.3.9 of the 2.0 Ops manual for "How to fix a Bad LAST_ID on an OST" seems to use prealloc_last_id. Which should we be using?
Using prealloc_next_id is technically more correct for your situation, since the lower-numbered objects have not been precreated on the OST. If you used prealloc_last_id then the MDS and OSS would assume the lower-numbered objects exist but the clients would get IO errors trying to access them.
Cheers, Andreas
--
Andreas Dilger
Lustre Technical Lead
Oracle Corporation Canada Inc.
Charles Taylor
Got it. Thank you again.
Charlie Taylor
UF HPC Center