Renaming storage node after replacement - replacing old storage nodes

[Trey Dockendorf]

We are in the process of expanding our BeeGFS filesystem and due to physical rack space limitations we have to retire our oldest storage nodes to make room for new storage nodes.  We are using ZFS on storage so the process would basically be a ZFS send before maintenance then putting the cluster into maintenance and doing a final incremental ZFS send of the old storage volume onto the new server.  However because this is a new system I'd ideally like the name given to the new storage node to be different.  We used to name our storage nodes in a way that was an abbreviation of their product name.  So JackRabbit 4 becomes "jr4".  The new systems won't be JackRabbit servers so not using the "jr4" in name would hopefully avoid potential confusion.

When I provisioned our nodes I forced specific ID name and numbers by putting values in /tank/fhgfs/storage/{nodeID,nodeNumID,targetID,targetNumID}.  The only file listed below I didn't set myself was originalNodeID.

# for f in `find /tank/fhgfs/storage -maxdepth 1 -type f -name '*ID'`; do echo -n "$f: " ; cat $f ; done

/tank/fhgfs/storage/targetID: jr4-1-tank

/tank/fhgfs/storage/nodeNumID: 5

/tank/fhgfs/storage/nodeID: jr4-1

/tank/fhgfs/storage/originalNodeID: jr4-1

/tank/fhgfs/storage/targetNumID: 5

When we ZFS send this filesystem to the server's replacement, it will still have these files.  I'm almost certain that the value in targetID and nodeID either can't be changed or can with dire consequences.  Is there a way to safely rename a node for the purposes of replacing old storage with new storage?  Is the migrate command in fhgfs-ctl the only option?

Thanks,

- Trey

[Sven Breuner]

Hi Trey,

in this case it's ok to set the nodeID and originalNodeID file to the new names.
Only the numeric IDs ({target,node}NumID files) are stored in the BeeGFS
metadata, while the string IDs are more for extra sanity checking and log
messages and thus can generally be changed without the need to "migrate"
anything or modify the metadata.
On the mgmtd node, you will also either need to use fhgfs-ctl to remove the old
node names before you change them or simply delete the meta.nodes and
storage.nodes files from the mgmtd directory.

Best regards,
Sven Breuner
Fraunhofer

[Trey Dockendorf]

Sven,

That's good to know, thanks!

Is it also safe to modify targetID?

If I go the fhgfs-ctl route, would I need both "--removenode" and "--unmaptarget" if I change the value in targetID?

If I go the route of removing files on mgmtd would I also need to remove targetNumIDs file as it seems to reference values in targetID?

With either route above, is it the case they only modify the name -> numeric ID mapping stored in mgmtd and won't break anything on metadata mapping to storage as long as the numeric ID doesn't change?

Thanks,

- Trey

[Bernd Lietzow]

Hi Trey!

On 05/11/2015 05:30 PM, Trey Dockendorf wrote:
> Sven,
>
> That's good to know, thanks!
>
> Is it also safe to modify targetID?

Yes you can modify that, but then you have to delete the targetNumIDs in
the management directory.

>
> If I go the fhgfs-ctl route, would I need both "--removenode" and
> "--unmaptarget" if I change the value in targetID?

No, targets are automatically unmapped if you remove a node.

>
> If I go the route of removing files on mgmtd would I also need to remove
> targetNumIDs file as it seems to reference values in targetID?

You only have to remove that if you want to change the (non-numeric)
target IDs.

>
> With either route above, is it the case they only modify the name ->
> numeric ID mapping stored in mgmtd and won't break anything on metadata
> mapping to storage as long as the numeric ID doesn't change?
>

The metadata servers only care for the numeric IDs, so as long as those
do not change, it won't break anything.

Kind regards,
Bernd.

> Thanks,
>
> - Trey
>
> On May 10, 2015 12:25 PM, "Sven Breuner"
> <sven.b...@itwm.fraunhofer.de

> --
> You received this message because you are subscribed to the Google
> Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fhgfs-user+...@googlegroups.com
> <mailto:fhgfs-user+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Trey Dockendorf]

Bernd,

Thanks for the clarifications.

Because this at least seems like a "risky" operation I want to be properly prepared and avoid any unforeseen issues when we expand our filesystem...so more questions :)

Due to past mistakes I've made our management server has sysAllowNewServers = false, so clearly this operation will require that being true.  Here are my tentative steps:

* filesystem remains in operation

* snapshot and send the ~24TB filesystem from old storage server to new one

* take filesystem offline

* stop all services (including fhgfs-storage) on old storage server that touch the ZFS filesystem

* make old storage server filesystem readonly in ZFS

* take snapshot and send incremental snapshot from old storage server to new one

* on new storage server rename contents of targetID, nodeID, and originalNodeID to match the new name of system

* Execute 'fhgfs-ctl --removenode --nodetype=storage 5' where 5 is the value assigned to old/new storage

* set sysAllowNewServers to true and restart fhgfs-mgmtd

* start fhgfs-storage on new storage server

* wait for new storage server to register with mgmtd

* set sysAllowNewServers to false and restart fhgfs-mgmtd

* bring filesystem back online

Do those sound like the correct steps?  My primary concern is ensuring that the "old" storage server is cleanly removed and the "new" storage server is able to come online and handle the client requests that would normally have gone to the old system.

As a side note if anyone knows how to open network stream over RDMA from command line I'd love to know.  Right now my ZFS send/receive involves the "nc" command to reduce the performance impact that would normally be experienced when performing the send/receive via SSH.  Our systems are connected to a DDR fabric and I can imagine using RDMA instead of IPoIB would be faster (if even possible).

Thanks,

- Trey

You received this message because you are subscribed to a topic in the Google Groups "beegfs-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fhgfs-user/xZc62RYBaYA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fhgfs-user+...@googlegroups.com.

[P Serocka]

On 2015 May 14. md, at 05:19 st, Trey Dockendorf wrote:

Bernd,

Thanks for the clarifications.

Because this at least seems like a "risky" operation I want to be properly prepared and avoid any unforeseen issues when we expand our filesystem...so more questions :)

Due to past mistakes I've made our management server has sysAllowNewServers = false, so clearly this operation will require that being true.  Here are my tentative steps:

* filesystem remains in operation

* snapshot and send the ~24TB filesystem from old storage server to new one

* take filesystem offline

* stop all services (including fhgfs-storage) on old storage server that touch the ZFS filesystem

* make old storage server filesystem readonly in ZFS

* take snapshot and send incremental snapshot from old storage server to new one

Suggest to backup the metadata servers at this stage,

so you can roll back in case the new cluster isn't going to fly...

Also, do you plan to stress-test the complete maneuvre

on a dummy cluster?

  

Cheers

-- Peter

To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.

[Trey Dockendorf]

Backups are handled, I keep ZFS snapshots on our metadata and also replicate the snapshots to a remote filesystem.

I don't know why I didn't think to test this out on a dummy cluster.  I'll throw some VMs together and try the procedure out there.

Thanks,

- Trey

[trey...@tamu.edu]

I finally got around to testing this out with a development metadata/mgmtd and some VMs acting as both old and new storage.  After stopping fhgfs-storage and replicating the filesystem via ZFS, I modified nodeID, targetID and originalNodeID.  I then executed "fhgfs-ctl --removenode --nodetype=storage 1".  Next I started fhgfs-storage on the new storage system and receive an error in logs for fhgfs-storage [1] and fhgfs-mgmtd [2].

fs04 (192.168.202.247) is metadata and mgmtd.  oss-test2 (192.168.200.91) is the 'new' storage system.

I noticed that the "targets" file for mgmtd is empty but "targetNumIDs" contains a reference to the old server, even though I ran the "--removenode" command.  I've tried doing "--unmaptarget" then "--removenode" and still same result.  I even modified the mgmtd targetNumIDs by hand but somehow upon restarting mgmtd the file goes back to containing old storage node's information.

In another attempt I tried removing storage.nodes, meta.nodes and targetNumIDs from mgmtd directory.  When I restart fhgfs-mgmtd I see this in the logs:

(3) Aug11 17:56:24 Main [App] >> Loaded metadata nodes: 1

(3) Aug11 17:56:24 Main [App] >> Loaded clients: 1

(4) Aug11 17:56:24 Main [App] >> Loaded target numeric ID mappings: 1

(3) Aug11 17:56:24 Main [App] >> Loaded target mappings: 0

The only file recreated is targetNumIDs and it contains "jr4-1-tank=1" which is the name of the old test server.

For completeness sake here's a before and after of storage files I changed after replicating filesystem with ZFS.

BEFORE:

/tank/fhgfs/storage/nodeID: jr4-1

/tank/fhgfs/storage/nodeNumID: 1

/tank/fhgfs/storage/originalNodeID: jr4-1

/tank/fhgfs/storage/targetID: jr4-1-tank

/tank/fhgfs/storage/targetNumID: 1

AFTER: 

/tank/fhgfs/storage/nodeID: oss01

/tank/fhgfs/storage/nodeNumID: 1

/tank/fhgfs/storage/originalNodeID: oss01

/tank/fhgfs/storage/targetID: oss01-tank

/tank/fhgfs/storage/targetNumID: 1

Did I miss a step in properly reassigning ownership of a storage pool from one system to another?

Thanks,

- Trey

[1]:

(4) Aug11 17:43:31 Main [App] >> Allowed interfaces: eth0

(4) Aug11 17:43:31 Main [App] >> Detaching process...

(3) Aug11 17:43:31 Main [RegDGramLis] >> Listening for UDP datagrams: Port 8003

(1) Aug11 17:43:31 Main [App] >> Waiting for fhgfs-mgmtd@fs04:8008...

(2) Aug11 17:43:31 RegDGramLis [Heartbeat incoming] >> New node: fhgfs-mgmtd fs04 [ID: 1];

(4) Aug11 17:43:31 RegDGramLis [RegDGramLis] >> Component stopped.

(4) Aug11 17:43:31 Main [NodeConn (acquire stream)] >> Establishing new TCP connection to: fhgfs...@192.168.202.247:8008

(3) Aug11 17:43:31 Main [NodeConn (acquire stream)] >> Connected: fhgfs...@192.168.202.247:8008 (protocol: TCP)

(0) Aug11 17:43:31 Main [App] >> Target ID reservation request was rejected by this mgmt node: fs04 [ID: 1]

(0) Aug11 17:43:31 Main [App] >> Target pre-registration at management node failed

(4) Aug11 17:43:31 Main [NodeConn (destruct)] >> Closing 1 connections...

(4) Aug11 17:43:31 Main [NodeConn (invalidate stream)] >> Disconnected: fhgfs...@192.168.202.247:8008

[2]:

(4) Aug11 17:43:31 StreamLis [StreamLis] >> Accepted new connection from 192.168.200.91:39280 [SockFD: 24]

(1) Aug11 17:43:31 Worker4 [RegisterTargetMsg incoming] >> Registration failed for target: oss01-tank; numID: 1

(4) Aug11 17:43:31 Worker2 [Work (process incoming data)] >> Soft disconnect from 192.168.200.91:39280

(4) Aug11 17:43:31 StreamLis [StreamLis] >> Accepted new connection from 192.168.200.91:39281 [SockFD: 24]

(4) Aug11 17:43:31 Worker4 [Work (process incoming data)] >> Soft disconnect from 192.168.200.91:39281

> <mailto:fhgfs-user+unsub...@googlegroups.com>.

[Bernd Lietzow]

Hello Trey!

Did you stop the mgmtd before editing the files? Otherwise they will get
overwritten with old values the mgmtd still has in memory when it shuts
down before the restart.

Also, the mgmtd does not delete the string ID (server name) when a
target is unmapped - only the numerical one. Then the new server can't
be registered under the same numeric ID as the old one (with a different
string ID). In that case you can try removing the string ID from the
mgmtd's targetNumIDs file.

Kind regards
Bernd

> > <mailto:fhgfs-user+...@googlegroups.com>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.
>

[Trey Dockendorf]

Bernd,

Thanks, stopping mgmtd and removing the "jr4-1-tank=1" line from targetNumIDs worked.  At first I was receiving IO errors when trying to read test files from the client side but once I restarted fhgfs-client everything started working.  So it seems I may need to restart all my clients when we perform this operation in production.

So the fhgfs-ctl commands to remove a node are not enough in this case?  I have to also stop mgmtd and remove the offending line from targetNumIDs?  I'll repeat my tests but this time with a second storage node so be sure modifications don't effect storage systems that are not being migrated.

Thanks,

- Trey

=============================

Trey Dockendorf 

Systems Analyst I 

Texas A&M University 

Academy for Advanced Telecommunications and Learning Technologies 

Phone: (979)458-2396 

Email: trey...@tamu.edu 

Jabber: trey...@tamu.edu

>

--

You received this message because you are subscribed to a topic in the Google Groups "beegfs-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fhgfs-user/xZc62RYBaYA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fhgfs-user+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Bernd Lietzow]

Hello Trey!

Exactly, the fhgfs-ctl only removes the server from the list of active
server IDs. It is not assumed that the ID is ever used again, so newly
added servers would get new IDs. The case that a server comes back with
the same ID and a new stringID is not covered by this, so the files have
to be edited manually...

Kind regards
Bernd

On 12.08.2015 18:00, Trey Dockendorf wrote:
> Bernd,
>
> Thanks, stopping mgmtd and removing the "jr4-1-tank=1" line from
> targetNumIDs worked. At first I was receiving IO errors when trying to
> read test files from the client side but once I restarted fhgfs-client
> everything started working. So it seems I may need to restart all my
> clients when we perform this operation in production.
>
> So the fhgfs-ctl commands to remove a node are not enough in this case?
> I have to also stop mgmtd and remove the offending line from
> targetNumIDs? I'll repeat my tests but this time with a second storage
> node so be sure modifications don't effect storage systems that are not
> being migrated.
>
> Thanks,
> - Trey
>
> =============================
>
> Trey Dockendorf
> Systems Analyst I
> Texas A&M University
> Academy for Advanced Telecommunications and Learning Technologies
> Phone: (979)458-2396

> Email: trey...@tamu.edu <mailto:trey...@tamu.edu>
> Jabber: trey...@tamu.edu <mailto:trey...@tamu.edu>

>
> On Wed, Aug 12, 2015 at 10:49 AM, Bernd Lietzow
> <bernd....@itwm.fraunhofer.de

> <http://fhgfs...@192.168.202.247:8008>

> > (3) Aug11 17:43:31 Main [NodeConn (acquire stream)] >> Connected:
> > fhgfs...@192.168.202.247:8008

> <http://fhgfs...@192.168.202.247:8008> (protocol: TCP)

> > (0) Aug11 17:43:31 Main [App] >> Target ID reservation request was
> > rejected by this mgmt node: fs04 [ID: 1]
> > (0) Aug11 17:43:31 Main [App] >> Target pre-registration at management
> > node failed
> > (4) Aug11 17:43:31 Main [NodeConn (destruct)] >> Closing 1
> connections...
> > (4) Aug11 17:43:31 Main [NodeConn (invalidate stream)] >>
> Disconnected:
> > fhgfs...@192.168.202.247:8008

> <http://fhgfs...@192.168.202.247:8008>

> >
> > [2]:
> > (4) Aug11 17:43:31 StreamLis [StreamLis] >> Accepted new
> connection from

> > 192.168.200.91:39280 <http://192.168.200.91:39280> [SockFD: 24]

> > (1) Aug11 17:43:31 Worker4 [RegisterTargetMsg incoming] >>
> Registration
> > failed for target: oss01-tank; numID: 1
> > (4) Aug11 17:43:31 Worker2 [Work (process incoming data)] >> Soft

> > disconnect from 192.168.200.91:39280 <http://192.168.200.91:39280>

> > (4) Aug11 17:43:31 StreamLis [StreamLis] >> Accepted new
> connection from

> > 192.168.200.91:39281 <http://192.168.200.91:39281> [SockFD: 24]

> > (4) Aug11 17:43:31 Worker4 [Work (process incoming data)] >> Soft

> > disconnect from 192.168.200.91:39281 <http://192.168.200.91:39281>

> > > <mailto:sven.b...@itwm.fraunhofer.de

> <mailto:fhgfs-user%2B...@googlegroups.com>
> > > <mailto:fhgfs-user+...@googlegroups.com
> <mailto:fhgfs-user%2Bunsu...@googlegroups.com>>.

> > > For more options, visit https://groups.google.com/d/optout
> > <https://groups.google.com/d/optout>.
> >
>
> --
> You received this message because you are subscribed to a topic in
> the Google Groups "beegfs-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/fhgfs-user/xZc62RYBaYA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> fhgfs-user+...@googlegroups.com

> <mailto:fhgfs-user%2Bunsu...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fhgfs-user+...@googlegroups.com
> <mailto:fhgfs-user+...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

[bourd...@googlemail.com]

Hello BeeGFS team,

I just stumbled over this old thread, because I had problems inserting a storage target after a harddisk failure and replacement. It seems there is still no command-line option to remove an orphaned targetNumID. Alternatively, it would also be fine, if re-registering a previously removed targetID would just work out-of-the-box, even though the removal was an automatic one due to a communication error or whatever. Maybe, if such default behavior is considered by the team to be "unsafe", one could introduce a setting to the management daemon that would allow for a quick and smooth replacement of destroyed storage targets.

Btw. I just also see that the option "--targetid" must always be given fully lower-case, even though the "beegfs-ctl" client prints out "--targetID". Would be great if this could be fixed either by ignoring the case of --options or by fixing the usage output of "beegfs-ctl".

Thanks and best greetings,
Ph. Bourdin.

[Sven Breuner]

Hi,

bourdin.kis via beegfs-user wrote on 06.01.2017 15:35:
> Btw. I just also see that the option "--targetid" must always be given fully
> lower-case, even though the "beegfs-ctl" client prints out "--targetID". Would
> be great if this could be fixed either by ignoring the case of --options or by
> fixing the usage output of "beegfs-ctl".

just to be sure: Which beegfs-ctl mode exactly were you referring to regarding
the "--targetID" / "--targetid" problem? And was this with a BeeGFS v6.x release
or a previous release series?

Thanks and best regards,
Sven

[kva...@gmail.com]

Hello Bernd and BeeGFS team,

Do I right understand that editing beegfs-mgmtd's targetNumIDs file for remove old targetid, than add new one with same targetid, this is correct way for replace broken target in the mirror buddy-group?

Is it completely safe?

How does the beegfs will know that this target is new and that data should be synchronized from second healthy target to this one, but not another direction?

>     >     > <mailto:fhgfs-user+unsub...@googlegroups.com

>     <mailto:fhgfs-user%2Bunsu...@googlegroups.com>>.
>     >     > For more options, visit https://groups.google.com/d/optout
>     >     <https://groups.google.com/d/optout>.
>     >
>
>     --
>     You received this message because you are subscribed to a topic in
>     the Google Groups "beegfs-user" group.
>     To unsubscribe from this topic, visit
>     https://groups.google.com/d/topic/fhgfs-user/xZc62RYBaYA/unsubscribe.
>     To unsubscribe from this group and all its topics, send an email to
>     fhgfs-user+...@googlegroups.com
>     <mailto:fhgfs-user%2Bunsu...@googlegroups.com>.
>     For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fhgfs-user+...@googlegroups.com

> <mailto:fhgfs-user+unsub...@googlegroups.com>.