Speeding up beeond startup

[meh...@soysal.eu]

Hi,
i noticed that when starting beeond with pdsh, there is still a serial part.
In that serial part the STATUSFILE on each node is updated one after one with ssh.

This results in pretty long startup for beeond, e.g around 4 minutes on 256 Nodes.

I think this can be improved, instead initiating a ssh connection to each node,
write the STATUSFILE locally on the node where beeond is invoked and then copy it to the nodes.
The copy itself can be done parallel.

Is there a reason why the update of the STATUSFILE is done serial.

I tried the modifications, first writing file locally and then copy it with "scp" and "gnu parallel".
Received very good speedup on the first try:
On 256 nodes only around 1 minute for startup.
I only checked for speeding up the startup process,
since im stopping beeond trough epilogue scripts on each node with stoplocal.

Of course this is a quick and dirty way of doing this,
for production usage it should be checked and maybe improved:
---------------------------------------------------------------
@@ -966,7 +968,8 @@
    fi

    INFO="${HOST},${SERVICE},${DATAPATH},${LOGFILE},${PIDFILE}"
-   execute_ssh_cmd ${HOST} "echo ${INFO} >> ${STATUSFILE}"
+   #execute_ssh_cmd ${HOST} "echo ${INFO} >> ${STATUSFILE}"
+   echo ${INFO} >> /var/tmp/beeond/${HOST}-`basename ${STATUSFILE}`
 }

 ### internal functions for beegfs-ondemand stop ###
@@ -1045,6 +1048,7 @@
          execute_ssh_cmd ${HOST} "${DELETESTATUSFILECMD}"
       done
    fi
+   rm /var/tmp/beeond/*
 }

 unmount_clients()
@@ -1245,6 +1249,14 @@

   # give the management daemon some time to get all information from servers
    sleep 8
+
+   #Copy the statusfiles to nodes
+   IFS=","
+   for HOST in ${ALLNODES}
+   do
+         echo "scp -q -i /root/.ssh/adafs /var/tmp/beeond/${HOST}-beeond.tmp ${HOST}:/tmp/beeond.tmp"
+   done | parallel -j 32
+   unset IFS

    # take all hosts as client
    # port shift and config path may be empty, but that's ok

----------------------------------------------------------------

What do you think of that idea ?



best regards
M.Soysal

[Bernd Lietzow]

Hi Mehmet!

That's a good catch. This particular point wasn't parallelized before
(in the way you suggested) because we did not want to introduce more
dependencies (scp, parallel) to the beeond tool. But now that you
pointed it out as a bottleneck, we will put it on our agenda again and
see how we can parallelize it and speed it up.

Thank you for your contribution!

Kind regards
Bernd

> --
> You received this message because you are subscribed to the Google
> Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fhgfs-user+...@googlegroups.com
> <mailto:fhgfs-user+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[meh...@soysal.eu]

Hi,
that was only a quick solution. I think with some time a more clean solution cloud be made.
If you guys put some work into this, can you please add some features to PDSH startup.

Optional fanout "pdsh -f" is really needed in bigger environments.
And specifiy a special SSH KEY for PDSH would be nice.


best regards
M.Soysal

[Sven Breuner]

Hi Mehmet,

meh...@soysal.eu wrote on 11.01.2017 09:57:
> If you guys put some work into this, can you please add some features to PDSH
> startup.
>
> Optional fanout "pdsh -f" is really needed in bigger environments.
> And specifiy a special SSH KEY for PDSH would be nice.

you can use environment variables for pdsh to configure this ("FANOUT" and
"PDSH_SSH_ARGS_APPEND").

Best regards,
Sven

[Mehmet Soysal]

Hi,

yes your are right. But it would be more comfortable when offering this as a beeond option.

An using beeond with pdsh has at the moment a plain ssh section which, like mentioned.

Having a dedicated option to specify a seprate Key would be much easier.

best regards

M.Soysal