NFS clients - puppet hang when mount point

[madAndroid]

We've only recently discovered that puppet can manage mount points
using the mount directive;
however, a short while back we built an nfs client and server classes
without using this resource, and we've encountered a problem where
puppet seems to hang when the nfs server is unavailable.

Using --debug doesn't seem to specify exactly at which point the run
is failing, which could steer us in the right direction around putting
something in place in the classes in question.

Is there anything we can do, short of switching over to using the
mount directive/resource, in order to mitigate the problem when the
nfs server is unavailable? It's preventing us from managing other
resources on the clients when this happens..

thanks in advance,
Andrew

[Guy Matz]

OK.  This may seem like a bad idea, but it's a workaround that has worked for me:
I add the nfs server to the 127.0.0.1 entry of the hosts file which causes NFS to time out pretty immediately.  :-\

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[madAndroid]

That sounds a little heavy handed ...

also, I feel that it would probably stop the mount from happening at
all?
how would the fstab initiate the nfs mount if it's not able to resolve
the address of the nfs server correctly?
unless I'm missing something ..

On Nov 9, 7:12 pm, Guy Matz <gm...@matz.org> wrote:
> OK.  This may seem like a bad idea, but it's a workaround that has worked
> for me:
> I add the nfs server to the 127.0.0.1 entry of the hosts file which causes
> NFS to time out pretty immediately.  :-\
>
>
>
>
>
>
>
> On Wed, Nov 9, 2011 at 10:00 AM, madAndroid <andrewsta...@gmail.com> wrote:
> > We've only recently discovered that puppet can manage mount points
> > using the mount directive;
> > however, a short while back we built an nfs client and server classes
> > without using this resource, and we've encountered a problem where
> > puppet seems to hang when the nfs server is unavailable.
>
> > Using --debug doesn't seem to specify exactly at which point the run
> > is failing, which could steer us in the right direction around putting
> > something in place in the classes in question.
>
> > Is there anything we can do, short of switching over to using the
> > mount directive/resource, in order to mitigate the problem when the
> > nfs server is unavailable? It's preventing us from managing other
> > resources on the clients when this happens..
>
> > thanks in advance,
> > Andrew
>
> > --
> > You received this message because you are subscribed to the Google Groups

> > "Puppet Users" group.> To post to this group, send email topuppe...@googlegroups.com.

[Len Rugen]

If you can reproduce this, you might try adding the "bg" option to the mount options (via options => in puppet).  I suspect that when puppet tries to do the nfs mount with the nfs server unavailable, it will hang for 120 seconds or more.  I think it you tried the same mount command at that time outside of puppet, it would hang. 

I suspect you have ensure => mounted, if ensure => present and atboot => true would be adequate, you might circumvent this problem.  Having puppet ensure the mount is cool, but the 30 minute lag isn't adequate in our shop to detect and correct a missing resource.  I would have our monitoring system (Zabbix) more frequently check resources, maybe every minute or two and not rely on puppet alone. 

To post to this group, send email to puppet...@googlegroups.com.

[jcbollinger]

It all comes down to mount options. I'd recommend you absorb the
material in nfs(5), but options of particular interest include
'retry', 'retrans', and 'timeo'. Do notice that the latter two will
affect all operations on the mounted filesystem, not just the initial
mount (and maybe not the initial mount at all -- the docs aren't quite
clear on that).

Unlike Len, I would not recommend using the "bg" option. Doing so
likely would prevent Puppet from hanging for a long time attempting
the mount, but it would also prevent Puppet from correctly managing
the resource. Puppet needs to know whether it has succeeded in
mounting the remote filesystem. It may also cause you trouble later
if the client never does succeed in mounting the filesystem.


John

[madAndroid]

Thank you so much to all who responded - we've managed to rewrite our
classes to use the mount directive rather (and quite quickly and
painlessly), and it seems a much better way than we had before (using
file and exec resources/directives). Still experiencing the timeouts
during the puppet run when the NFS box goes away (in a dev environment
during testing the new class structure), but we haven't tried the two
options that Len suggested - will be giving those a go, with the trade-
offs in mind. Yes, I agree with John that it does feel like more of a
problem with the way we are implementing the mounts, will give those
suggestions a try as well.

Thanks again,
Andrew

[madAndroid]

FYI,

We ended up going with a mix of the two suggestions: having 'bg'
specified as one of the mount point options,
and also having the mount resource specify => present instead of =>
mounted, and then we use an exec to force
the remount, constrained by an unless and onlyif:

exec { "remount-storage":
command => "mount -a",
unless => "grep $brand /proc/mounts",
onlyif => "nc -zv $nfs_server 2049 -w 3",
}

The works for us in all use cases, and seems to be the best way
forwards :)

- Andrew

> > > > > however, a short while back we built annfs clientand server classes

[madAndroid]

FYI,

We ended up going with a mix of the two suggestions: having 'bg'
specified as one of the mount point options,
and also having the mount resource specify => present instead of =>
mounted, and then we use an exec to force
the remount, constrained by an unless and onlyif:

exec { "remount-storage":
command => "mount -a",
unless => "grep $brand /proc/mounts",
onlyif => "nc -zv $nfs_server 2049 -w 3",
}

The works for us in all use cases, and seems to be the best way
forwards :)

- Andrew


On Nov 11, 11:03 am, madAndroid <andrewsta...@gmail.com> wrote:

> > > > > however, a short while back we built annfs clientand server classes