unresponsive system during moves
16 views
Skip to first unread message
vajonam
Aug 14, 2019, 7:05:23 PM8/14/19
to EON ZFS Storage
I completed the upgrade to the latest and updated the pools and file systems. since I do need to move to more recent code. all went well with the upgrade.
I do notice that sometimes when I "move" data between the drives the whole system becomes unresponsive and I can't even ping the box from the outside. Are there anything I need to tune to for the system to remain responsive? and for other operations to continue?
Andre Lue
Aug 14, 2019, 10:35:48 PM8/14/19
to EON ZFS Storage on behalf of vajonam
No, you shouldn't have to tune anything. Does the system recover from the freeze or do you have to power it off/on?
Is this on a VM or a bare metal system?
On Wed, Aug 14, 2019 at 7:05 PM vajonam via EON ZFS Storage <eonstorage+APn2wQeaGveeVtAFGo6St...@googlegroups.com> wrote:
I completed the upgrade to the latest and updated the pools and file systems. since I do need to move to more recent code. all went well with the upgrade.I do notice that sometimes when I "move" data between the drives the whole system becomes unresponsive and I can't even ping the box from the outside. Are there anything I need to tune to for the system to remain responsive? and for other operations to continue?
--
You received this message because you are subscribed to the Google Groups "EON ZFS Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eonstorage+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eonstorage/5610c6d6-b99e-486e-92f8-875f08eabf3c%40googlegroups.com.
Manojav Sridhar
Aug 14, 2019, 11:33:42 PM8/14/19
to EON ZFS Storage on behalf of dre2kse
The system recovers from the freeze. sometimes takes like 30 seconds.
its a bare metal system. no VMs, I am not even running anything on the box just serving up files.
To view this discussion on the web visit https://groups.google.com/d/msgid/eonstorage/CADsvidC64g2Xqd_vYjxeC2-UrkNSgA9d51zk9uaSW_a8YFnyRA%40mail.gmail.com.
Manojav Sridhar
Aug 14, 2019, 11:39:43 PM8/14/19
to EON ZFS Storage on behalf of dre2kse
hmm this is bit concerning looks I am not able to very much on that box in terms of writes.. before it locks up the system.
This is what I see in the client logs.
[ 537.443278] nfs: server getafix not responding, timed out
[ 545.639689] nfs: server getafix not responding, timed out
[ 545.639689] nfs: server getafix not responding, timed out
vajonam
Aug 14, 2019, 11:43:23 PM8/14/19
to EON ZFS Storage
what rsize and wsize should I be using the nfs mounts on the clients?
On Wednesday, August 14, 2019 at 11:39:43 PM UTC-4, vajonam wrote:
hmm this is bit concerning looks I am not able to very much on that box in terms of writes.. before it locks up the system.This is what I see in the client logs.[ 537.443278] nfs: server getafix not responding, timed out
[ 545.639689] nfs: server getafix not responding, timed out
On Wed, Aug 14, 2019 at 11:33 PM Manojav Sridhar <man...@manojav.com> wrote:
The system recovers from the freeze. sometimes takes like 30 seconds.its a bare metal system. no VMs, I am not even running anything on the box just serving up files.
On Wed, Aug 14, 2019 at 10:35 PM EON ZFS Storage on behalf of dre2kse <eonst...@googlegroups.com> wrote:
No, you shouldn't have to tune anything. Does the system recover from the freeze or do you have to power it off/on?Is this on a VM or a bare metal system?
On Wed, Aug 14, 2019 at 7:05 PM vajonam via EON ZFS Storage <eonstorage+APn2wQeaGveeVtAFGo6StkzOM-yyat2Qk_yK0TpsH4zA9NaHLqHYt@googlegroups.com> wrote:
I completed the upgrade to the latest and updated the pools and file systems. since I do need to move to more recent code. all went well with the upgrade.--I do notice that sometimes when I "move" data between the drives the whole system becomes unresponsive and I can't even ping the box from the outside. Are there anything I need to tune to for the system to remain responsive? and for other operations to continue?
You received this message because you are subscribed to the Google Groups "EON ZFS Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eonstorage+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eonstorage/5610c6d6-b99e-486e-92f8-875f08eabf3c%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "EON ZFS Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eonstorage+unsubscribe@googlegroups.com.
Message has been deleted
vajonam
Aug 15, 2019, 12:01:02 AM8/15/19
to EON ZFS Storage
Not sure what to do... dead in the water.. i ran it for 2 days without doing the zfs / zpool upgrade and no issues... crap..
vajonam
Aug 15, 2019, 12:08:24 AM8/15/19
to EON ZFS Storage
even the console is kind of busy, the return key moves the screen up, but I dont see the login prompt.. this is the longest its frozen.. maybe worth a reboot
vajonam
Aug 15, 2019, 12:11:13 AM8/15/19
to EON ZFS Storage
and its back.
on the bootup screen I see an error about no mircocode for my intel processor is this a concern?
last pid: 5560; load avg: 76.5, 47.9, 31.2; up 0+16:54:23 04:10:16
35 processes: 33 sleeping, 1 stopped, 1 on cpu
CPU states: 99.9% idle, 0.5% user, 0.1% kernel, 0.0% iowait, 0.0% swap
Memory: 4094M phys mem, 501M free mem, 4096M total swap, 4091M free swap
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
5547 root 1 59 0 4820K 2844K cpu/1 0:00 0.01% top
1190 root 15 59 0 11M 4996K sleep 0:01 0.00% smbd
1503 root 1 59 0 6504K 3812K sleep 0:01 0.00% ntpd
5502 root 1 59 0 8252K 4904K sleep 0:00 0.00% sshd
8 root 11 59 0 6068K 2184K sleep 0:00 0.00% svc.startd
10 root 14 59 0 7104K 4284K sleep 0:03 0.00% svc.configd
476 daemon 6 59 0 4680K 2792K sleep 0:00 0.00% idmapd
1134 root 1 59 0 4140K 36K sleep 0:00 0.00% ipmon
481 root 7 59 0 4820K 1888K sleep 0:00 0.00% devfsadm
1322 root 1 59 0 1624K 876K sleep 0:00 0.00% utmpd
2443 daemon 2 60 -20 3032K 1768K sleep 88:31 0.00% nfsd
5522 root 1 59 0 2500K 1892K stop 0:00 0.00% iostat
5542 root 1 59 0 8252K 4908K sleep 0:00 0.00% sshd
5541 root 1 59 0 6520K 3344K sleep 0:00 0.00% sshd
5501 root 1 59 0 6520K 3312K sleep 0:00 0.00% sshd
On Thursday, August 15, 2019 at 12:01:02 AM UTC-4, vajonam wrote:
Andre Lue
Aug 15, 2019, 12:29:55 AM8/15/19
to EON ZFS Storage on behalf of vajonam
I think this is more evident in the older versions. I will try to get you a newer release or a newer version to test, depending on time BW.
To view this discussion on the web visit https://groups.google.com/d/msgid/eonstorage/CAGaAVNVuT_otkCaMyr5bMwKxNQK4OxNqq%2B7%2Bf1DGB%3DQgZOEwaQ%40mail.gmail.com.
vajonam
Aug 15, 2019, 12:40:28 AM8/15/19
to EON ZFS Storage
Thank you so much! Even though am using this for personal use, I only think its fair that I donate to your cause this has given me really good performance over the years !
On Thursday, August 15, 2019 at 12:29:55 AM UTC-4, dre2kse wrote:
I think this is more evident in the older versions. I will try to get you a newer release or a newer version to test, depending on time BW.
On Wed, Aug 14, 2019 at 11:33 PM EON ZFS Storage on behalf of vajonam <eonst...@googlegroups.com> wrote:
The system recovers from the freeze. sometimes takes like 30 seconds.its a bare metal system. no VMs, I am not even running anything on the box just serving up files.
On Wed, Aug 14, 2019 at 10:35 PM EON ZFS Storage on behalf of dre2kse <eonst...@googlegroups.com> wrote:
No, you shouldn't have to tune anything. Does the system recover from the freeze or do you have to power it off/on?Is this on a VM or a bare metal system?
On Wed, Aug 14, 2019 at 7:05 PM vajonam via EON ZFS Storage <eonstorage+APn2wQeaGveeVtAFGo6StkzOM-yyat2Qk_yK0TpsH4zA9NaHLqHYt@googlegroups.com> wrote:
I completed the upgrade to the latest and updated the pools and file systems. since I do need to move to more recent code. all went well with the upgrade.--I do notice that sometimes when I "move" data between the drives the whole system becomes unresponsive and I can't even ping the box from the outside. Are there anything I need to tune to for the system to remain responsive? and for other operations to continue?
You received this message because you are subscribed to the Google Groups "EON ZFS Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eonst...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eonstorage/5610c6d6-b99e-486e-92f8-875f08eabf3c%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "EON ZFS Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eonst...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eonstorage/CADsvidC64g2Xqd_vYjxeC2-UrkNSgA9d51zk9uaSW_a8YFnyRA%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "EON ZFS Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eonst...@googlegroups.com.
vajonam
Aug 15, 2019, 12:54:40 AM8/15/19
to EON ZFS Storage
Also for more info. I am using ZIL log device to speed up writes maybe this is what is causing this runaway jobs?
capacity operations bandwidth
pool alloc free read write read write
------------------------- ----- ----- ----- ----- ----- -----
gobi 2.78T 868G 49 52 199K 5.24M
mirror 2.78T 868G 49 52 199K 5.24M
c0t50014EE2B94B1268d0 - - 20 52 83.0K 5.24M
c0t50014EE2B94B2234d0 - - 28 52 115K 5.24M
logs - - - - - -
c0t5000000000000000d0 37.8M 222G 0 0 0 0
------------------------- ----- ----- ----- ----- ----- -----
vajonam
Aug 15, 2019, 1:39:28 AM8/15/19
to EON ZFS Storage
Also note that if set the "sync" on my nfs clients this seems to throttle the transfer into the ZFS file system and doesn't become unresponsive.
On Thursday, August 15, 2019 at 12:29:55 AM UTC-4, dre2kse wrote:
I think this is more evident in the older versions. I will try to get you a newer release or a newer version to test, depending on time BW.
On Wed, Aug 14, 2019 at 11:33 PM EON ZFS Storage on behalf of vajonam <eonst...@googlegroups.com> wrote:
The system recovers from the freeze. sometimes takes like 30 seconds.its a bare metal system. no VMs, I am not even running anything on the box just serving up files.
On Wed, Aug 14, 2019 at 10:35 PM EON ZFS Storage on behalf of dre2kse <eonst...@googlegroups.com> wrote:
No, you shouldn't have to tune anything. Does the system recover from the freeze or do you have to power it off/on?Is this on a VM or a bare metal system?
On Wed, Aug 14, 2019 at 7:05 PM vajonam via EON ZFS Storage <eonstorage+APn2wQeaGveeVtAFGo6StkzOM-yyat2Qk_yK0TpsH4zA9NaHLqHYt@googlegroups.com> wrote:
I completed the upgrade to the latest and updated the pools and file systems. since I do need to move to more recent code. all went well with the upgrade.--I do notice that sometimes when I "move" data between the drives the whole system becomes unresponsive and I can't even ping the box from the outside. Are there anything I need to tune to for the system to remain responsive? and for other operations to continue?
You received this message because you are subscribed to the Google Groups "EON ZFS Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eonst...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eonstorage/5610c6d6-b99e-486e-92f8-875f08eabf3c%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "EON ZFS Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eonst...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eonstorage/CADsvidC64g2Xqd_vYjxeC2-UrkNSgA9d51zk9uaSW_a8YFnyRA%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "EON ZFS Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eonst...@googlegroups.com.
Andre Lue
Aug 15, 2019, 8:44:00 AM8/15/19
to EON ZFS Storage on behalf of Donovan Kaardal
The data move between filesystems is over nfs only? Smb only? Or nfs, smb at the same time?
To unsubscribe from this group and stop receiving emails from it, send an email to eonstorage+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eonstorage/3f5417bf-aa32-445a-9546-0f792acb69f7%40googlegroups.com.
vajonam
Aug 15, 2019, 8:43:21 PM8/15/19
to EON ZFS Storage
On NFS. I am using Ubuntu 18.04 clients over NFSv4. I have cifs enabled but don’t use much.
Andre Lue
Aug 15, 2019, 8:50:07 PM8/15/19
to EON ZFS Storage on behalf of Donovan Kaardal
If you're using nfsv4 on clients, make sure you configure eon nfs svr to match. I think its v3 by default which makes sense when you enable sync. Test v3 client, svr , v4 pair. I think v3 pair may perform better.
On Thu, Aug 15, 2019, 8:43 PM vajonam via EON ZFS Storage <eonstorage+APn2wQeaGveeVtAFGo6St...@googlegroups.com> wrote:
On NFS. I am using Ubuntu 18.04 clients over NFSv4. I have cifs enabled but don’t use much.
--
You received this message because you are subscribed to the Google Groups "EON ZFS Storage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eonstorage+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eonstorage/0c5678ec-5adc-4a89-9b94-2dda3e007b71%40googlegroups.com.
vajonam
Aug 16, 2019, 6:44:18 AM8/16/19
to EON ZFS Storage
Okay let me check the server versions and client versions.
vajonam
Aug 16, 2019, 6:47:29 AM8/16/19
to EON ZFS Storage
Just checked this, both sides are NFSv4 and the mounts are showing up with nfsv4
Message has been deleted
vajonam
Aug 16, 2019, 8:35:30 AM8/16/19
to EON ZFS Storage
tested out these combos.
client -- server -- result
---------------------------
v3 v3 freeze
v4 v3 freeze
v4 v4 freeze
v3,udp v3 freeze
v4,udp v3 freeze
v4,udp v4 freeze
v3,sync v3 okay,but slower
v4,sync v4 okay,but slower
vajonam
Aug 30, 2019, 10:14:34 AM8/30/19
to EON ZFS Storage
@dre2ske? any updates for me?
On somewhat related note, zpool import -a doesnt seem to setup the NFS shares.
I have to do zfs sharenfs pool/dataset after for 5 of my 7 pools. Once I do this however all is well.
I have re-added and SSD removed it as ZIL device, and added it as a swap device. writes are overall slower, but less lockups.
Reply all
Reply to author
Forward
0 new messages