Any FreeNAS experts - very slow NAS

[Mark]

I have an HP Microserver running FreeNAS. It was OK for a while and I
hadn't used it again for a while until I booted it up a couple of days
ago. Since then it has been horrendously slow. All file transfers
have been slow and logging in via the web interface takes an age.

Can anyone suggest the quickest/easiest way to diagnose the problem?
[CPU load is low, Memory utilitization OK, plenty of free disk space]

[jkn]

Hi Mark

This is pretty anecdotal, but FWIW: I have a Microserver running VortexBox.
Every now and again it seems to run slowly for me; at least, browsing the share
directories via SMB takes an age. I typically either ignore it or reboot,
just for the hell of it. Not very impressive I know...

Your posting has prompted me - next time it happens I'll investigate a bit more
and let you know if I discover anything.

Regards
Jon N

[Theo Markettos]

Can you SSH in and run 'top' when it is being slow? What processes are
taking CPU? What do the user/nice/system/interrupt/idle numbers look like?
If you press 'm' you should get I/O usage instead of CPU usage.

Theo

[Mark]

On 08 Oct 2015 15:03:12 +0100 (BST), Theo Markettos

last pid: 17488; load averages: 0.06, 0.05, 0.01 up 0+05:59:33
15:31:04
38 processes: 1 running, 37 sleeping
CPU: 15.2% user, 0.0% nice, 2.4% system, 1.0% interrupt, 81.5% idle
Mem: 187M Active, 105M Inact, 3873M Wired, 1492K Cache, 95M Buf, 3619M
Free
ARC: 3423M Total, 107M MFU, 3265M MRU, 699K Anon, 13M Header, 37M
Other
Swap: 6144M Total, 6144M Free

PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT
COMMAND
18442 root 24 35 0 0 0 0 0.00%
smbd
2614 root 0 0 0 0 0 0 0.00%
collectd
2445 root 2 0 0 0 0 0 0.00%
python2.7

Looks OK to me.

[Jaimie Vandenbergh]

Given you've turned it on after a long downtime, it might just be doing
a complete disk scrub. How much storage is there on it? Two days seems a
long time, mind.

You've already tried giving it another reboot? Checked for OS updates?

Cheers - Jaimie
--
human /mia'ow/ n.: Combination can-opener and heated chair-cover

[Rob Morley]

On Thu, 08 Oct 2015 08:53:39 +0100

What does SMART say?

[Mark]

On Fri, 09 Oct 2015 18:28:27 +0100, Jaimie Vandenbergh
<jai...@sometimes.sessile.org> wrote:

>On Thu, 08 Oct 2015 08:53:39 +0100, Mark
><i...@dontgetlotsofspamanymore.net> wrote:
>
>>I have an HP Microserver running FreeNAS. It was OK for a while and I
>>hadn't used it again for a while until I booted it up a couple of days
>>ago. Since then it has been horrendously slow. All file transfers
>>have been slow and logging in via the web interface takes an age.
>>
>>Can anyone suggest the quickest/easiest way to diagnose the problem?
>>[CPU load is low, Memory utilitization OK, plenty of free disk space]
>
>Given you've turned it on after a long downtime, it might just be doing
>a complete disk scrub.

According to the logs it hasn't done a scrub recently.

>How much storage is there on it? Two days seems a
>long time, mind.

3 * 3Tb disks but there's very little on them ATM.

>You've already tried giving it another reboot?

Yes. Didn't make it any quicker though.

>Checked for OS updates?

There are OS updates available but I am not sure this is the next step
given there may be a hardware issue.

[Mark]

On Fri, 9 Oct 2015 18:36:00 +0100, Rob Morley <nos...@ntlworld.com>
wrote:

All tests passed (short). I haven't done a long test recently.

[Mark]

Unfortunately there's an even worse problem. According to my backup
program (Syncback) it is failing to copy many files to the NAS device.

[Mike Tomlinson]

En el artículo <hr5q1b9gh99kj0vor...@4ax.com>, Mark
<i...@dontgetlotsofspamanymore.net> escribió:

>Unfortunately there's an even worse problem. According to my backup
>program (Syncback) it is failing to copy many files to the NAS device.

Error message(s) from the logs would help.

--
(\_/)
(='.'=) Bunny says: Windows 10? Nein danke!
(")_(")

[Mark]

On Tue, 13 Oct 2015 16:37:25 +0100, Mike Tomlinson
<mi...@jasper.org.uk> wrote:

>En el artículo <hr5q1b9gh99kj0vor...@4ax.com>, Mark
><i...@dontgetlotsofspamanymore.net> escribió:
>
>>Unfortunately there's an even worse problem. According to my backup
>>program (Syncback) it is failing to copy many files to the NAS device.
>
>Error message(s) from the logs would help.

"Error: Failed to copy from Source : Cannot copy file (59): An
unexpected network error occurred"

That's all. As far as I am concerned there's no problem with the
network. I was pinging the NAS server continuously with no losses.

[Chris Ridd]

That's not conclusive IMO. Does FreeNAS have any way to get the error
counts from the network interface stats? Effectively the output of
"netstat -i".

But what Syncback describes as a "network error" might be a disconnect,
or not something deemed an error by netstat. Does FreeNAS have any logs?
--
Chris

[Mike Tomlinson]

En el artículo <7o9q1b90ldmjj075i...@4ax.com>, Mark
<i...@dontgetlotsofspamanymore.net> escribió:

>"Error: Failed to copy from Source : Cannot copy file (59): An
>unexpected network error occurred"

C:\Users\mike>net helpmsg 59

An unexpected network error occurred.

H'm. Helpful. Not! :) Googling suggests this is caused by the server
(your NAS box) becoming somehow invisible to the client.

Things that come to mind:

1) ensure all network interfaces are set to use autonegotiate

2) is the connection between the PC and the NAS wired or wireless? if
wireless, try a wired connection

3) at the command prompt:

net use X: \\NASname\share

net use Y: \\NAS_IP_address\share

Does either of these work?

4) Is Syncback using UNC paths (\\servername\share) or mapped drive
letters?

[Mark]

On Tue, 13 Oct 2015 17:43:23 +0100, Chris Ridd <chri...@mac.com>
wrote:

>On 2015-10-13 15:46:10 +0000, Mark said:
>
>> On Tue, 13 Oct 2015 16:37:25 +0100, Mike Tomlinson
>> <mi...@jasper.org.uk> wrote:
>>
>>> En el artículo <hr5q1b9gh99kj0vor...@4ax.com>, Mark
>>> <i...@dontgetlotsofspamanymore.net> escribió:
>>>
>>>> Unfortunately there's an even worse problem. According to my backup
>>>> program (Syncback) it is failing to copy many files to the NAS device.
>>>
>>> Error message(s) from the logs would help.
>>
>> "Error: Failed to copy from Source : Cannot copy file (59): An
>> unexpected network error occurred"
>>
>> That's all. As far as I am concerned there's no problem with the
>> network. I was pinging the NAS server continuously with no losses.
>
>That's not conclusive IMO. Does FreeNAS have any way to get the error
>counts from the network interface stats? Effectively the output of
>"netstat -i".

0 errors according to netstat

>But what Syncback describes as a "network error" might be a disconnect,
>or not something deemed an error by netstat. Does FreeNAS have any logs?

Here's an extract from smbd log:

[2015/10/14 08:08:40.918276, 1]
../source3/smbd/service.c:1130(close_cnum)
pcname (ipv4:......:53145) closed connection to service .....
STATUS=daemon 'smbd' finished starting up and ready to serve
connections

Looks like smbd is misbehaving.

[Mark]

On Tue, 13 Oct 2015 17:43:51 +0100, Mike Tomlinson
<mi...@jasper.org.uk> wrote:

>En el artículo <7o9q1b90ldmjj075i...@4ax.com>, Mark
><i...@dontgetlotsofspamanymore.net> escribió:
>
>>"Error: Failed to copy from Source : Cannot copy file (59): An
>>unexpected network error occurred"
>
>C:\Users\mike>net helpmsg 59
>
>An unexpected network error occurred.
>
>H'm. Helpful. Not! :) Googling suggests this is caused by the server
>(your NAS box) becoming somehow invisible to the client.
>
>Things that come to mind:
>
>1) ensure all network interfaces are set to use autonegotiate

They are.

>2) is the connection between the PC and the NAS wired or wireless? if
>wireless, try a wired connection

Wired.

>3) at the command prompt:
>
> net use X: \\NASname\share
>
> net use Y: \\NAS_IP_address\share
>
>Does either of these work?

Both of these work.

>4) Is Syncback using UNC paths (\\servername\share) or mapped drive
>letters?

UNC paths.

The problem seems to occur more frequently with larger files and I do
also see failures when copying via Windows explorer:

"An unexpected error is keeping you from copying the file ......
Error 0x8007003B: An unexpected network error occurred."

If I click on the "Try Again" button each time then it eventually
succeeds.

I am keeping searching for this error message but haven't found
anything useful yet.

[Chris Ridd]

Still not necessarily ;-) What you need to determine is why one end is
closing the connection. It looks from that logging as though your PC
client is closing it, and smbd is just reacting to that by closing its
end.

The only place I know about on Windows to look at would be the Windows
Event Viewer, but I have no idea if that would have anything relevant
in it.

<passes baton to a Windows expert>
--
Chris

[Mark]

Well I have updated to the latest version of FreeNAS, but no
improvement.

[Mark]

On Wed, 14 Oct 2015 17:38:36 +0100, Chris Ridd <chri...@mac.com>

wrote:

>On 2015-10-14 07:21:51 +0000, Mark said:
>
>> On Tue, 13 Oct 2015 17:43:23 +0100, Chris Ridd <chri...@mac.com>
>> wrote:
>>
>>> But what Syncback describes as a "network error" might be a disconnect,
>>> or not something deemed an error by netstat. Does FreeNAS have any logs?
>>
>> Here's an extract from smbd log:
>>
>> [2015/10/14 08:08:40.918276, 1]
>> ../source3/smbd/service.c:1130(close_cnum)
>> pcname (ipv4:......:53145) closed connection to service .....
>> STATUS=daemon 'smbd' finished starting up and ready to serve
>> connections
>>
>> Looks like smbd is misbehaving.
>
>Still not necessarily ;-) What you need to determine is why one end is
>closing the connection. It looks from that logging as though your PC
>client is closing it, and smbd is just reacting to that by closing its
>end.
>
>The only place I know about on Windows to look at would be the Windows
>Event Viewer, but I have no idea if that would have anything relevant
>in it.

I don't see anything relevant in the Event Viewer.

[Theo Markettos]

Mark <i...@dontgetlotsofspamanymore.net> wrote:
> Looks OK to me.

Next up, what do your thermals look like?

sysctl -a | grep temp

You may need to:
sudo kldload coretemp
first.

I get things like:

dev.cpu.15.temperature: 33.0C
dev.cpu.15.coretemp.throttle_log: 0
dev.cpu.15.coretemp.tjmax: 87.0C
dev.cpu.15.coretemp.resolution: 1
dev.cpu.15.coretemp.delta: 54
dev.cpu.14.temperature: 32.0C
dev.cpu.14.coretemp.throttle_log: 0
dev.cpu.14.coretemp.tjmax: 87.0C
dev.cpu.14.coretemp.resolution: 1
dev.cpu.14.coretemp.delta: 55
dev.cpu.13.temperature: 32.0C
dev.cpu.13.coretemp.throttle_log: 0
dev.cpu.13.coretemp.tjmax: 87.0C
dev.cpu.13.coretemp.resolution: 1
dev.cpu.13.coretemp.delta: 55

If your thermals are high, do you have a fan issue? CPU heatsink loose,
dry thermal paste?

Theo

[Theo Markettos]

Mark <i...@dontgetlotsofspamanymore.net> wrote:
> I don't see anything relevant in the Event Viewer.

Anything relevant in /var/log/messages on the server?
Any other files in /var/log have anything of interest?

Theo

[Chris Ridd]

In particular, look for things around the same time that smbd logs that
the client is disconnecting.

--
Chris

[Mark]

On 15 Oct 2015 20:17:51 +0100 (BST), Theo Markettos

dev.cpu.0.temperature: 24.0C
dev.cpu.1.temperature: 24.0C
dev.amdtemp.0.%desc: AMD CPU On-Die Thermal Sensors
dev.amdtemp.0.%driver: amdtemp
dev.amdtemp.0.%parent: hostb4
dev.amdtemp.0.sensor0.core0: 24.0C

[Mark]

On 15 Oct 2015 20:20:09 +0100 (BST), Theo Markettos

There's lots of:
"Oct 16 07:40:20 server smbd[22032]: STATUS=daemon 'smbd' finished

starting up and ready to serve connections

matchname: host name/name mismatch: <IP address> != (NULL)"

[about one per 1-2 minutes]

I can't see anything else of interest.

[Mark]

Update:

I have bought a new switch (Gigabit) and disconnected everything from
the network. Then I have reconnected each thing one by one and seen
what happens. After reconnecting all but one thing (a NAS box) things
are now working well. I assume that it was the NAS box that was
causing the trouble, even though it was connected to a different
switch!

[JoeJoe]

I have experienced similar behaviour with my setup that is very similar
to yours (first with an N40L, and now with Gen8 microserver).

I always suspected the microserver to be the cause, but never managed to
get to the bottom of it.

One thing I did notice was when I used "LAN Speed Test". The speed is
only slow when trying to write to the HDDs on the server. The speed is
as it should when either reading from the HDDs on the server, or when
communicating with the NIC of the server.

This suggests to me that there may be a problem with the HDD controller
and/or driver?