What networked filesystem to use

[Harold Johanssen]

I have a setup in which my system has a limited amount of disk
space, but I am in a network with access to a server with all but
infinite amounts of disk space available. I need to use my system to
carry out lengthy software compilations, the kind that takes lots of disk
space. I therefore have to access disk space from the server above over
the 100 Mb LAN.

My question would be, What filesystem is likely to deliver the
best performance under the circumstances described?

I have tried with NFS, and it is not great. I will try next SSHFS
and WebDAV, but if anybody has experience with these, and other networked
filesystems, your feedback would be much appreciated.

[Tauno Voipio]

Both SSHFS and WebDAV have extra protocol overhead compared to
NFS, whch is the de facto standard on Unix family systems.

--

-TV

[Scott Hemphill]

What throughput are you getting? You might just be seeing the
limitation of the LAN. 100 Mb is pretty slow compared with disk
speeds.

Scott
--
Scott Hemphill hemp...@alumni.caltech.edu
"This isn't flying. This is falling, with style." -- Buzz Lightyear

[Grant Taylor]

On 11/25/20 11:00 AM, Harold Johanssen wrote:
> I have tried with NFS, and it is not great.

Did you use NFS's default parameters? Or did you try any tuning?



--
Grant. . . .
unix || die

[Grant Taylor]

On 11/25/20 11:00 AM, Harold Johanssen wrote:

> I have tried with NFS, and it is not great.

What version of NFS?

What protocol was it running over?

[David Brown]

On 25/11/2020 19:00, Harold Johanssen wrote:
> I have a setup in which my system has a limited amount of disk
> space, but I am in a network with access to a server with all but
> infinite amounts of disk space available. I need to use my system to
> carry out lengthy software compilations, the kind that takes lots of disk
> space. I therefore have to access disk space from the server above over
> the 100 Mb LAN.
>

That's going to limit your bandwidth to about 10 MB per second. Is that
your bottleneck?

> My question would be, What filesystem is likely to deliver the
> best performance under the circumstances described?
>
> I have tried with NFS, and it is not great. I will try next SSHFS
> and WebDAV, but if anybody has experience with these, and other networked
> filesystems, your feedback would be much appreciated.
>

sshfs will encrypt your traffic (usually). If you have slow processors
to go with your slow LAN, that might be a bottleneck. But if your
processors are fast enough for that not to be a problem, then you can
use the "-C" flag to sshfs to get compression in the traffic - for
compressible files, that will let you get higher bandwidth than the LAN
supports directly.

But the obvious solution to increased speed is to join the 21st century
and upgrade to 1Gb Ethernet, if that is practical for you.

[Harold Johanssen]

Thanks to all who replied. I have tried to do some fine tuning in
my NSF settings, as described in a number of places in the net. As for my
network - well, I'd love to be able to have a 1 Gb network, but that is
not going to happen in the short term, unfortunately.

My intent was, when posing this question, to get feedback on the
performance of different network filesystems with the networking
limitations that I described, from anybody who has had experience with
comparable setups. This is an all-Linux network.

[William Unruh]

Or buy a big disk to attach to your local system. If you can, go for
solid state disk, but if not get a 1TB or 2TB disk and use it.

100Mb is pretty slow compared to anything local. (And in practice you
will probably only get 50 or 30Mb/s)

[Scott Hemphill]

I no longer have a 100Mb network, but my memory is that I got similar
proportional throughput that I now get from my 1Gb network.

I used the command:

dd if=<bigNetworkFile> of=/dev/null bs=4k

and the dd summary line was:

7655663616 bytes (7.7 GB) copied, 65.8552 s, 116 MB/s

If you take the file size and divide by the time and multiply by 8
bits/byte, you get a read throughput of almost exactly 930Mb, or 93% of
the unattainable maximum. I haven't done anything fancy like trying to
use jumbo frames. This is using nfs version 4.

It's possible that with some work, you might do (a little) better with
sshfs.

https://www.admin-magazine.com/HPC/Articles/Sharing-Data-with-SSHFS

[Carlos E.R.]

Depends on what the actual load your software exerts. Large files? Small
files? Random r/w access? That's the worst. Just write a file and send
it whole? The later could be cached locally and then sent slowly while
the process is doing calculations.

I would simply test it.

sshfs should be slower, as it uses encryption (more CPU load). But it
can also use compression. So, test it.


Is a double network interface an option? Just a wild idea. No, I have
never done that.


--
Cheers, Carlos.

[Jorgen Grahn]

On Wed, 2020-11-25, Carlos E.R. wrote:
> On 25/11/2020 21.26, Harold Johanssen wrote:

...

>> My intent was, when posing this question, to get feedback on the
>> performance of different network filesystems with the networking
>> limitations that I described, from anybody who has had experience with
>> comparable setups. This is an all-Linux network.
>
> Depends on what the actual load your software exerts. Large files? Small
> files? Random r/w access? That's the worst.

But (I think) it could also be the scenario where you gain the most by
offloading the actual disk I/O to another machine, which perhaps has
disks with better random access, and more cache.

(Not that I know anything about this -- I've always managed to avoid
networked file systems. I use rsync or Git to make the data show up
in a local file system.)

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[David Brown]

Testing it is the only way to know, as the results depend on the cpu
speed (especially on the sending end) and the compressibility of the files.

>
>
> Is a double network interface an option? Just a wild idea. No, I have
> never done that.
>

Channel bundling can help on a good network setup, but it's unlikely to
help if he has cheapo or ancient switches, as he must have, since he is
on 100 Mb - they won't get enough parallel port-to-port transfers to
give significant improvements in throughput. (And if he has hubs,
rather than switches, it certainly won't help.)

Still, it's always fun to play with that kind of thing if you have the
hardware for it. You might not get much of a speed increase, but you
might learn something new and interesting.

[William Unruh]

On 2020-11-26, Jorgen Grahn <grahn...@snipabacken.se> wrote:
> On Wed, 2020-11-25, Carlos E.R. wrote:
>> On 25/11/2020 21.26, Harold Johanssen wrote:
> ...
>>> My intent was, when posing this question, to get feedback on the
>>> performance of different network filesystems with the networking
>>> limitations that I described, from anybody who has had experience with
>>> comparable setups. This is an all-Linux network.
>>
>> Depends on what the actual load your software exerts. Large files? Small
>> files? Random r/w access? That's the worst.
>
> But (I think) it could also be the scenario where you gain the most by
> offloading the actual disk I/O to another machine, which perhaps has
> disks with better random access, and more cache.

network speeds are (always?) slower than disk bus access speeds. Thus i
wold expect [ local<-> network<->remote computer <-> disk] to be slower than
[local computer <-> disk.]

[Carlos E.R.]

On 27/11/2020 17.21, William Unruh wrote:
> On 2020-11-26, Jorgen Grahn <grahn...@snipabacken.se> wrote:
>> On Wed, 2020-11-25, Carlos E.R. wrote:
>>> On 25/11/2020 21.26, Harold Johanssen wrote:
>> ...
>>>> My intent was, when posing this question, to get feedback on the
>>>> performance of different network filesystems with the networking
>>>> limitations that I described, from anybody who has had experience with
>>>> comparable setups. This is an all-Linux network.
>>>
>>> Depends on what the actual load your software exerts. Large files? Small
>>> files? Random r/w access? That's the worst.
>>
>> But (I think) it could also be the scenario where you gain the most by
>> offloading the actual disk I/O to another machine, which perhaps has
>> disks with better random access, and more cache.
>
> network speeds are (always?) slower than disk bus access speeds. Thus i
> wold expect [ local<-> network<->remote computer <-> disk] to be slower than
> [local computer <-> disk.]

Hum.

Server side has fast SSD disk and gigabyte Ethernet (let's be modest,
not fibre). Client side is laptop with rotating rust disk, could be 5400
rpm.

In this case, NFS may have faster access speed :-D

--
Cheers, Carlos.

[David Brown]

On 28/11/2020 07:37, Johann Beretta wrote:

> On 11/26/20 12:22 AM, David Brown wrote:
>
>>
>> Channel bundling can help on a good network setup, but it's unlikely to
>> help if he has cheapo or ancient switches, as he must have, since he is
>> on 100 Mb - they won't get enough parallel port-to-port transfers to
>> give significant improvements in throughput. (And if he has hubs,
>> rather than switches, it certainly won't help.)
>>
>

> It's a lot easier to upgrade a switch than to upgrade the wires in the
> walls. One can have a modern switch and old wiring. I'd not be so sure
> the person has "ancient" switches.

I think that sort of thing varies, and I'd be sceptical of guessing
which is most likely. Only the OP can tell us what applies in his case.
But I'd be surprised to find many cases where you have old wiring that
can't be replaced, good new switches that support channel bundling, and
where you have multiple old lines that can be used for bundling.

It's also worth noting that old wires can often support higher speeds.
Ethernet cables are rated for a particular speed and length - usually
110m. Very often, you have much shorter cables than 110m, and these can
sometimes (but not always) support higher speeds than their rating might
suggest. You might have to be careful about details, such as minimising
the length of the final segment (the bit from the socket in the wall to
the computer in the office).

Again, we know very little about the OP's setup.

>
> The odds of encountering someone with a hub is almost nil. I remember
> when the first consumer switches hit the market and that was more than
> 20 years ago.
>

I actually kept a couple of old hubs around for some kinds of testing -
they could be quite handy when trying to monitor the traffic of a
particular system. (You put the hub between the device and the rest of
the network, then you can have a laptop with wireshark listening to all
the traffic.) A managed switch set up with a mirror port does the job
too, but the hubs I had were going free!

I suspect that in older networks, you might still find a few hubs -
equipment like this rarely gets replaced unless there is a problem, and
just like old wiring, there is often old equipment still in use.

[David Brown]

On 28/11/2020 21:37, Johann Beretta wrote:

> On 11/28/20 7:22 AM, David Brown wrote:
>
>>
>> I suspect that in older networks, you might still find a few hubs -
>> equipment like this rarely gets replaced unless there is a problem, and
>> just like old wiring, there is often old equipment still in use.
>>
>

> Nah... A switch can be had for $10 and have been in production for far
> longer than hubs.

The switches that existed when hubs were popular were massively
expensive. And no switch has ever cost $10 - excluding thrown-out
second-hand devices that would not be used in a serious network.

But in corporate networks, there is a lot more involved than just the
cost of the hardware. The cost of changing things is often more
relevant than the hardware itself - the downtime, the mounting, the
paperwork, and so on.

> When was the last time you could even find a hub for
> sale in a store? 2005?
>

I can't remember - but I think it was a long time before that when we
last bought one. However, it's perhaps 5 years since I last removed an
old hub from one of our networks.

[William Unruh]

On 2020-11-29, Johann Beretta <ber...@nun-ya-bizness.com> wrote:
> This is a multi-part message in MIME format.
> --------------F2A5B81AEF67C13D33627186
> Content-Type: text/plain; charset=utf-8
> Content-Transfer-Encoding: quoted-printable

>
> On 11/29/20 4:26 AM, David Brown wrote:
>> On 28/11/2020 21:37, Johann Beretta wrote:
>>> On 11/28/20 7:22 AM, David Brown wrote:
>>>
>>>>
>>>> I suspect that in older networks, you might still find a few hubs -

>>>> equipment like this rarely gets replaced unless there is a problem, a=

> nd
>>>> just like old wiring, there is often old equipment still in use.
>>>>
>>>

>>> Nah... A switch can be had for $10 and have been in production for far=
>
>>> longer than hubs.
>>=20

>> The switches that existed when hubs were popular were massively
>> expensive. And no switch has ever cost $10 - excluding thrown-out

^^^^^^^^^

>> second-hand devices that would not be used in a serious network.

>>=20
>
> Be careful when making absolute claims. You said no switch, including
^^^^^^^^^
> thrown-out second hand devices....
>
> https://www.ebay.com/itm/HP-Procurve-2610-48-J9088A-48-Port-10-100-1000-E=
> thernet-Switch/274592476654?hash=3Ditem3feefcadee:g:3sUAAOSw~kFfwUyq
>
> $10 opening bid (which means it MIGHT sell for $10.)
>

[David Brown]

On 29/11/2020 19:44, Johann Beretta wrote:
> On 11/29/20 4:26 AM, David Brown wrote:

>> On 28/11/2020 21:37, Johann Beretta wrote:
>>> On 11/28/20 7:22 AM, David Brown wrote:
>>>
>>>>
>>>> I suspect that in older networks, you might still find a few hubs -
>>>> equipment like this rarely gets replaced unless there is a problem, and
>>>> just like old wiring, there is often old equipment still in use.
>>>>
>>>
>>> Nah... A switch can be had for $10 and have been in production for far
>>> longer than hubs.
>>
>> The switches that existed when hubs were popular were massively
>> expensive. And no switch has ever cost $10 - excluding thrown-out
>> second-hand devices that would not be used in a serious network.
>>
>

> Be careful when making absolute claims. You said no switch, including

> thrown-out second hand devices....
>
> https://www.ebay.com/itm/HP-Procurve-2610-48-J9088A-48-Port-10-100-1000-Ethernet-Switch/274592476654?hash=item3feefcadee:g:3sUAAOSw~kFfwUyq

>
> $10 opening bid (which means it MIGHT sell for $10.)
>

Yes - it's an old used, second-hand device, as I said.

But I was perhaps a bit quick in suggesting that no one would buy such a
device for a serious network. When a new 48 port switch can be bought
for as little as about $100, I don't think companies would want to get
critical infrastructure as left-overs found cheaply on Ebay. However,
it is entirely possible that someone would want this particular type of
switch to replace an existing one, and buy it for the model rather than
for the price.

[David Brown]

On 29/11/2020 23:01, Johann Beretta wrote:

> On 11/29/20 1:02 PM, David Brown wrote:
>
>> Yes - it's an old used, second-hand device, as I said.
>

> I know. I misread "excluding" as "including". Mea culpa

No problem. ("Mea culpa" is a rarely used phrase on Usenet!)

>
>>
>> But I was perhaps a bit quick in suggesting that no one would buy such a
>> device for a serious network. When a new 48 port switch can be bought
>> for as little as about $100, I don't think companies would want to get
>> critical infrastructure as left-overs found cheaply on Ebay. However,
>> it is entirely possible that someone would want this particular type of
>> switch to replace an existing one, and buy it for the model rather than
>> for the price.
>

> Agreed.
>

It's also worth noting that while a little long in the tooth, the HP
switch is a managed switch with more features than the cheap unmanaged
48 port device I found. And the modern replacement for the HP 2610-48
is, as far as I can see, the HP 2620-48 with a new price of about $1800
(that's Norway prices, which might be higher than you see in other
countries). I don't know what that switch does more than you'd get from
a new D-Link DGS-1210-48 managed switch for about $350, but I've never
really understood how some manufacturers seem to charge huge prices for
network equipment. (And these same HPE folk make some wonderful little
microservers at entirely reasonable prices.)

[Tristan B. Kildaire]

You might hate me for this, sshfs

[Jim Jackson]

I use sshfs regularly - but the performance is not good.

[William Unruh]

I have no idea what "not great" means. eo have a 100Mb/s ethernet, that
is about 10MB/s maximum, which is usually more like 1-5MB/s (remember
b=bits, and B=bytes, with a ratio of about 10 between them.) I suspect
that is the blockage and anything else is going add additional overhead
and slow things down more. sshfs has the advantage of encrytion in
flight, so if you are worried about someone hacking your connection
between your computer and the server, that may be a good idea. On the
other hand if it is inside your home, and you are worried about hackers,
maybe you have more severe problems than networking. If the server is
remote, then you have the problems of all of the routers between you and
the server slowing things down still more.
Buy another disk for your computer.

Note that software compilations are horrible to do with remote storage.
You want a local disk, preferably an SSD. There is far too much data churn in
compilations.


>