Advice needed about ZFS

stripes theotoky

unread,

Aug 20, 2020, 2:53:43 AM8/20/20

to LUV mailing list, mlu...@googlegroups.com

Dear Group,

I am looking for advice on ZFS. My computer is running painfully slowly. I assume this is due to ZFS snapshots eating the entire disk space. My knowledge of ZFS is almost zero, this box was installed by a friend in the UK back in 2016 in the days when Ubuntu 16:04 was experimental and is still running 16.04. I have run the following commands

sudo zpool set listsnapshots=on alexandria

zfs list -o space -r alexandria

Which seems to show the space is being eaten by snapshots of the home directory and oddly that there are no snapshots from before 2019. The computer is in effect unuseable at the moment. All suggestions are very welcome.

The box is a dual boot Linux / Windows but the windows partition could be drastically reduced in size if this is the real problem.

NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
alexandria 1.06G 332G 320K 96K 0 332G
alexandria@zfs-auto-snap_monthly-2019-07-13-0141 - 64K - - - -
alexandria@zfs-auto-snap_monthly-2019-08-25-1034 - 0 - - - -
alexandria@zfs-auto-snap_monthly-2019-10-09-0831 - 0 - - - -
alexandria@zfs-auto-snap_monthly-2019-11-09-2312 - 0 - - - -
alexandria@zfs-auto-snap_monthly-2019-12-09-1923 - 0 - - - -
alexandria@zfs-auto-snap_monthly-2020-01-23-0728 - 0 - - - -
alexandria@zfs-auto-snap_monthly-2020-02-27-0408 - 0 - - - -
alexandria@zfs-auto-snap_monthly-2020-03-28-0921 - 0 - - - -
alexandria@zfs-auto-snap_monthly-2020-04-27-1130 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-05-18-0221 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-05-19-0117 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-05-20-0358 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-05-21-0350 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-05-22-0105 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-05-27-0155 - 0 - - - -
alexandria@zfs-auto-snap_monthly-2020-05-27-0156 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-05-28-0349 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-06-02-0055 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-06-04-0725 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-06-09-0753 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-06-10-1034 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-06-17-0905 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-06-23-0415 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-06-29-0722 - 0 - - - -
alexandria@zfs-auto-snap_monthly-2020-06-29-0732 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-06-30-0701 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-07-02-0619 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-07-03-0401 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-07-09-0110 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-07-19-0706 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-03-0048 - 0 - - - -
alexandria@zfs-auto-snap_monthly-2020-08-03-0055 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-05-0229 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-06-0052 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-07-0202 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-08-1109 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-09-1053 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-10-0157 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-11-0347 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-12-0849 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-16-0601 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-16-1517 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-16-1617 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-16-1717 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-16-1817 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-16-1917 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-16-2017 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-16-2117 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-16-2217 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-16-2317 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-0017 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-0117 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-0217 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-0317 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-0417 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-0517 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-0617 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-0717 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-17-0738 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-0817 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-0917 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-17-1017 - 0 - - - -
alexandria@zfs-auto-snap_daily-2020-08-20-0201 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-20-0217 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-20-0317 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-20-0417 - 0 - - - -
alexandria@zfs-auto-snap_frequent-2020-08-20-0430 - 0 - - - -
alexandria@zfs-auto-snap_frequent-2020-08-20-0445 - 0 - - - -
alexandria@zfs-auto-snap_frequent-2020-08-20-0500 - 0 - - - -
alexandria@zfs-auto-snap_frequent-2020-08-20-0515 - 0 - - - -
alexandria@zfs-auto-snap_hourly-2020-08-20-0517 - 0 - - - -
alexandria/home 1.06G 332G 90.3G 242G 0 0
alexandria/home@zfs-auto-snap_monthly-2019-07-13-0141 - 328M - - - -
alexandria/home@zfs-auto-snap_hourly-2019-07-18-0319 - 46.6M - - - -
alexandria/home@zfs-auto-snap_hourly-2019-07-18-0818 - 37.5M - - - -
alexandria/home@zfs-auto-snap_hourly-2019-07-19-0019 - 83.3M - - - -
alexandria/home@zfs-auto-snap_monthly-2019-08-25-1034 - 282M - - - -
alexandria/home@zfs-auto-snap_monthly-2019-10-09-0831 - 241M - - - -
alexandria/home@zfs-auto-snap_monthly-2019-11-09-2312 - 787M - - - -
alexandria/home@zfs-auto-snap_monthly-2019-12-09-1923 - 975M - - - -
alexandria/home@zfs-auto-snap_monthly-2020-01-23-0728 - 513M - - - -
alexandria/home@zfs-auto-snap_monthly-2020-02-27-0408 - 810M - - - -
alexandria/home@zfs-auto-snap_monthly-2020-03-28-0921 - 754M - - - -
alexandria/home@zfs-auto-snap_monthly-2020-04-27-1130 - 704M - - - -
alexandria/home@zfs-auto-snap_daily-2020-05-20-0358 - 277M - - - -
alexandria/home@zfs-auto-snap_daily-2020-05-21-0350 - 41.6M - - - -
alexandria/home@zfs-auto-snap_daily-2020-05-22-0105 - 34.3M - - - -
alexandria/home@zfs-auto-snap_daily-2020-05-27-0155 - 2.57M - - - -
alexandria/home@zfs-auto-snap_monthly-2020-05-27-0156 - 2.84M - - - -
alexandria/home@zfs-auto-snap_daily-2020-05-28-0349 - 67.7M - - - -
alexandria/home@zfs-auto-snap_daily-2020-06-02-0055 - 63.7M - - - -
alexandria/home@zfs-auto-snap_daily-2020-06-04-0725 - 11.3M - - - -
alexandria/home@zfs-auto-snap_weekly-2020-06-04-0730 - 14.9M - - - -
alexandria/home@zfs-auto-snap_daily-2020-06-09-0753 - 82.0M - - - -
alexandria/home@zfs-auto-snap_daily-2020-06-10-1034 - 137M - - - -
alexandria/home@zfs-auto-snap_daily-2020-06-17-0905 - 3.11M - - - -
alexandria/home@zfs-auto-snap_weekly-2020-06-17-0910 - 1.27M - - - -
alexandria/home@zfs-auto-snap_daily-2020-06-23-0415 - 276K - - - -
alexandria/home@zfs-auto-snap_daily-2020-06-23-0416 - 280K - - - -
alexandria/home@zfs-auto-snap_daily-2020-06-29-0722 - 0 - - - -
alexandria/home@zfs-auto-snap_daily-2020-06-29-0723 - 0 - - - -
alexandria/home@zfs-auto-snap_weekly-2020-06-29-0728 - 14.2M - - - -
alexandria/home@zfs-auto-snap_monthly-2020-06-29-0732 - 4.17M - - - -
alexandria/home@zfs-auto-snap_daily-2020-06-30-0701 - 58.6M - - - -
alexandria/home@zfs-auto-snap_daily-2020-07-02-0619 - 152K - - - -
alexandria/home@zfs-auto-snap_daily-2020-07-02-0620 - 156K - - - -
alexandria/home@zfs-auto-snap_daily-2020-07-03-0401 - 69.9M - - - -
alexandria/home@zfs-auto-snap_daily-2020-07-09-0110 - 6.30M - - - -
alexandria/home@zfs-auto-snap_weekly-2020-07-09-0113 - 6.14M - - - -
alexandria/home@zfs-auto-snap_daily-2020-07-19-0706 - 10.7M - - - -
alexandria/home@zfs-auto-snap_weekly-2020-07-19-0710 - 7.14M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-03-0048 - 1.11M - - - -
alexandria/home@zfs-auto-snap_weekly-2020-08-03-0050 - 732K - - - -
alexandria/home@zfs-auto-snap_monthly-2020-08-03-0055 - 1.06M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-05-0229 - 5.16M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-06-0052 - 119M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-07-0202 - 105M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-08-1109 - 52.5M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-09-1053 - 57.5M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-10-0157 - 5.29M - - - -
alexandria/home@zfs-auto-snap_weekly-2020-08-10-0202 - 5.28M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-11-0347 - 86.0M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-12-0849 - 71.0M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-16-0601 - 80K - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-16-0602 - 80K - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-16-1517 - 8.85M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-16-1617 - 8.65M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-16-1717 - 8.75M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-16-1817 - 8.55M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-16-1917 - 8.61M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-16-2017 - 8.54M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-16-2117 - 8.59M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-16-2217 - 8.65M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-16-2317 - 8.70M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-0017 - 8.49M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-0117 - 9.46M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-0217 - 8.75M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-0317 - 19.0M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-0417 - 19.0M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-0517 - 21.2M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-0617 - 38.5M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-0717 - 92.2M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-17-0738 - 15.0M - - - -
alexandria/home@zfs-auto-snap_weekly-2020-08-17-0740 - 9.04M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-0817 - 34.8M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-0917 - 20.9M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-17-1017 - 19.5M - - - -
alexandria/home@zfs-auto-snap_daily-2020-08-20-0201 - 15.0M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-20-0217 - 14.7M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-20-0317 - 19.5M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-20-0417 - 2.98M - - - -
alexandria/home@zfs-auto-snap_frequent-2020-08-20-0430 - 1.77M - - - -
alexandria/home@zfs-auto-snap_frequent-2020-08-20-0445 - 1.74M - - - -
alexandria/home@zfs-auto-snap_frequent-2020-08-20-0500 - 1.76M - - - -
alexandria/home@zfs-auto-snap_frequent-2020-08-20-0515 - 1.44M - - - -
alexandria/home@zfs-auto-snap_hourly-2020-08-20-0517 - 1.23M - - - -
stripinska@Stripinska:~$

--

Stripes Theotoky

-37 .713869
145.050562

Russell Shaw

unread,

Aug 20, 2020, 3:33:22 AM8/20/20

to 'stripes theotoky' via mlug-au

On 20/8/20 4:53 pm, 'stripes theotoky' via mlug-au wrote:
> Dear Group,
>
> I am looking for advice on ZFS. My computer is running painfully slowly.

Check your pc has 8G of ram or more.

I'm still using an old pc with DDR2 ram and it runs pretty fast after
upgrading it to 16GB with ram from ebay.

stripes theotoky

unread,

Aug 20, 2020, 5:56:35 AM8/20/20

to mlu...@googlegroups.com

On Thu, 20 Aug 2020 at 07:33, Russell Shaw <rjs...@netspace.net.au> wrote:

On 20/8/20 4:53 pm, 'stripes theotoky' via mlug-au wrote:
> Dear Group,
>
> I am looking for advice on ZFS. My computer is running painfully slowly.

Check your pc has 8G of ram or more.

It has 16G

I'm still using an old pc with DDR2 ram and it runs pretty fast after
upgrading it to 16GB with ram from ebay.

--
You received this message because you are subscribed to the Google Groups "mlug-au" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlug-au+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mlug-au/9dbdfda6-ca70-bcdd-10de-9a6cbbdf6dc9%40netspace.net.au.

Russell Shaw

unread,

Aug 20, 2020, 6:05:00 AM8/20/20

to 'stripes theotoky' via mlug-au

On 20/8/20 7:56 pm, 'stripes theotoky' via mlug-au wrote:
> On Thu, 20 Aug 2020 at 07:33, Russell Shaw <rjs...@netspace.net.au
> <mailto:rjs...@netspace.net.au>> wrote:
>
> On 20/8/20 4:53 pm, 'stripes theotoky' via mlug-au wrote:
> > Dear Group,
> >
> > I am looking for advice on ZFS. My computer is running painfully slowly.
>
> Check your pc has 8G of ram or more.
>
> It has 16G

NAME AVAIL USED
alexandria 1.06G 332G

The free space looks pretty low if much buffering is needed.

Think more free space is needed.

stripes theotoky

unread,

Aug 20, 2020, 8:40:19 AM8/20/20

to mlu...@googlegroups.com, LUV mailing list, Colin Fee

When we started this discussion we had this

stripinska@Stripinska:~$ sudo zfs list

NAME USED AVAIL REFER MOUNTPOINT
alexandria 332G 1.06G 96K /alexandria

Now having moved 47.3 GB of files to an external drive I have this.

stripinska@Stripinska:~$ sudo zfs list
NAME USED AVAIL REFER MOUNTPOINT
alexandria 332G 782M 96K /alexandria

What is eating my space?

It is not the cache for Firefox as this is only 320M.

How do I get it back before the box freezes.

The more disk space I free the less available space I have.

Stripes.

--
You received this message because you are subscribed to the Google Groups "mlug-au" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlug-au+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mlug-au/8bcd2b83-a63c-3cc5-26c7-d2d05dbadf4e%40netspace.net.au.

--

Stripes Theotoky

-37 .713869
145.050562

Russell Shaw

unread,

Aug 20, 2020, 12:13:14 PM8/20/20

to 'stripes theotoky' via mlug-au

On 20/8/20 10:40 pm, 'stripes theotoky' via mlug-au wrote:
> When we started this discussion we had this
>
> stripinska@Stripinska:~$ sudo zfs list
> NAME USED AVAIL REFER
> MOUNTPOINT
> alexandria 332G 1.06G 96K
> /alexandria
>
> Now having moved 47.3 GB of files to an external drive I have this.
>
> stripinska@Stripinska:~$ sudo zfs list
> NAME USED AVAIL REFER
> MOUNTPOINT
> alexandria 332G 782M 96K
> /alexandria
>
> What is eating my space?
> It is not the cache for Firefox as this is only 320M.
> How do I get it back before the box freezes.
> The more disk space I free the less available space I have.

You could try: du -h|sort -n

but do it in subdirectories or you might be waiting a while.

Malcolm Herbert

unread,

Aug 20, 2020, 10:02:54 PM8/20/20

to mlug-au

Stripes - you will need to remove old/stale snapshots in order to recover space in your zpool. I have found that ZFS becomes very laggy once zpool capacity goes above 80% - I've had to dial back the rate at which snapshots are created on my hosts for the same reasons

Deleting files in your filesystem that are still referred to by snapshots doesn't free up the data - all references to those blocks need to be removed before the usage in the zpool will drop.

feel free to email me offline if you want to discuss this further

Regards,
Malcolm

--
Malcolm Herbert
mj...@mjch.net

Gary Pope

unread,

Aug 22, 2020, 5:37:49 AM8/22/20

to mlu...@googlegroups.com

https://www.ixsystems.com/community/threads/space-not-reclaimed-after-deleting-files.62318/

Any clues from that help?

Gary

--

--
Gary A. Pope
B.Bus(ACC)
DIRECTOR

Alchester Business Systems
m: 0408-994799 anytime.
p: 03-97626293
e: g...@alchester.com.au
w: www.alchester.com.au
Remote: Communications powered by ABSoutback3!
“We take care of everything!"

Darren Wurf

unread,

Aug 22, 2020, 8:25:30 PM8/22/20

to mlu...@googlegroups.com

I noticed the snapshots follow a grandparent-parent-child pattern, something is managing these snapshots for you - which would explain why you have no older ones as well as why there are many recent snapshots and few older ones.

I would try deleting the oldest snapshot and checking your free space again. Repeat until you have more space. I agree with Malcolm, zfs hates life when it doesn't have some free space to play with and that is likely the reason your computer is slow

--
You received this message because you are subscribed to the Google Groups "mlug-au" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlug-au+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mlug-au/be6c5382-ffbd-4e44-af2b-4e005253dfaf%40www.fastmail.com.

Keith Bainbridge

unread,

Aug 23, 2020, 1:55:36 AM8/23/20

to mlu...@googlegroups.com

On 23/8/20 10:25 am, Darren Wurf wrote:
> I noticed the snapshots follow a grandparent-parent-child pattern,
> something is managing these snapshots for you - which would explain why
> you have no older ones as well as why there are many recent snapshots
> and few older ones.

And if that is the case, that manager may be using hard links to reduce
the space being used. So deleting older snapshots may not have a big
affect.

I know timeshift works that way - the grandparent-parent-child pattern
and hard links.

--
Keith Bainbridge

keithrb...@gmail.com
or ke1tho...@gmx.com

stripes theotoky

unread,

Aug 24, 2020, 3:47:38 AM8/24/20

to mlu...@googlegroups.com, LUV mailing list, Malcolm Herbert

Thank you all for the very helpful advice:-

If I understand you all the problems are ZFS is choking because I don't have enough disk space. The reason I don't have enough disk space is because the files I deleted are still in the old snapshots, plus it was already pretty full.

This seems like a good way forward but is it really and if so how do I do it?
First things first ZFS is choking due to lack of disk space. However, I have a lot of totally unused diskspace on a windows partition so how can I reduce the windows partition and increase the ZFS to stop it choking. It has 560GB of which I do not need more than 200max to leave for Windoze.

Gparted apparently doesn't like ZFS but does this matter? I can I assume use it to shrink the Windows partition and free up about 360GB, then how do I expand the ZFS to use that free space?

Second, I have checked /etc/cron and find auto snap shot commands for hourly, daily, weekly & monthly

hourly = 24, daily = 31, weekly = 8, monthly = 6. It seems I could change that to hourly = 24, daily = 7, weekly = 4, monthly = 6 and get pretty much the same coverage with a lot less snapshots.

Listing snapshots shows some from 2019, where are they coming from as with monthly only storing 6 months there shouldn't be anything newer than February 2020 or am I not understanding something here?

The computer is a Lenovo W541 laptop, the longer term plan is to double the memory to 32GB and put a 2TB SSD in this box. Does that sound sensible?

In the meantime, a thorough backup is running in at least two external drives (one incremental and one fresh).

Stripes.

--
You received this message because you are subscribed to the Google Groups "mlug-au" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlug-au+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mlug-au/a8673e8a-117b-e3b3-3f61-6316b46fac13%40gmail.com.

Malcolm Herbert

unread,

Aug 24, 2020, 8:15:37 PM8/24/20

to stripes theotoky, mlug-au, LUV mailing list

On Mon, 24 Aug 2020, at 17:47, stripes theotoky wrote:
> If I understand you all the problems are ZFS is choking because I don't
> have enough disk space. The reason I don't have enough disk space is
> because the files I deleted are still in the old snapshots, plus it was
> already pretty full.

that's my understanding, yes.

> This seems like a good way forward but is it really and if so how do I
> do it?

see below for creating an offline copy of your data first, but the command you will need to remove snapshots will be of the form

zfs destroy <dataset>@<snapshot>

I think you already have something similar to this, but to identify which snapshots you have, use

zfs list -t all

the snapshots are those with the '@' in their name. When destroying these, work from the oldest forward

> First things first ZFS is choking due to lack of disk space. However, I
> have a lot of totally unused diskspace on a windows partition so how
> can I reduce the windows partition and increase the ZFS to stop it
> choking. It has 560GB of which I do not need more than 200max to leave
> for Windoze.

I'd not be contemplating doing this until you had an offline copy of your data somewhere first. Whilst zfs can grow, it can't shrink and I'm not familiar enough with windows to advise there, but it certainly seems possible

> Gparted apparently doesn't like ZFS but does this matter? I can I
> assume use it to shrink the Windows partition and free up about 360GB,
> then how do I expand the ZFS to use that free space?

I'm hesitant to advise here, because I don't think we have a good knowledge of your disk layout ... I have vague recollections that you might also have LVM on there somewhere and also an SD card? Did you provide a list of drives and what's on them earlier? I might have missed that, sorry

> Second, I have checked /etc/cron and find auto snap shot commands for
> hourly, daily, weekly & monthly
> hourly = 24, daily = 31, weekly = 8, monthly = 6. It seems I could
> change that to hourly = 24, daily = 7, weekly = 4, monthly = 6 and get
> pretty much the same coverage with a lot less snapshots.

yeah, I don't have it on hand, but that was a similar realisation I came to when I reviewed it last

> Listing snapshots shows some from 2019, where are they coming from as
> with monthly only storing 6 months there shouldn't be anything newer
> than February 2020 or am I not understanding something here?

that I'm not sure of, but from the sounds of it those might be good candidates to remove first as they'll likely be referencing blocks that are in the oldest versions of the files in your zpool

> The computer is a Lenovo W541 laptop, the longer term plan is to double
> the memory to 32GB and put a 2TB SSD in this box. Does that sound
> sensible?

ZFS eats memory, so the more the better and certainly more disk space is required given your current experiences ...

there seem to be many variants of your laptop though - does yours have two drive bays or just the one with a DVD drive? I'd recommend keeping your OS and data segregated as much as possible and having it on another drive gives you better recovery options when one or the other fails, so maybe losing the DVD drive and replacing with an HDD or SDD for data might be something to contemplate ... not sure whether you might still be able to get the drive bay blank any longer, but it's probably on ebay somewhere

> In the meantime, a thorough backup is running in at least two external
> drives (one incremental and one fresh).

ok ... even so, if you have a spare external drive of at least the same capacity as your current zpool (2x would be best), I would recommend creating a new zpool on the entire disk purely for making large scale changes easier in the future. as you have snapshots being created automatically for you, that's most of the work done in being able to do zfs-level backups into another zpool - once that's created, you can use zfs send and zfs receive to push data between the two pools as required ... this would preserve the structure and snapshots as per your original zpool ... a few more moving parts, but it's not intended as an archive so much as a point-in-time backup to get you out of a jam at the moment.

anyway, that's enough from me - hope this has helped, will keep a closer eye on the mailing list if you need further suggestions

stripes theotoky

unread,

Aug 26, 2020, 6:43:42 AM8/26/20

to Mark Trickett, LUV mailing list, mlu...@googlegroups.com, Malcolm Herbert

Hello Mark,

Question, do you really need to recover files from the snapshots on a
routine basis?

No I don't. I have never recovered anything from ZFS and hope to never have to. ZFS is a last ditch line of defense against some of my co-authors who have a terrible habit of working on old files, saving them out with the same name as the current working file. More than once a changed introduction has destroyed years of work and it has only been my paranoia in having about 6 backups of everything that has saved the project.

I would suggest that you consider the frequency of the
snapshots, and of backups. If you truly understand, and take care with
your actions, how much do you need to recover and undelete as the
snapshots enable? If you do not need quite so frequent, nor so many
snapshots, then you will have more usable space when you transfer to a
new drive.

The trouble is I am often working on 4 - 5 academic papers at the same time; some of them maybe neglected for a couple of months hence errors won't be noticed until I come back to work on it again. In the worst case I had to dig up files from 3 years ago to recover data.

The other matter is that the snapshots are not a backup strategy, they
too are lost along with the current copies if you have a hard drive
crash.

I realise that. The reasons for ZFS are outlined above. I know a better strategy would be to use GIT but as my co-authors are very smart in maths and economics but have almost zero computer knowledge, trying to tell them what GIT is let alone why we should be using it is impossible.

I would suggest looking at something like a Network Attached
Storage device, with multiple drives in a suitable RAID array.

This is the ultimate plan to build a NAS from an HP Microserver. I am leaning towards Nas4Free on an SSD or internal USB and 3, 6TB mirrors. This is a project that has to wait because right now due to Covid19 and Brexit we are not sure where we are. I am here and can't leave but expecting to be out of work (which won't stop my research), my husband is British/Australian, resident in Austria to avoid Brexit but is stranded by Covid in Greece. When it all settles down and we have a home again building this NAS is going to be pretty high on the list of things to do.

There are others on the LUV lists who can better advise on which strategies
will actually provide a reasonable measure of security of data.
Remember that there are repositories from which you can restore the OS
and applications, but that your data, including particular
configuration, can only be recovered from a suitable backup on another
device, and best often stored on a separate site.

Much of the data is on external disks located in 3 different countries so hopefully it is safe.

These are pointers to think about as to what is important, and what
can you afford to loose, possibly because you can regenerate, possibly
because it is not really that critical. As to life, yes the data has
meaning to each of us, but it is not food and water even if it can be
used in exchange for such.

True.

> First things first ZFS is choking due to lack of disk space. However, I
> have a lot of totally unused diskspace on a windows partition so how can I
> reduce the windows partition and increase the ZFS to stop it choking. It
> has 560GB of which I do not need more than 200max to leave for Windoze.

From this I gather that you find a need to have the Microsoft malware
available. I have problems now and then when I have to deal
interactively with a Microsoft Word document in the newest format. I
usually point out that Word may well be widespread, but that is not
universal, and a PDF can be locked to prevent being tampered with.

Windows certainly is malware. I have it to run Scientific Workplace (SW) as some of my co-authors are unable to deal with Latex. In an ideal world I would only use Kile or Eclipse but sometimes I don't have the hours / days necessary to edit the perverted abomination that Scientific Workplace thinks is a tex file into an actual tex file, hence the need to have the windoze to run SW.

> Gparted apparently doesn't like ZFS but does this matter? I can I assume
> use it to shrink the Windows partition and free up about 360GB, then how do
> I expand the ZFS to use that free space?
> Second, I have checked /etc/cron and find auto snap shot commands for
> hourly, daily, weekly & monthly
> hourly = 24, daily = 31, weekly = 8, monthly = 6. It seems I could change
> that to hourly = 24, daily = 7, weekly = 4, monthly = 6 and get pretty much
> the same coverage with a lot less snapshots.
>
> Listing snapshots shows some from 2019, where are they coming from as with
> monthly only storing 6 months there shouldn't be anything newer than
> February 2020 or am I not understanding something here?

I would think that if the computer was not in use at the critical
time, the snapshots could remain. Cron cannot do things while the
computer is off. I know that there is an alternative, anacron, that is
coded to cope with doing the actions that fell while the computer was
off, when it is next booted. It would be worth checking which is
installed and used.

I didn't know about anacron, but it makes sense that it could be installed, I will have to check.

> The computer is a Lenovo W541 laptop, the longer term plan is to double the
> memory to 32GB and put a 2TB SSD in this box. Does that sound sensible?
>
> In the meantime, a thorough backup is running in at least two external
> drives (one incremental and one fresh).

Again, consider backups on a separate device. Again, look to your use
patterns and practices so that you have less need for the snapshots
and recovering deleted files, and as such can use less snapshots. Look
at what you are trying to achieve, and ask how to be reasonably
effective. Try to make the most of the resources you can afford,
rather than spending too much to compensate for suboptimal habits and
practices.

Good advice. Find new co-authors? LOL

Thanks again for your help.

> Stripes.

Regards,

Mark Trickett

Russell Shaw

unread,

Aug 27, 2020, 12:47:40 AM8/27/20

to 'stripes theotoky' via mlug-au

On 26/8/20 8:43 pm, 'stripes theotoky' via mlug-au wrote:
> Hello Mark,
>
> Question, do you really need to recover files from the snapshots on a
> routine basis?
>
>
> No I don't. I have never recovered anything from ZFS and hope to never
> have to. ZFS is a last ditch line of defense against some of my
> co-authors who have a terrible habit of working on old files, saving
> them out with the same name as the current working file. More than once
> a changed introduction has destroyed years of work and it has only been
> my paranoia in having about 6 backups of everything that has saved the
> project.

...

You could make a bash script that daily or weekly rsyncs copies of local
and external hard disks onto one hard disk and burn a bluray backup when
needed.

Changing the local HDD with 3 others is useful if the blurays going
missing or corrupt too.

Then you could use any standard common ext2/3/4 file system.

Such backups scripts commonly evolve from something simple as a start.

Russell Shaw

unread,

Aug 27, 2020, 12:48:38 AM8/27/20

to 'stripes theotoky' via mlug-au

On 26/8/20 8:43 pm, 'stripes theotoky' via mlug-au wrote:
> Hello Mark,
>
> Question, do you really need to recover files from the snapshots on a
> routine basis?
>
>
> No I don't. I have never recovered anything from ZFS and hope to never
> have to. ZFS is a last ditch line of defense against some of my
> co-authors who have a terrible habit of working on old files, saving
> them out with the same name as the current working file. More than once
> a changed introduction has destroyed years of work and it has only been
> my paranoia in having about 6 backups of everything that has saved the
> project.

...

You could make a bash script that daily or weekly rsyncs copies of local
and external hard disks onto one hard disk and burn a bluray backup when
needed.

Changing the local HDD with 3 others is useful if the blurays going
missing or corrupt too.

Then you could use any standard common ext2/3/4 file system.

Such backups scripts commonly evolve from something simple as a start.

A total pain if the users are on Window$ tho.

stripes theotoky

unread,

Aug 29, 2020, 9:03:10 AM8/29/20

to mlu...@googlegroups.com

Hi,

You could make a bash script that daily or weekly rsyncs copies of local
and external hard disks onto one hard disk and burn a bluray backup when
needed.

That is a good idea how do we make one that sticks a number in the file name and increments it so I have all versions of the paper backed up? So if the paper was called Environmental Regulation.tex the copies would be called something like

Environmental Regulaton-00001.tex, Environmental Regulaton-00002.tex, Environmental Regulaton-00003.tex, etc or possibly replacing the numbers with the time and date.

Then you could use any standard common ext2/3/4 file system.

Yes I am thinking very seriously of moving to ext4.

Such backups scripts commonly evolve from something simple as a start.

A total pain if the users are on Window$ tho.

I am in Australia on Ubuntu 16.04

Co-authors are in England on Windows 7

China on Mac OS 10.7 (I think)

Files are often shared by e-mail as China's firewall can play hell with Dropbox at times and sometimes due to tiredness from working across time zones files are saved out over current working files destroying years of work if not fully backed up. So far my back ups have always saved the day but at times at the expense of my husband's peace of mind spending about a week with diff swearing quietly about academics.

I think the suggestion of a back up script is good and when one of co-authors particularly is working it could usefully be taking 5 minutely backups. We are only talking of tex files of about 150kb here so even with that frequency of back up you wouldn't fill a 1TB disk in a lifetime.

Stripes.

--
You received this message because you are subscribed to the Google Groups "mlug-au" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlug-au+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mlug-au/c26ce483-2da2-3810-9a60-b42092007c96%40netspace.net.au.

Russell Shaw

unread,

Sep 2, 2020, 12:03:49 AM9/2/20

to 'stripes theotoky' via mlug-au

On 29/8/20 11:02 pm, 'stripes theotoky' via mlug-au wrote:
> Hi,
>
> You could make a bash script that daily or weekly rsyncs copies of local
> and external hard disks onto one hard disk and burn a bluray backup when
> needed.
>
> That is a good idea how do we make one that sticks a number in the file name and
> increments it so I have all versions of the paper backed up? So if the paper was
> called Environmental Regulation.tex the copies would be called something like
> Environmental Regulaton-00001.tex, Environmental Regulaton-00002.tex,
> Environmental Regulaton-00003.tex, etc or possibly replacing the numbers with
> the time and date.

In a shell command line or a script:

NAME="Environmental Regulaton"

echo $NAME-`date +%F_%R`.tex

gives: Environmental Regulaton-2020-09-02_13:54.tex

> Then you could use any standard common ext2/3/4 file system.
>
>
> Yes I am thinking very seriously of moving to ext4.
>
> Such backups scripts commonly evolve from something simple as a start.
>
> A total pain if the users are on Window$ tho.
>
>
> I am in Australia on Ubuntu 16.04
> Co-authors are in England on Windows 7
> China on Mac OS 10.7 (I think)
> Files are often shared by e-mail as China's firewall can play hell with Dropbox
> at times and sometimes due to tiredness from working across time zones files are
> saved out over current working files destroying years of work if not fully
> backed up. So far my back ups have always saved the day but at times at the
> expense of my husband's peace of mind spending about a week with diff swearing
> quietly about academics.
>
> I think the suggestion of a back up script is good and when one of co-authors
> particularly is working it could usefully be taking 5 minutely backups. We are
> only talking of tex files of about 150kb here so even with that frequency of
> back up you wouldn't fill a 1TB disk in a lifetime.

Getting the remote users to setup and check scripts are running properly would
be difficult for computer illiterate users.

Ideally your pc would use something like SSH in a script to get copies from the
remote PCs, which would have an SSH server installed with your public SSH key
stored on their PCs.

Getting this stuff to work from windows boxes is another layer of difficulty.

They'll just have to keep making local backups themselves manually.
--
regards,
Russell

Russell Shaw

unread,

Sep 2, 2020, 12:06:16 AM9/2/20

to 'stripes theotoky' via mlug-au

scp is the SSH copy command

Russell Shaw

unread,

Sep 2, 2020, 12:09:14 AM9/2/20

to 'stripes theotoky' via mlug-au

On 29/8/20 11:02 pm, 'stripes theotoky' via mlug-au wrote:
> Hi,
>
> You could make a bash script that daily or weekly rsyncs copies of local
> and external hard disks onto one hard disk and burn a bluray backup when
> needed.
>
> That is a good idea how do we make one that sticks a number in the file name and
> increments it so I have all versions of the paper backed up? So if the paper was
> called Environmental Regulation.tex the copies would be called something like
> Environmental Regulaton-00001.tex, Environmental Regulaton-00002.tex,
> Environmental Regulaton-00003.tex, etc or possibly replacing the numbers with
> the time and date.

In a shell command line or a script:

NAME="Environmental Regulaton"

echo $NAME-`date +%F_%R`.tex

gives: Environmental Regulaton-2020-09-02_13:54.tex

> Then you could use any standard common ext2/3/4 file system.
>
>
> Yes I am thinking very seriously of moving to ext4.
>
> Such backups scripts commonly evolve from something simple as a start.
>
> A total pain if the users are on Window$ tho.
>
>
> I am in Australia on Ubuntu 16.04
> Co-authors are in England on Windows 7
> China on Mac OS 10.7 (I think)
> Files are often shared by e-mail as China's firewall can play hell with Dropbox
> at times and sometimes due to tiredness from working across time zones files are
> saved out over current working files destroying years of work if not fully
> backed up. So far my back ups have always saved the day but at times at the
> expense of my husband's peace of mind spending about a week with diff swearing
> quietly about academics.
>
> I think the suggestion of a back up script is good and when one of co-authors
> particularly is working it could usefully be taking 5 minutely backups. We are
> only talking of tex files of about 150kb here so even with that frequency of
> back up you wouldn't fill a 1TB disk in a lifetime.

Getting the remote users to setup and check scripts are running properly would
be difficult for computer illiterate users.

Ideally your pc would use something like SSH in a script to get copies from the
remote PCs, which would have an SSH server installed with your public SSH key
stored on their PCs.

Getting this stuff to work from windows boxes is another layer of difficulty.

They'll just have to keep making local backups themselves manually.

rsync also can copy from remote systems if an rsync daemon is installed on the
remote system, but i haven't used that.
--
regards,
Russell

Malcolm Herbert

unread,

Sep 2, 2020, 3:12:55 AM9/2/20

to mlug-au

Slightly off-topic but I think you have Dropbox in the mix as well - can I suggest using git?

For my choir I have a shared Dropbox folder with the other members of the choir. Reasonably regularly I will use git's ability to check in the current state that is present in my copy of the shared Dropbox folder into a repository which is outside the shared folder ... doing this means I have a tool I can use to manage versioning of text and binary files (.pdf music scores in our case) and because the repo is outside the shared folder, it also doesn't count against their Dropbox quota.

None of the other members in the choir need to know how to drive it, nor do they need to install any extra software onto their host - the only requirement is that the Dropbox folder be in sync which isn't always guaranteed, but at least the client is well documented and works fine for windows and linux (on ext4 filesystems)

Being a git repo, you can then restore any file offline or revert to a previous commit at will ... you can also push that somewhere else or do whatever you prefer to keep that backed up on a remote host.

stripes theotoky

unread,

Sep 4, 2020, 2:00:41 AM9/4/20

to mlu...@googlegroups.com

Dear Group,

sorry for taking so long to respond your answers are very welcome I have been sick but back now and will soon have this worked out.

NAME AVAIL USED
alexandria 1.06G 332G

The free space looks pretty low if much buffering is needed.

Think more free space is needed.

Yes now I feel alive again I will be deleting old snapshots especialls some from 2019 that shouldn't be there at all.

Stripes.

--
You received this message because you are subscribed to the Google Groups "mlug-au" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlug-au+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mlug-au/8bcd2b83-a63c-3cc5-26c7-d2d05dbadf4e%40netspace.net.au.

--

Stripes Theotoky

-37 .713869
145.050562

stripes theotoky

unread,

Sep 4, 2020, 2:50:21 AM9/4/20

to mlu...@googlegroups.com

In a shell command line or a script:

NAME="Environmental Regulaton"

echo $NAME-`date +%F_%R`.tex

gives: Environmental Regulaton-2020-09-02_13:54.tex

Brilliant I can work with that and have it back up all tex files in the folder with the current date and time. It can work on Dropbox so any changes to the files as saved and can be recovered.

Getting the remote users to setup and check scripts are running properly would
be difficult for computer illiterate users.

That shouldn't be a problem as I can run the script locally via cron on my Dropbox and get all the changes that they make. They don't even have to know about it let alone set up anything.

Ideally your pc would use something like SSH in a script to get copies from the
remote PCs, which would have an SSH server installed with your public SSH key
stored on their PCs.

Getting this stuff to work from windows boxes is another layer of difficulty.

That is too hard just back up the tex files in the local Dropbox folder.

This will even work for the one in China as when he e-mails me new versions I save them out in Dropbox so he has access when Dropbox is working in China or when he is travelling so even if I accidentally lose work because he has sent the wrong copy of a file this script will have saved it.

I will start playing with this very soon.

Thank you.

Stripes.

They'll just have to keep making local backups themselves manually.
--
regards,
Russell

--
You received this message because you are subscribed to the Google Groups "mlug-au" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlug-au+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mlug-au/0c4a9df6-490b-83a0-6e16-0cf00510b239%40netspace.net.au.

Russell Shaw

unread,

Sep 4, 2020, 3:16:06 AM9/4/20

to 'stripes theotoky' via mlug-au

On 4/9/20 4:50 pm, 'stripes theotoky' via mlug-au wrote:
>
...

> I will start playing with this very soon.
>
> Thank you.

Welcome ;)

I've never looked at dropbox.

There must be other online cloud things that would work too.

Even a gitlab or github repository.

Reply all

Reply to author

Forward