Bug#871619: zfs-dkms: Please package ZFS 0.7.0

[Jason Cohen]

Package: zfs-dkms
Version: 0.6.5.9-5
Severity: wishlist

Dear Maintainer,

Please consider packaging ZFS 0.7.1. The new 0.7.0 release, which released on
July 26, 2017, includes a number of valuable new features including better
memory management (ARC uses scatter lists rather than virtual memory,
minimizing fragmentation and Compressed ARC), improved performance (vectorized
RAID-Z and fletcher4 math, faster resilvering, improved metadata performance),
resumable and compressed 'zfs send' (invaluable for very large sends!), and
improved statistics (SMART data, IO latency, average request size).

Thanks for packing ZFS for Debian,

Jason



-- System Information:
Debian Release: 9.1
APT prefers stable
APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8), LANGUAGE=en_US.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages zfs-dkms depends on:
ii debconf [debconf-2.0] 1.5.61
ii dkms 2.3-2
ii lsb-release 9.20161125
ii spl-dkms 0.6.5.9-1

Versions of packages zfs-dkms recommends:
ii zfs-zed 0.6.5.9-5
ii zfsutils-linux 0.6.5.9-5

zfs-dkms suggests no packages.

-- debconf information:
* zfs-dkms/note-incompatible-licenses:
zfs-dkms/stop-build-for-unknown-kernel: true
zfs-dkms/stop-build-for-32bit-kernel: true

[Fabian Grünbichler]

I started some (unofficial, not-ready-for-production-use-yet) branches
for SPL and ZFS 0.7.0, available on github in case anyone wants to take
a look:

https://github.com/Fabian-Gruenbichler/zfs/tree/debian/wip-0.7
https://github.com/Fabian-Gruenbichler/spl/tree/debian/wip-0.7

no pre-built packages yes, but hopefully we can get some 0.7 goodness
into unstable soon.

[Jason Wittlin-Cohen]

For those too inpatient to wait for an official Debian build, I managed to compile kmod packages from the ZoL source.  See here for details: https://github.com/zfsonlinux/zfs/issues/6606

[Fabian Grünbichler]

On Tue, Sep 19, 2017 at 09:35:10AM -0400, Chris Zubrzycki wrote:
> Any update or possibly a mailing list for the group we can subscribe to? ubuntu has had 0.7 for a while now…
>
> Thanks.
>
>

Ubuntu is (also) still on 0.6.5.11[1]. you can subscribe to the
pkg-zfsonlinux-devel list[2], but there is not much going on over there.


1: https://packages.ubuntu.com/source/artful/zfs-linux
2: http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-zfsonlinux-devel

[D4rk4]

---------- Forwarded message ----------
From: Petter Reinholdtsen <pe...@debian.org>
Date: 2017-08-10 8:33 GMT+02:00
Subject: Re: No ZFS 0.7 in repos
To: D4rk4 <h31...@gmail.com>, ar...@debian.org, clo...@igalia.com


[D4rk4]
> Please buld
>
> https://qa.debian.org/cgi-bin/watch?pkg=zfs-linux

Hi.  Very nice to see more people interested in ZFS in Debian.  The ZFS
team is heavily understaffed.  Perhaps you want to help?  If so, please
join the mailinglist and #debian-zfs IRC channel.

The request for a updated ZFS version is available in
<URL: https://bugs.debian.org/871619 >. Please follow up there if you
have anything to add.

PS: Please use the team mailing list instead of sending email to
  individual team members.  This allow the entire team to contribute.
--
Happy hacking
Petter Reinholdtsen

--

С уважением,
Дмитрий.

[Holger Schröder]

zfs-dkms and spl-dkms 0.6.5.11 not build with new Kernel 4.13.4, please
update to 0.7.x


Thanks

    Holger...

[Antonio Russo]

Hi!

I forked the Alioth repository, and got the debian packaging able
to build 0.7.2 (the spl ./debian required no changes).

The fork is available on github:

https://github.com/aerusso/pkg-zfsonlinux-zfs/tree/proposed-changes/debian

I dropped a patch that seemed to diverge from upstream's support for powerpc,
mostly because I have no ability to test that platform (and the code around
that seemed to have quite a bit of churn). Also several patches were applied
upstream, so they also got removed.

How can I help out the packaging team on this? Who should I email?

Thanks,
Antonio Russo

[Fabian Grünbichler]

note: I am not the maintainer or even a DM in general..

you could have probably saved yourself a lot of effort if you had read
the full history of this bug, and noticed I already have a working 0.7.2
branch for SPL and ZFS up on my github ;)

I am planning to do an NMU for 0.6.5.11 with compat patches for 4.13 if
I find a sponsor (this weekend), and then a bigger NMU for 0.7.2 (in
which I will introduce a new binary package zfs-test in accordance with
upstream's packaging, which means a trip through NEW) soon after.

the zfsonlinux maintainers for Debian are pretty much inactive
currently, so if you want to help out you are of course more than
welcome.

[Dmitry Galenko]

I successful build deb packages from upstream repo, for kernel 4.10 (custom) without any changes with this instruction:

apt-get install -y build-essential autoconf libtool gawk alien fakeroot linux-headers-$(uname -r) zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev libdevmapper-dev parted lsscsi ksh

export ZFS_BUILD_PATH=/tmp/zfs
rm -rf $ZFS_BUILD_PATH
mkdir $ZFS_BUILD_PATH

cd $ZFS_BUILD_PATH && rm -rf spl && git clone https://github.com/zfsonlinux/spl && cd spl && git checkout remotes/origin/spl-0.7-release && sh autogen.sh && ./configure && make -s -j2 && make deb

cd $ZFS_BUILD_PATH && rm -rf zfs && git clone https://github.com/zfsonlinux/zfs && cd zfs && git checkout remotes/origin/zfs-0.7-release && sh autogen.sh && ./configure && make -s -j2 && make deb

# Delete the many ZFS kernel modules non-optionally installed by Ubuntu and instead install ours:
cd $ZFS_BUILD_PATH && dpkg -i spl/*.deb zfs/*.deb
rm -rf /lib/modules/*/kernel/zfs
depmod -a

_______________________________________________
Pkg-zfsonlinux-devel mailing list
Pkg-zfsonlinux-devel@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-zfsonlinux-devel

--

С уважением,
Дмитрий.

[Petter Reinholdtsen]

[Antonio Russo]

> How can I help out the packaging team on this? Who should I email?

The best way to help is to continue contributing and participating on
the team mailing list and IRC channel.

Aron Xu has been most active with maintenance, and I hope he can comment
on how you best can contribute. I am not competent do to so. :)

Any project admin on
<URL: https://alioth.debian.org/projects/pkg-zfsonlinux/ > can grant
access, ie Aron, Liang and Carlos.

CC to Aron, and email to the bts also go to the project mailing list.

[Aron Xu]

Hi,

I've started to update the package to 0.7.2 release and it appears
there are some build system changes that installs quite some files
into non-FHS compliant paths, and possibly a new package (could be
"zfs-tests") is needed. If you are eager to help, please start from
these changes and my latest work is available on alioth git.

Regards,
Aron

[Fabian Grünbichler]

On Fri, 6 Oct 2017 08:49:51 +0200 Dmitry Galenko <dmi...@galenko.su> wrote:
> I successful build deb packages from upstream repo, for kernel 4.10
> (custom) without any changes with this instruction:
>

> [snip]

this is not a good idea for production use IMHO - those converted packages using alien lack all of the integration into Debian proper (including missing all the systemd service stuff, among other things)..

[Fabian Grünbichler]

> Aron Xu <ar...@debian.org> hat am 6. Oktober 2017 um 18:19 geschrieben:

did you see my WIP branch on github, linked earlier here (and also separately on pkg-zfsonl...@lists.alioth.debian.org)[1,2]? I already did most of the work (including splitting out zfs-test) except for updating debian/copyright.. note that I merged the actual upstream tags and not the upstream release tar balls, so you probably only want to take a closer look at commits touching debian/ ;) I have been running this branch (recently with the addition of upstream PR#6616[3]) for quite some time on my personal machines (Sid and Stretch), and have been testing a variant of it with a 4.13 kernel at work as well - all without any problems so far (except for the send/receive incompatibility with 0.6.5, for which I made the PR ;))

the only other point up for discussion besides splitting out zfs-test was whether the zfs and zpool binaries should move to /bin instead of /sbin, as they are now usable by unprivileged users for certain read-only operations..

I would be glad to assist in any future work here, both for the 0.7.2 release as well as further packaging (I am responsible for most of the downstream ZFS packaging in Proxmox VE, which is based on Debian - so I am familiar with both ZoL and Debian packaging ;)).

if you have any questions about my branch or want to discuss stuff, just drop me an e-mail (here, on-list or directly) or catch me on IRC (f_g on OFTC and freenode)

1: https://github.com/Fabian-Gruenbichler/zfs/commits/debian/wip-0.7
2: https://lists.alioth.debian.org/pipermail/pkg-zfsonlinux-devel/2017-August/001196.html
3: https://github.com/zfsonlinux/zfs/pull/6616

[Antonio Russo]

On 10/12/17 06:12, Fabian Grünbichler wrote:
>
> I would also suggest cherry-picking/backporting the following two
> upstream PRs on top of 0.7.2:
>
> #6616: send/recv compatibility with 0.6.5.x [1]
> #6695: receive_freeobjects() skips freeing some objects [2]
>
> the first (DISCLAIMER: written by me :P) fixes an incompatibility
> between zfs send on 0.7 and zfs recv on 0.6.5 which would otherwise lead
> to lots of never-terminating, 100% cpu usage zfs recv threads on
> not-yet-upgraded replication/backup targets.
>
> the second fixes a potential inconsistency when receiving incremental
> streams, which would cause subsequent receives to fail.
>
> both are slated for inclusion in 0.7.3 ([3]).
>
> 1: https://github.com/zfsonlinux/zfs/pull/6616
> 2: https://github.com/zfsonlinux/zfs/pull/6695
> 3: https://github.com/zfsonlinux/zfs/projects/12
>

Just as a follow up, all of these proposed stable additions are
currently under review for inclusion into the stable branch by
upstream [1]. I've included them tentatively in the
repositories [2,3].

1: https://github.com/zfsonlinux/zfs/pull/6752
2: https://github.com/aerusso/pkg-zfsonlinux-zfs
3: https://github.com/aerusso/pkg-zfsonlinux-spl

[Fabian Grünbichler]

CC-ing bug/list again, IMHO this should be discussed publicly.

On Wed, Oct 11, 2017 at 06:41:54PM -0400, Antonio Russo wrote:
> On 10/10/17 17:43, Antonio Russo wrote:
>
> > I'll check the FHS concerns, time-permitting.
>
> With Fabian's changes, the only things that maybe looked questionable
> were the files installed to
>
> /usr/lib/${ARCH}/zfs
>
> which seem to be architecture independent shellscript. Is this what
> you are talking about? Should these files instead be located in
>
> /usr/lib/zfsutils-linux
>
> Antonio
>

either that (see attached patch, I reused /usr/lib/zfs-linux which
already contains the scrub script referenced in the cron job), or move
them to /usr/share/zfs like the zfs-test scripts? IMHO both places are
fine, /usr/lib/ARCH/zfs is definitely wrong ;)

[Fabian Grünbichler]

[Antonio Russo]

The two repositories

github.com/aerusso/pkg-zfsonlinux-spl
github.com/aerusso/pkg-zfsonlinux-zfs

include patches from the stable branches of upstream, as well
as Fabian's zfs-test packaging, which was suggested was important
for a release. I also believe this addresses the issues regarding
FHS non-compliance, esp under /usr/lib. I'm currently using packages
built from these changes, without issue. Given that 0.6.11 isn't
supported on the current testing kernel, it would probably be nice
for everyone if 0.7.2 was packaged.

Aron, please let me know what actions would need to be taken
to get my branch pulled into the alioth git, and what other
blockers there are for a debian 0.7.2 release.

Thanks,
Antonio Russo

[Holger Schröder]

Hi

I use 3 Patches from Ubuntu to build 0.6.5.11 on Kernel 4.13.... build
and works :)

spl:

0001-Update-struct-member-intializers-to-C89.patch
0002-Linux-4.13-compat-wait-queues.patch


zfs:

2001-Linux-4.13-compat-bio-bi_status-and-blk_status_t.patch

Greetings

        Holger...

[Fabian Grünbichler]

and there has now been a 0.7.3 release incorporating those and other
fixes:

https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.3

[Antonio Russo]

The two github repositories

https://github.com/aerusso/pkg-zfsonlinux-zfs
https://github.com/aerusso/pkg-zfsonlinux-spl

have upstream branches representing an import of the zfs/spl
sources, and a "proposed-updates" branch which includes now
a changelog entry. It builds, installs, and runs on a few
machines, and (as best as I can tell) matches the style
of the alioth repository. Can the changes in these repositories
just be pulled into alioth? Is there anything I can do to
help?

Antonio

[Aron Xu]

I've pulled the changes, but still some minor issues need to be fixed,
I'm working on that right now.

Regards,
Aron