Bug#998627: linux: please enable the new NTFS3 driver in 5.15

[Wenbin Lv]

Package: src:linux
Version: 5.15-1~exp1
Severity: wishlist

Hi,
Paragon's NTFS3 driver has been merged to 5.15, and it offers much
better performance compared to ntfs-3g. Currently it is not enabled in
Debian's kernel config, nor can I mount NTFS partitions using '-t
ntfs3'.
Please enable it if you consider it stable enough. Thank you!

[Boyuan Yang]

Hi,

Now that 5.15 has been uploaded to Unstable, this issue will attract wider
attention. Please consider enabling it in future uploads.

Thanks!
Boyuan Yang

[Vincent Blut]

Hi,

I'll send a merge request to the kernel team later today.

> Thanks!
> Boyuan Yang

Have a good day,
Vincent

[Salvatore Bonaccorso]

Hi,

Are tools available to handle creation and checking of such NTFS3
filesystems? The last time I went to the paragon software site it
mentioned it was planning. This is not a must, but kept me for
slightly on the on hold position for enabling it.

Regards,
Salvatore

[Vincent Blut]

Le 2021-11-23 22:34, Salvatore Bonaccorso a écrit :
> Hi,
>
> On Tue, Nov 23, 2021 at 12:37:02PM +0100, Vincent Blut wrote:
> > Hi,
> >
> > Le 2021-11-22 18:54, Boyuan Yang a écrit :
> > > Hi,
> > >
> > > On Fri, 5 Nov 2021 16:55:33 +0800 Wenbin Lv <wenb...@gmail.com> wrote:
> > > > Package: src:linux
> > > > Version: 5.15-1~exp1
> > > > Severity: wishlist
> > > >
> > > > Hi,
> > > > Paragon's NTFS3 driver has been merged to 5.15, and it offers much
> > > > better performance compared to ntfs-3g. Currently it is not enabled in
> > > > Debian's kernel config, nor can I mount NTFS partitions using '-t
> > > > ntfs3'.
> > > > Please enable it if you consider it stable enough. Thank you!
> > >
> > > Now that 5.15 has been uploaded to Unstable, this issue will attract wider
> > > attention. Please consider enabling it in future uploads.
> >
> > I'll send a merge request to the kernel team later today.
>
> Are tools available to handle creation and checking of such NTFS3
> filesystems?

Not yet AFAIK.

> The last time I went to the paragon software site it mentioned it was
> planning. This is not a must, but kept me for slightly on the on hold
> position for enabling it.

Understandable! Let's hold off for the moment then.

> Regards,
> Salvatore

Cheers,
Vincent

[Wenbin Lv]

Hi,

On Wed, Nov 24, 2021 at 5:36 AM Salvatore Bonaccorso <car...@debian.org> wrote:
>
> Hi,

>
>
> Are tools available to handle creation and checking of such NTFS3
> filesystems? The last time I went to the paragon software site it
> mentioned it was planning. This is not a must, but kept me for
> slightly on the on hold position for enabling it.
>
> Regards,
> Salvatore
>

Paragon only mentioned they are planning to release mkfs.ntfs in their
FAQ, not the fsck tool[1]. So we'll need ntfs-3g or Windows for fsck
anyway if they don't release it. Stability and further code
maintenance, however, are matters of greater concern as Ted Ts'o
pointed out[2]. Maybe we should hold it back for some time to see what
will happen.

Best regards,
Wenbin Lv

[1] https://www.paragon-software.com/home/ntfs3-driver-faq/
[2] https://lore.kernel.org/lkml/YQnHxIU+...@mit.edu/

[Salvatore Bonaccorso]

Hi,

On Wed, Nov 24, 2021 at 03:14:56PM +0800, Wenbin Lv wrote:
> Hi,
>
> On Wed, Nov 24, 2021 at 5:36 AM Salvatore Bonaccorso <car...@debian.org> wrote:
> >
> > Hi,
> >
> >
> > Are tools available to handle creation and checking of such NTFS3
> > filesystems? The last time I went to the paragon software site it
> > mentioned it was planning. This is not a must, but kept me for
> > slightly on the on hold position for enabling it.
> >
> > Regards,
> > Salvatore
> >
>
> Paragon only mentioned they are planning to release mkfs.ntfs in their
> FAQ, not the fsck tool[1]. So we'll need ntfs-3g or Windows for fsck
> anyway if they don't release it. Stability and further code
> maintenance, however, are matters of greater concern as Ted Ts'o
> pointed out[2]. Maybe we should hold it back for some time to see what
> will happen.

Yes right, the missing tools was not an argument against it, just an
additional indication. As you pointed out is more concerning, and
hinting that we should wait a bit more to see how it evolves.

Apart from Ted Ts'o comments, I looked up the mailinglist,
https://lore.kernel.org/ntfs3/ which relatively quiet. The same
goes for commits in fs/ntfs3 in mainline, since the merge in 5.15
there ws almost no activity, which is slightly unusual for something
new entering.

Regards and thank you!
Salvatore

[Heinz Repp]

On Tue, 23 Nov 2021 22:34:25 +0100 Salvatore Bonaccorso
<car...@debian.org> wrote:
> Are tools available to handle creation and checking of such NTFS3
> filesystems? The last time I went to the paragon software site it
> mentioned it was planning. This is not a must, but kept me for
> slightly on the on hold position for enabling it.

You mean you are holding back the DRIVER because you are missing
USERSPACE utilities to handle and examine NTFS 3.1 filesystems?!?

One has nothing to do with the other. There are a lot of NTFS utilities
available with the ntfs-3g package, namely:

> mkntfs
> ntfscat
> ntfsclone
> ntfscluster
> ntfscmp
> ntfscp
> ntfsdecrypt
> ntfsfallocate
> ntfsfix
> ntfsinfo
> ntfslabel
> ntfsls
> ntfsrecover
> ntfsresize
> ntfssecaudit
> ntfstruncate
> ntfsundelete
> ntfswipe

Nearly all of them (apart from ntfssecaudit cases) work on devices, not
on mounted file systems, so the NTFS driver used is not in involved. An
idea would be to split the ntfs-3g package into
ntfs-3g-tools
ntfs-3g-fusedriver
libntfs-3g (on which the former depends)

A Paragon donated mkfs.ntfs would be a nice thing, but mkntfs from
ntfs-3g already fits for nearly all purposes. And I guess most people
would want the new ntfs3 driver for filesystems created with Windows.

On Sat, 27 Nov 2021 14:14:53 +0100 Salvatore Bonaccorso
<car...@debian.org> wrote:
> Apart from Ted Ts'o comments, I looked up the mailinglist,
> https://lore.kernel.org/ntfs3/ which relatively quiet. The same
> goes for commits in fs/ntfs3 in mainline, since the merge in 5.15
> there ws almost no activity, which is slightly unusual for something
> new entering.

Looking on the same mailinglist, I think they recently did a lot to fix
the test issues and much more, as can be seen:

> [PATCH] ntfs3: Fix showing umask option
> 2021-11-27 13:51 UTC
>
> [PATCH 1/2] fs/ntfs3: clarify why $Extend init is being skipped
> 2021-11-26 12:04 UTC (2+ messages)
>
> [PATCH] Doc/fs/ntfs3: Fix a trivial typo
> 2021-11-26 11:52 UTC (2+ messages)
>
> [ramfs] 0858d7da8a: canonical_address#:#[##]
> 2021-11-26 11:36 UTC (2+ messages)
>
> [PATCH] fs/ntfs3: Fix some memory leaks in an error handling path of 'log_replay()'
> 2021-11-23 15:47 UTC (2+ messages)
>
> Bug using new ntfs3 file system driver (5.15.2 on Arch Linux)
> 2021-11-19 14:19 UTC (3+ messages)
>
> No more accessible (compromised) NTFS volume
> 2021-11-16 17:16 UTC
>
> [PATCH v2 0/4] fs/ntfs3: Various fixes for xattr and files
> 2021-11-15 15:50 UTC (11+ messages)
> ` [PATCH 1/4] fs/ntfs3: Keep preallocated only if option prealloc enabled
> ` [PATCH 2/4] fs/ntfs3: Restore ntfs_xattr_get_acl and ntfs_xattr_set_acl functions
> ` [PATCH 3/4] fs/ntfs3: Update i_ctime when xattr is added
> ` [PATCH 4/4] fs/ntfs3: Optimize locking in ntfs_save_wsl_perm
>
> [BUG] Unable to mount NTFS drive, no error in dmesg
> 2021-11-11 4:39 UTC
>
> Readahead for compressed data
> 2021-10-29 6:15 UTC (16+ messages)
>
> [PATCH v2] checkpatch: Improve CVS revision marker check
> 2021-10-27 10:56 UTC
>
> [PATCH] checkpatch: Remove cvs keyword check
> 2021-10-27 9:21 UTC (4+ messages)
>
> [PATCH 0/4] fs/ntfs3: Various fixes for xfstests problems
> 2021-10-26 21:37 UTC (11+ messages)
> ` [PATCH 1/4] fs/ntfs3: In function ntfs_set_acl_ex do not change inode->i_mode if called from function ntfs_init_acl
> ` [PATCH 2/4] fs/ntfs3: Fix fiemap + fix shrink file size (to remove preallocated space)
> ` [PATCH 3/4] fs/ntfs3: Check new size for limits
> ` [PATCH 4/4] fs/ntfs3: Update valid size if -EIOCBQUEUED
>
> [PATCH] fs/ntfs3: clarify emitted log message when marking volumes as dirty
> 2021-10-26 20:56 UTC (3+ messages)
>
> [PATCH 2/2] fs/ntfs3: clarify emitted log message when marking volumes as dirty
> 2021-10-26 20:55 UTC
>
> [PATCH] fs/ntfs3: clarify why $Extend init is being skipped
> 2021-10-26 20:49 UTC (2+ messages)
>
> (very low importance) NTFS version message
> 2021-10-26 18:03 UTC (2+ messages)
>
> [PATCH 0/4] fs/ntfs3: Various fixes for xattr and files
> 2021-10-24 11:13 UTC (12+ messages)
> ` [PATCH 1/4] fs/ntfs3: Keep preallocated only if option prealloc enabled
> ` [PATCH 2/4] fs/ntfs3: Restore ntfs_xattr_get_acl and ntfs_xattr_set_acl functions
> ` [PATCH 3/4] fs/ntfs3: Optimize locking in ntfs_save_wsl_perm
> ` [PATCH 4/4] fs/ntfs3: Update i_ctime when xattr is added
>
> NTFS3: junctions are not properly resolved
> 2021-10-22 12:50 UTC
>
> [PATCH 0/2] Removing of old code
> 2021-10-19 17:11 UTC (4+ messages)
> ` [PATCH 1/2] fs/ntfs3: Remove unnecessary functions
>
> cleanup block device inode syncing
> 2021-10-19 13:54 UTC (14+ messages)
> ` [PATCH 1/7] fs: remove __sync_filesystem
> ` [PATCH 2/7] block: remove __sync_blockdev
> ` [PATCH 3/7] xen-blkback: use sync_blockdev
> ` [PATCH 4/7] btrfs: "
> ` [PATCH 5/7] fat: use sync_blockdev_nowait
> ` [PATCH 6/7] ntfs3: "
> ` [PATCH 7/7] block: simplify the block device syncing code
>
> don't use ->bd_inode to access the block device size v3
> 2021-10-19 1:04 UTC (40+ messages)
> ` [PATCH 01/30] block: move the SECTOR_SIZE related definitions to blk_types.h
> ` [PATCH 02/30] block: add a bdev_nr_bytes helper
> ` [PATCH 03/30] bcache: remove bdev_sectors
> ` [PATCH 04/30] drbd: use bdev_nr_sectors instead of open coding it
> ` [PATCH 05/30] dm: use bdev_nr_sectors and bdev_nr_bytes instead of open coding them
> ` [PATCH 06/30] md: use bdev_nr_sectors instead of open coding it
> ` [PATCH 07/30] nvmet: use bdev_nr_bytes "
> ` [PATCH 08/30] target/iblock: "
> ` [PATCH 09/30] fs: use bdev_nr_bytes instead of open coding it in blkdev_max_block
> ` [PATCH 10/30] fs: simplify init_page_buffers
> ` [PATCH 11/30] affs: use bdev_nr_sectors instead of open coding it
> ` [PATCH 12/30] btrfs: use bdev_nr_bytes "
> ` [PATCH 13/30] cramfs: "
> ` [PATCH 14/30] fat: use bdev_nr_sectors "
> ` [PATCH 15/30] hfs: "
> ` [PATCH 16/30] hfsplus: "
> ` [PATCH 17/30] jfs: use bdev_nr_bytes "
> ` [PATCH 18/30] nfs/blocklayout: "
> ` [PATCH 19/30] nilfs2: "
> ` [PATCH 20/30] ntfs3: "
> ` [PATCH 21/30] pstore/blk: "
> ` [PATCH 22/30] reiserfs: "
> ` [PATCH 23/30] squashfs: "
> ` [PATCH 24/30] block: use bdev_nr_bytes instead of open coding it in blkdev_fallocate
> ` [PATCH 25/30] block: add a sb_bdev_nr_blocks helper
> ` [PATCH 26/30] ext4: use sb_bdev_nr_blocks
> ` [PATCH 27/30] jfs: "
> ` [PATCH 28/30] ntfs: "
> ` [PATCH 29/30] reiserfs: "
> ` [PATCH 30/30] udf: "
>
> don't use ->bd_inode to access the block device size v2
> 2021-10-18 6:22 UTC (63+ messages)
> ` [PATCH 01/30] block: move the SECTOR_SIZE related definitions to blk_types.h
> ` [PATCH 02/30] block: add a bdev_nr_bytes helper
> ` [PATCH 03/30] bcache: remove bdev_sectors
> ` [PATCH 04/30] drbd: use bdev_nr_sectors instead of open coding it
> ` [PATCH 05/30] dm: use bdev_nr_sectors and bdev_nr_bytes instead of open coding them
> ` [PATCH 06/30] md: use bdev_nr_sectors instead of open coding it
> ` [PATCH 07/30] nvmet: use bdev_nr_bytes "
> ` [PATCH 08/30] target/iblock: "
> ` [PATCH 09/30] fs: use bdev_nr_bytes instead of open coding it in blkdev_max_block
> ` [PATCH 10/30] fs: simplify init_page_buffers
> ` [PATCH 11/30] affs: use bdev_nr_sectors instead of open coding it
> ` [PATCH 12/30] btrfs: use bdev_nr_bytes "
> ` [PATCH 13/30] cramfs: "
> ` [PATCH 14/30] fat: use bdev_nr_sectors "
> ` [PATCH 15/30] hfs: "
> ` [PATCH 16/30] hfsplus: "
> ` [PATCH 17/30] jfs: use bdev_nr_bytes "
> ` [PATCH 18/30] nfs/blocklayout: "
> ` [PATCH 19/30] nilfs2: "
> ` [PATCH 20/30] ntfs3: "
> ` [PATCH 21/30] pstore/blk: "
> ` [PATCH 22/30] reiserfs: "
> ` [PATCH 23/30] squashfs: "
> ` [PATCH 24/30] block: use bdev_nr_bytes instead of open coding it in blkdev_fallocate
> ` [PATCH 25/30] block: add a sb_bdev_nr_blocks helper
> ` [PATCH 26/30] ext4: use sb_bdev_nr_blocks
> ` [PATCH 27/30] jfs: "
> ` [PATCH 28/30] ntfs: "
> ` [PATCH 29/30] reiserfs: "
> ` [PATCH 30/30] udf: "
>
> [PATCH 0/5] fs/ntfs3: Reworking symlink functions
> 2021-10-17 8:10 UTC (3+ messages)
> ` [PATCH 1/5] fs/ntfs3: Rework ntfs_utf16_to_nls

Would be interesting to rerun Ted Ts'o's fstests on the patched driver -
Ted's results are from early August!

Greetings

Heinz

[chucrut]

On Mon, 29 Nov 2021 15:59:41 +0100 Heinz Repp <heinz...@arcor.de>
wrote:

Hello,
ntfs-3g and ntfsprogs was merged to ntfs-3g. I think it would be better
if utilities package get it own old name.

[Ben Hutchings]

On Fri, 2022-03-18 at 20:58 +0100, lenni_na1 wrote:
> Hi,
>
> are there any news on this?
>
> We are now at kernel 5.16 in testing and as far as I can tell the ntfs3
> driver still hasn't been enabled.

The recent traffic on the ntfs3 list seems to consist of bug reports
and small fixes, none of them being addressed by the supposed
maintainer of the filesystem (who last posted at the end of November).

I think that we would be doing our users a disservice by enabling ntfs3
in this state.

Ben.

--
Ben Hutchings
If the facts do not conform to your theory, they must be disposed of.

[Marc Lehmann]

Hi!

I was just stumblinmg over this bugreport, and must say I am surprised
at the logic here - the ntfs3 module does not conflict with existing
filesystem drivers (such as ntfs-3g), so existing systems shouldn't be
negatively affected as they would continue to either fail to mount or use
ntfs-3g, if installed.

also, the logic of keeping it out is very flawed, namely that the feature is
new. but the code in question existed for a long time outside the kernel, so
arguing there might possibly, potentially, maybe be some problems in the code
(that haven't surfaced in the half a year since the release) is very
strange.

while I appreciate that the kernel package maintainers try to keep
dangerous or broken modules disabled, I think doing so based on zero
actual evidence is going to far, especially given that other broken
modules are enabled (e.g. ufs, which does get used by default and can eat
ufs filesystems for breakfast).

the only actual evidence so far is that there is no working fsck - well,
the same argument could be used to keep btrfs out of debian.

I would have understood this decision if the ntfs3 module conflicted with the
existing ntfs or ntfs-3g drivers, but it doesn't.

please reconsider your decision - thanks!

[Salvatore Bonaccorso]

Hi,

On Sun, Mar 20, 2022 at 10:25:23PM +0100, Ben Hutchings wrote:
> On Fri, 2022-03-18 at 20:58 +0100, lenni_na1 wrote:
> > Hi,
> >
> > are there any news on this?
> >
> > We are now at kernel 5.16 in testing and as far as I can tell the ntfs3
> > driver still hasn't been enabled.
>
> The recent traffic on the ntfs3 list seems to consist of bug reports
> and small fixes, none of them being addressed by the supposed
> maintainer of the filesystem (who last posted at the end of November).
>
> I think that we would be doing our users a disservice by enabling ntfs3
> in this state.

In meanwhile the current state of NTFS3 driver has been discussed
upstream starting in
https://lore.kernel.org/lkml/da20d32b-5185-f40b...@stargateuniverse.net/
.

Regards,
Salvatore

[Mina Morcose Farage]

hello

sid and teasting made for this purpose i  suggest to enable it in sid at
least and see where it goes


Thanks

Mina

[Heinz Repp]

On Fri, 29 Apr 2022 22:27:27 +0200 Salvatore Bonaccorso
<car...@debian.org> wrote:
> In meanwhile the current state of NTFS3 driver has been discussed
> upstream starting in
> https://lore.kernel.org/lkml/da20d32b-5185-f40b...@stargateuniverse.net/

You're well aware that this discussion ENDED IN early _MAY 2022_ ?!? If
your arguments repeatedly cite ancient statements their validity is at
least questionable.

In the meanwhile we are on Linux 6.0.12, and NTFS3 is still not enabled.
Just to make it clear: enabling NTFS3 will NOT change any existing usage
of ntfs3-ng, it just creates a new possibility to increase the
performance of using ntfs3 partitions. I am glad that on my production
server I have Arch Linux, which enabled NTFS3 in their kernel with
5.15.2, and it ist running performant and rock stable. I would
appreciate to be able to use this driver also on my Debian Testing
system, but am still waiting in vain for many months to see this come true.

Jm2c

Heinz

[Norbert Lange]

It's been ages, why isn't this enabled by now? How should this driver mature when no one can test it (without going through the hassle if compiling the Kernel).

[Salvatore Bonaccorso]

Hi Norbert,

Thanks for asking back. NTFS3 driver is still not in astate making it
confident it's wise to enable it for a stable release.

https://lore.kernel.org/ntfs3/784f82c4-de71-b8c3...@paragon-software.com/T/#me2324a967514564949f7ebcf3f9a5965f66921bf
is an example (it took from august to now, until the fix landed in
mainline).

I think thus the arguments from https://bugs.debian.org/998627#75
still holds.

Regards,
Salvatore

[Norbert Lange]

Hello Salvatore,

AFAIK kernels receive a ton of patches in distros,

Backported and what not.

Is it not possible to atleast build the module but either blacklist it or offer it in a separate package?

If it's in a sorry state, then that would direct the complaints to the module, not to the configuration.

Thanks for the quick response.

Norbert 

[BW]

Is NTFS3 still not available, not even in Debian 12? After so long?

Please don't be patronizing and think you now better than the users and decide what they need and don't.
This is Linux, users are capable of deciding if this is useable for them or not.

I don't think any will use NTFS as a production file-system on Linux! So don't use any excuse of missing utils or it is not stable. NTFS-3G is not stable and useable as production FS either.
NTFS3 will for the huge majority of users be used in a utility-context, e.g. accessing/grapping files from a disk from a MS box etc.
Please make it available

[Salvatore Bonaccorso]

Hi,

No it is not available even in Debian 12; Unless paragon-software show
to be active more on their subsystem. Or let's say "maybe for trixie
it will be made available at some point". With a highlight on "maybe".

There is not enough confidence that it is well maintained upstream.
Additionally a stream of CVEs would have affected bookworm if we would
have enabled it. So, while not written in stone, it was the right
decision to not enable it for Debian 12.

https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=ntfs3 (gives a rough
overview)

And the story around fs/ntfs* and Debian requiring to mark it as
broken (cf.
https://salsa.debian.org/kernel-team/linux/commit/becaca2c809a9c8eeb45287d3ecc5c572cc49733)
should not be needed to be repeated.

Regards,
Salvatore

[so54p1+4s6...@cs.email]

Package: linux
Followup-For: Bug #998627

Dear Maintainer,

Please consider enabling the NTFS3 driver for debian-testing, so that it
receives more test coverage before debian 13 (trixie) is released as
stable. Thank you.

-- System Information:
Debian Release: 12.0
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 6.1.0-9-amd64 (SMP w/1 CPU thread; PREEMPT)
Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

[an xiao]

Hello,

Paragon-software is active enough on their subsystem,

and NTFS3 is extremely stable now.

Please consider enabling the NTFS3 driver for sid. 

Thinks and best regards,

Littlewhite