migrating from sdcardfs to esdfs

[Huy Minh Bùi]

Recently with the deprecation of sdcardfs and the fact the maintaining the file-system is pretty time-consuming especially when it comes to maintaining it alongside the Google's common source (or Google LTS) , I am trying to find another way to fix the apps can't read data & obb problem. By looking at what ChromeOS and Project Celadon did, I realized that they are reusing an old file-system called "esdfs" which dated before sdcardfs and being used to mount on top of the data file-system, as an example from some devices dmesg

[ 32.441432,2] ESDFS-fs (esdfs): mounted on top of /data/media type f2fs

[ 32.441497,2] SELinux: initialized (dev esdfs, type esdfs), uses genfs_contexts

[ 32.441793,2] ESDFS-fs (esdfs): mounted on top of /data/media type f2fs

[ 32.441819,2] SELinux: initialized (dev esdfs, type esdfs), uses genfs_contexts

[ 32.442040,2] ESDFS-fs (esdfs): mounted on top of /data/media type f2fs

[ 32.442063,2] SELinux: initialized (dev esdfs, type esdfs), uses genfs_contexts

It turns out bring back esdfs is not a difficult problem, you just need to go to ChromiumOS's third-party kernel source to find, Google did everything so we just need to cherry-pick it

For example, here is the kernel 5.15 source

https://chromium.googlesource.com/chromiumos/third_party/kernel/+/1704af1484c6d2ad8e4452ecedc02acf5b764136
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/2d49999db99ebd1ccd1be9e170defd24f9133823

2 commits for the driver, and a commit for PKGLIST, esdfs require this driver

https://chromium.googlesource.com/chromiumos/third_party/kernel/+/e5b3b81d1d68b255b8315890a22dcf5d9cf488c6

I compiled the modules with a kernel of mine that is based on Google's common kernel + some Android-x86 patches. It does compile and the file-system does showed up when I check /proc/filesystems (also logcat showed it got "registered"). But there is no sign of Android picking it up and use so far. I know that Chih-Wei back then have a commit on system/core that basically revert esdfs support and bring back FUSE, I revert it and it esdfs still not picking up. 

https://github.com/supremegamers/android_system_core/commit/9d9d62a29b1bb2f54761aac11062632f513930f8

Even when set ro.sys.sdcardfs to false like this commit told nothing happened. So I wonder if we can find a way to use it so that we can see if it can helped us with the apps permission issue. I am building r-x86, more specifically a Lineage-based ROM which I ported to x86

[youli...@gmail.com]

mount sdcard as fuse, android_x86:/sdcard # time busybox dd if=/dev/zero of=testw.dbf bs=1M count=1000

1000+0 records in

1000+0 records out

1048576000 bytes (1000.0MB) copied, 1.513464 seconds, 660.7MB/s

    0m01.53s real 0m00.00s user 0m00.53s system

android_x86:/sdcard #

mount sdcard as sdcardfs, android_x86:/sdcard # time busybox dd if=/dev/zero of=testw.dbf bs=1M count=1000

1000+0 records in

1000+0 records out

1048576000 bytes (1000.0MB) copied, 0.721600 seconds, 1.4GB/s

    0m00.74s real 0m00.00s user 0m00.64s system

android_x86:/sdcard #

[youli...@gmail.com]

how esdfs work under chromeos userspace?

https://chromium-review.googlesource.com/q/project:chromiumos/platform2+esdfs

[Huy Minh Bùi]

Can you try esdfs mount and see the speed ? FUSE is being known for its slow speed so I want to see how esdfs perform

Vào lúc 19:49:56 UTC+7 ngày Chủ Nhật, 10 tháng 4, 2022, youli...@gmail.com đã viết:

[Huy Minh Bùi]

I am guessing that ChromeOS use an external tool called "arc-setup" to handle the bind mount of the emulated storage

I found this post in 2020 written in Chinese

https://www.cxymm.net/article/b178903294/115395826

Vào lúc 19:59:40 UTC+7 ngày Chủ Nhật, 10 tháng 4, 2022, youli...@gmail.com đã viết:

[Huy Minh Bùi]

On r-x86, I found this commit to mount the ext4/f2fs storage to sdcard, I tried but currently it's only bind mount in FUSE, also they didn't mount Android/data and Android/obb

https://review.lineageos.org/c/LineageOS/android_system_vold/+/286951

Vào lúc 19:49:56 UTC+7 ngày Chủ Nhật, 10 tháng 4, 2022, youli...@gmail.com đã viết:

[youli...@gmail.com]

How to make fuse passthrough work? need kernel and userspace libfuse support passthrough.

https://android-review.googlesource.com/q/project:platform%252Fexternal%252Flibfuse+status:open

DO NOT MERGE FUSE passthrough: handle unstable kernel interface

DO NOT MERGE Enable passthrough mode for read/write operations

 

but the two libfuse patch not merge, so how fuse passthrough work on android12? https://source.android.com/devices/storage/fuse-passthrough

[youli...@gmail.com]

android kernel and android MediaProvider FuseDaemon support passthrough, but libfuse not support passthrough, still can't improve the reading and writing speed. 

[Huy Minh Bùi]

I don't think we should use libfuse anymore and focusing on trying esdfs

Vào lúc 20:56:51 UTC+7 ngày Chủ Nhật, 10 tháng 4, 2022, youli...@gmail.com đã viết:

[youli...@gmail.com]

I start used ext4 casefold begin with linux 5.2, https://www.reddit.com/r/linux_gaming/comments/cbrc7s/52_kernel_supports_caseinsensitive_directories/

Android sdcard mount as fuse or sdcardfs, in order to case insensitive, but now ext4 support case insensitive, why not chattr +F data/media/0, so android init.rc "exec - media_rw media_rw -- /system/bin/chattr +F /data/media.

but how to use ext4 casefold on android, i notice external_storage.casefold.enabled=1.

[Huy Minh Bùi]

yes that's the prop on higher Android like 11. device/generic/common already add it in

Here's the vold commit

https://android.googlesource.com/platform/system/vold/+/477f3e5f4afbf23618ae248f6fc65b3693a70a14%5E%21/#F0

I wonder if it actually works on Android-x86.

Vào lúc 23:50:15 UTC+7 ngày Thứ Năm, 28 tháng 4, 2022, youli...@gmail.com đã viết:

[youli...@gmail.com]

mkfs.ext4 -O project,quota,casefold /dev/sda3, grub.cfg, DATA=/dev/block/sda3

will try "encoding=utf8 quotatype=prjquota".

[youli...@gmail.com]

I wonder it can bind mount /data/media to /storage/emulated, but it isn't.

[youli...@gmail.com]

android support esdfs a long ago, https://android.googlesource.com/platform/system/core/+/c336f86284ee817a2e846452ad80e5d014d11414

disable fuse will use esdfs.

revert androidx86 fuse.cpp, rebuild system/bin/sdcard binary.

disable fuse, add these to default.prop.

persist.sys.fuse=false

persist.fuse_sdcard=false

persist.sys.fuse.default_fuse_enabled=false

persist.sys.fflag.override.settings_fuse=false

persist.device_config.storage_native_boot.fuse_enabled=false

[youli...@gmail.com]

persist.sys.fuse=false

persist.fuse_sdcard=false

persist.sys.fuse.default_fuse_enabled=false

persist.sys.fflag.override.settings_fuse=false

persist.device_config.storage_native_boot.fuse_enabled=false

ro.sys.sdcardfs=false

persist.sys.sdcardfs=force_off

04-29 21:01:15.075 0 0 E ESDFS-fs (esdfs): unrecognized mount option "unshared_obb" or missing value

04-29 21:01:15.075 0 0 I ESDFS-fs (esdfs): mounted on top of /data/media type ext4

[youli...@gmail.com]

system.prop, 

external_storage.projid.enabled=0

external_storage.casefold.enabled=0

external_storage.sdcardfs.enabled=1

default.prop,

[Huy Minh Bùi]

From the picture seems like the mounting is good. However if you force FUSE to off like this can you still be able to use ntfs-3g ?

Vào lúc 20:23:10 UTC+7 ngày Thứ Sáu, 29 tháng 4, 2022, youli...@gmail.com đã viết:

[Huy Minh Bùi]

But it seems like Android/data and Android/obb still haven't got mounted yet

Vào lúc 20:33:29 UTC+7 ngày Thứ Sáu, 29 tháng 4, 2022, Huy Minh Bùi đã viết:

[youli...@gmail.com]

on my android11 phone, dev/fuse mount on /storage/emulated,  data/media on /storage/emulated/Android/data type sdcardfs.

if storage/emulated type sdcardfs, Android/data and obb needn't mounted.

I interest with androidx86 11 how to storage/emulated type fuse, Android/data type sdcardfs.

esdfs and sdcardfs based wrapfs.

ntfs-3g too old, i will remove userspace ntfs-3g, how to make system/bin/sdcard use linux kernel ntfs3 mount ntfs.

remove userspace ntfs-3g and exfat, use linux kernel ntfs3 and exfat.

[Huy Minh Bùi]

> on my android11 phone, dev/fuse mount on /storage/emulated,  data/media on /storage/emulated/Android/data type sdcardfs.

> if storage/emulated type sdcardfs, Android/data and obb needn't mounted.

> I interest with androidx86 11 how to storage/emulated type fuse, Android/data type sdcardfs.

My phone does the same as you, storage/emulated in /dev/fuse and data & obb mount as sdcardfs.

And I hope we can do the same as phones but with esdfs

> ntfs-3g too old, i will remove userspace ntfs-3g, how to make system/bin/sdcard use linux kernel ntfs3 mount ntfs.

I am still planning to use ntfs-3g because although ntfs3 looks good, there are some function of the filesystem it isn't properly implemented yet. So I hope that the drvier can be improved soon.

As for how to make sdcard mount it, I assume that you can try to edit file.te like this and mount ? Also only compile NTFS3, do not compile the older NTFS in the kernel

https://github.com/supremegamers/device_generic_common/commit/9af84a5f39898bb2cfc6c101c30c7afb656226c8

Vào lúc 20:49:24 UTC+7 ngày Thứ Sáu, 29 tháng 4, 2022, youli...@gmail.com đã viết:

[Huy Minh Bùi]

also I don't know about exfat, if the kernel driver is good enough then we can move on from the fuse one

Vào lúc 21:26:56 UTC+7 ngày Thứ Sáu, 29 tháng 4, 2022, Huy Minh Bùi đã viết:

[youli...@gmail.com]

vold support use kernel ntfs driver a long ago, vold: Native NTFS kernel driver support, but only for Android.mk, https://github.com/LineageOS/android_system_vold/commit/9087457933f336b9d6057725472ec8ab2f223fe2

ifeq ($(TARGET_KERNEL_HAVE_NTFS),true)

  vold_cflags += -DCONFIG_KERNEL_HAVE_NTFS

endif

https://github.com/LineageOS/android_system_vold/tree/lineage-19.1/fs doesn't has ntfs.cpp

http://git.osdn.net/view?p=android-x86/system-vold.git;a=commit;h=846a83a1adc98f6cbcde161a2bde23028e0011b6  androidx86 porting ntfs support for android.bp, not porting "Native NTFS kernel driver support".

[youli...@gmail.com]

http://git.osdn.net/view?p=android-x86/system-vold.git;a=commit;h=6126528f5e2cd375b3bdf63f9c11e88bfb9a3a94

http://git.osdn.net/view?p=android-x86/system-vold.git;a=commit;h=9087457933f336b9d6057725472ec8ab2f223fe2

porting them to r-x86 android.bp.

[youli...@gmail.com]

vold support mount ntfs3, test on androidx86 7.

[ 208.614117] sdcardfs version 2.0

[ 208.614122] sdcardfs: dev_name -> /mnt/media_rw/5F4BBAE56AE257B0

[ 208.614124] sdcardfs: gid=1015,mask=6

[ 208.614140] sdcardfs: mounted on top of /mnt/media_rw/5F4BBAE56AE257B0 type ntfs3

[youli...@gmail.com]

mount | grep fuse, mount | grep esd

在2022年4月29日星期五 UTC+8 22:26:56<buin...@gmail.com> 写道：

> on my android11 phone, dev/fuse mount on /storage/emulated,  data/media on /storage/emulated/Android/data type sdcardfs.

> if storage/emulated type sdcardfs, Android/data and obb needn't mounted.

> I interest with androidx86 11 how to storage/emulated type fuse, Android/data type sdcardfs.

My phone does the same as you, storage/emulated in /dev/fuse and data & obb mount as sdcardfs.

And I hope we can do the same as phones but with esdfs

[youli...@gmail.com]

system.prop, 

external_storage.projid.enabled=0

external_storage.casefold.enabled=0

external_storage.sdcardfs.enabled=1

default.prop, only add these, doesn't disable fuse

ro.sys.sdcardfs=false

persist.sys.sdcardfs=force_off

[youli...@gmail.com]

system.prop, 

external_storage.projid.enabled=0

external_storage.casefold.enabled=0

external_storage.sdcardfs.enabled=1

default.prop, only add these, switch to sdcardfs.

ro.sys.sdcardfs=true

persist.sys.sdcardfs=force_on

[youli...@gmail.com]

vold: support mount ntfs3 for r-x86, test on androidx86 11, i don't know how to use TARGET_KERNEL_HAVE_NTFS in Android.bp.

[youli...@gmail.com]

oh, a mistake, my write "system.prop" is system/build.prop, 

external_storage.projid.enabled, external_storage.casefold.enabled, external_storage.sdcardfs.enabled, they are in system/build.prop

[Huy Minh Bùi]

Hi, I put the three properties into build.prop and 2 (ro.sys.sdcardfs=false and persist.sys.sdcardfs=force_off) properties to default.prop but my system still got no esdfs bind-mounted

Here are my system/core and system/vold

https://github.com/supremegamers/android_system_core

https://github.com/supremegamers/platform_system_vold

Please tell me in case something is missing

Also, one thing to mention is that I am currently using a Magisk solution that is patched directly into initrd.img

https://github.com/supremegamers/bootable_newinstaller/pull/2 (this is for building the initrd.img)

https://github.com/HuskyDG/initrd-magisk (this is for using a seperate initrd to load Magisk)

I wonder if Magisk affect the bind-mounting process.

Vào lúc 03:04:29 UTC+7 ngày Thứ Bảy, 30 tháng 4, 2022, youli...@gmail.com đã viết:

[youli...@gmail.com]

CONFIG_PKGLIST=y

CONFIG_PKGLIST_USE_CONFIGFS=y

[Huy Minh Bùi]

It still not working

Vào lúc 01:37:59 UTC+7 ngày Chủ Nhật, 1 tháng 5, 2022, youli...@gmail.com đã viết:

[Huy Minh Bùi]

Maybe it only detect when you have both sdcardfs and esdfs ? My kernel doesn't have sdcardfs

Vào lúc 10:01:26 UTC+7 ngày Thứ Ba, 3 tháng 5, 2022, Huy Minh Bùi đã viết:

[youli...@gmail.com]

check casefold storage properties right, getprop | grep storage.

what is emulated_storage.mk doing? it only add three properties to build.prop? http://git.osdn.net/view?p=android-x86/device-generic-common.git;a=commit;h=4ee8a8fafca7b131b59d72d8afba3e16421f852b  https://android.googlesource.com/platform/build/+/master/target/product/emulated_storage.mk

[Huy Minh Bùi]

Yes

https://android.googlesource.com/platform/build/+/master/target/product/emulated_storage.mk

Vào lúc 00:27:38 UTC+7 ngày Thứ Tư, 4 tháng 5, 2022, youli...@gmail.com đã viết:

[youli...@gmail.com]

always use esdfs.

persist.sys.sdcardfs force_on or off, always use esdfs.

/data/media on /mnt/runtime/default/emulated type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:1015:771:771,derive=multi,noconfine,derive_gid,default_normal,unshared_obb)

/data/media on /storage/emulated type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:1015:771:771,derive=multi,noconfine,derive_gid,default_normal,unshared_obb)

/data/media on /mnt/runtime/read/emulated type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:9997:750:750,derive=multi,noconfine,derive_gid,default_normal,unshared_obb)

/data/media on /mnt/runtime/write/emulated type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:9997:770:770,derive=multi,noconfine,derive_gid,default_normal,unshared_obb)

/data/media on /mnt/runtime/full/emulated type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:9997:770:770,derive=multi,noconfine,derive_gid,default_normal,unshared_obb)

/dev/block/vold/public:8,2 on /mnt/media_rw/1551-23CF type vfat (rw,dirsync,nosuid,nodev,noexec,noatime,gid=1023,fmask=0007,dmask=0007,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)

/mnt/media_rw/1551-23CF on /mnt/runtime/default/1551-23CF type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:1015:771:771,derive=public,noconfine)

/mnt/media_rw/1551-23CF on /storage/1551-23CF type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:1015:771:771,derive=public,noconfine)

/mnt/media_rw/1551-23CF on /mnt/runtime/read/1551-23CF type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:9997:755:755,derive=public,noconfine)

/mnt/media_rw/1551-23CF on /mnt/runtime/write/1551-23CF type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:9997:755:755,derive=public,noconfine)

/mnt/media_rw/1551-23CF on /mnt/runtime/full/1551-23CF type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:9997:770:770,derive=public,noconfine)

/dev/block/vold/public:8,3 on /mnt/media_rw/24C97AC14E629DF4 type ntfs3 (rw,nosuid,nodev,noexec,noatime,uid=0,gid=1023,fmask=37777600007,dmask=37777600007,iocharset=utf8)

/mnt/media_rw/24C97AC14E629DF4 on /mnt/runtime/default/24C97AC14E629DF4 type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:1015:771:771,derive=public,noconfine)

/mnt/media_rw/24C97AC14E629DF4 on /storage/24C97AC14E629DF4 type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:1015:771:771,derive=public,noconfine)

/mnt/media_rw/24C97AC14E629DF4 on /mnt/runtime/read/24C97AC14E629DF4 type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:9997:755:755,derive=public,noconfine)

/mnt/media_rw/24C97AC14E629DF4 on /mnt/runtime/write/24C97AC14E629DF4 type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:9997:755:755,derive=public,noconfine)

/mnt/media_rw/24C97AC14E629DF4 on /mnt/runtime/full/24C97AC14E629DF4 type esdfs (rw,nosuid,nodev,noexec,noatime,lower=1023:1023:664:775,upper=0:9997:770:770,derive=public,noconfine)

:/ # getprop | grep sdcard

[external_storage.sdcardfs.enabled]: [1]

[persist.fuse_sdcard]: [false]

[persist.sys.sdcardfs]: [force_on]

[ro.sys.sdcardfs]: [true]

:/ # getprop | grep sdcardfs

[external_storage.sdcardfs.enabled]: [1]

[persist.sys.sdcardfs]: [force_on]

[ro.sys.sdcardfs]: [true]

:/ #

[Huy Minh Bùi]

It still not working, I am thinking that might be because of the Magisk I used

can you help me to try ?

https://github.com/supremegamers/bootable_newinstaller/pull/2

Vào lúc 23:01:45 UTC+7 ngày Thứ Bảy, 7 tháng 5, 2022, youli...@gmail.com đã viết: