dm-verity snapshot restore

[Bogdan Chifor]

Hello,

I am using cuttlefish crosvm and trying to do a restore from snapshot. Sometimes, I can see that device is being rebooted after successful restore from snapshot with the following dm-verity error:

[ 2723.241203] libprocessgroup: Still waiting on process(es) to exit for cgroup /sys/fs/cgroup/uid_1020/pid_606 after 0 ms^M
[ 2723.246324] libprocessgroup: Removed cgroup /sys/fs/cgroup/uid_1020/pid_606^M
[ 2723.247391] binder: undelivered death notification, 000070afa0f2b110^M
[ 2723.247770] binder: send failed reply for transaction 100142 to 744:911^M
[ 2723.249311] init: Sending signal 15 to service 'vendor.confirmationui_default' (pid 572) process group...^M
[ 2723.255775] ACPI: PM: Preparing to enter system sleep state S5^M
[ 2723.256553] kvm: exiting hardware virtualization^M
[ 2723.256801] reboot: Restarting system with command 'dm-verity device corrupted'^M
[ 2723.257067] reboot: machine restart^M

or

[ 861.316061] device-mapper: verity-fec: 254:0: FEC: recursion too deep [ 861.316322] device-mapper: verity: 254:0: metadata block 172300 is corrupted [ 861.348338] ACPI: PM: Preparing to enter system sleep state S5 [ 861.349098] kvm: exiting hardware virtualization [ 861.349285] reboot: Restarting system with command 'dm-verity device corrupted' [ 861.349498] reboot: machine restart

Do you happen to experience anything similar?

Best regards,

Bogdan Chifor.

[Bogdan Chifor]

Yes, using cvd suspend and then cvd snapshot_take. In most cases the snapshot can be resumed.

I don't have any overlays (starting the cvd with -use_overlay=false), so all the dm-verity partitions should be read-only, unless I am missing something. Between snapshot take and restore, disk files are not changed.

On Fri, Jul 25, 2025 at 1:00 AM Frederick Mayle <fma...@google.com> wrote:

Can you share how you are performing the snapshot and restore steps? Are you using cuttlefish's snapshot_util_cvd command?

My first guess for a dm-verity error is that the underlying disk image contents changed between when the snapshot and restore happened. The disk image contents aren't part of the snapshot (except for cuttlefish's copy-on-write overlays). Cuttlefish does some basic checks to detect if the disk image changed, but might not catch everything.

--
You received this message because you are subscribed to the Google Groups "crosvm-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crosvm-dev+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/crosvm-dev/8aabae76-0a57-4745-85ba-5d5c386c7d7an%40chromium.org.

[Frederick Mayle]

Can you share how you are performing the snapshot and restore steps? Are you using cuttlefish's snapshot_util_cvd command?

My first guess for a dm-verity error is that the underlying disk image contents changed between when the snapshot and restore happened. The disk image contents aren't part of the snapshot (except for cuttlefish's copy-on-write overlays). Cuttlefish does some basic checks to detect if the disk image changed, but might not catch everything.

On Thu, Jul 24, 2025 at 10:32 AM Bogdan Chifor <chiforb...@gmail.com> wrote:

--

[Frederick Mayle]

I'm not sure about the status of `cvd suspend/snapshot_take`. I know `snapshot_util_cvd --subcmd=snapshot_take --auto_suspend --snapshot_path=$SOME_PATH` should work.

> I don't have any overlays (starting the cvd with -use_overlay=false), so all the dm-verity partitions should be read-only

I'm not familiar with how CF behaves when overlays are off. My vague memory is that it would just modify the underlying disk images and we never tested how it interacts with snapshotting.

dm-verity is readonly in general IIUC, so I think you are right they shouldn't be modified unless done so outside the VM. OTOH, other disk images, like those used for the /data/ partition, will almost certainly be modified if you let the vCPU run after snapshotting, or restore the snapshot multiple times. Our workaround for that was to snapshot the overlays.

[tej s]

Hi Frederick,

We are trying to capture cuttlefish snapshot for arm64. We are able to successfully take the snapshot after disabling vhost-user-vsock. But after restoring we are getting  a reset from openwrt and cuttlefish vm.

So far we have followed the below steps to take the snapshot -

1. HOME=$PWD ./bin/cvd start --enable_virtiofs=false --gpu_mode=guest_swiftshader --vhost_user_vsock=false --daemon
2. Do some action , connect Wi-Fi 
3. cvd snapshot_take --force --auto_suspend --snapshot_path="/home/ubuntu/snap"
4. HOME=$PWD ./bin/stop-cvd
5. cvd reset
6. HOME=$PWD ./bin/cvd start --snapshot_path="/home/ubuntu/snap" --daemon  

But facing the below error - 

crosvm I 12-24 07:20:56 272412 272412 log_tee.cpp:164] [2025-12-24T07:20:56.056732013+00:00 INFO vm_control] Starting crosvm resume crosvm I 12-24 07:20:56 272412 272412 log_tee.cpp:164] [2025-12-24T07:20:56.056790990+00:00 INFO vm_control] Finished crosvm resume successfully [2025-12-24T07:20:56.056864383+00:00 INFO crosvm] exiting with success proxy_adb I 12-24 07:20:56 272344 272344 socket_vsock_proxy.cpp:183] restoring proxy on CUTTLEFISH_HOST - success proxy_adb I 12-24 07:20:56 272344 272344 socket_vsock_proxy.cpp:149] From: tcp: 6522 proxy_adb I 12-24 07:20:56 272344 272344 socket_vsock_proxy.cpp:150] To: vsock: 5:5555 vhost_user: false proxy_adb D 12-24 07:20:56 272344 272344 socket_vsock_proxy.cpp:188] Start reading events to start/stop proxying 12-24 07:20:56.307 272358 272598 I openwrt_control_server: http_client.cc:265 Attempting to download "http://192.168.94.10/devices/cvd_3-3/openwrt/cgi-bin/luci/rpc/auth" * Trying 192.168.94.10:80... adb_connector D 12-24 07:20:58 272343 272351 adb_connection_maintainer.cpp:229] adb connected to 0.0.0.0:6522 webRTC E 12-24 07:20:58 272340 272697 vsock_connection.cpp:222] Failed to connect:Operation timed out * connect to 192.168.94.10 port 80 from 192.168.94.9 port 33696 failed: Host is unreachable * Failed to connect to 192.168.94.10 port 80 after 3088 ms: Could not connect to server * closing connection #0 run_cvd F 12-24 07:20:59 272316 272317 boot_state_machine.cc:360] Check failed: status.ok() Failed to send network service reset14: Luci authentication request failed: run_cvd F 12-24 07:20:59 272316 272317 boot_state_machine.cc:360] run_cvd F 12-24 07:20:59 272316 272317 boot_state_machine.cc:360] 3. main.cpp:165 | UpdateLuciRpcAuthKey | run_cvd F 12-24 07:20:59 272316 272317 boot_state_machine.cc:360] 2. http_client.cc:228 | DownloadToJson | run_cvd F 12-24 07:20:59 272316 272317 boot_state_machine.cc:360] 1. http_client.cc:254 | DownloadToString | run_cvd F 12-24 07:20:59 272316 272317 boot_state_machine.cc:360] | device/google/cuttlefish/host/libs/web/http_client/http_client.cc:298 run_cvd F 12-24 07:20:59 272316 272317 boot_state_machine.cc:360] | Result<HttpResponse<void>> cuttlefish::(anonymous namespace)::CurlClient::DownloadToCallback(HttpMethod, DataCallback, const std::string &, const std::vector<std::string> &, const std::string &) run_cvd F 12-24 07:20:59 272316 272317 boot_state_machine.cc:360] v CF_EXPECT(res == CURLE_OK) run_cvd F 12-24 07:20:59 272316 272317 boot_state_machine.cc:360] curl_easy_perform() failed. Code was "7". Strerror was "Could not connect to server". Error buffer was "Failed to connect to 192.168.94.10 port 80 after 3088 ms: Could not connect to server". run_cvd E 12-24 07:20:59 272308 272308 boot_state_machine.cc:147] Failed to read a complete exit code, read 0 bytes only instead of the expected 4

Note - same steps are working for x86_64 (both suspending and resume of snapshot)

So please share your valuable inputs and the fixes required to restore the snapshot for arm64.

Regards,

Tejbir