As far as BLFS is concerned, the installation seems to have gone without issue, and I have successfully built a kernel that matches the hardware required for XEN and a working lightdm X11 environment.
Where I'm running into the issue is that qvm-run from dom0 does not seem to have any effect on the HVM itself. Here are the basic tests for functionality I have confirmed:
SERVICES RUNNING:
qubes-db.service loaded active running
qubes-early-vm-config.service loaded active exited
qubes-gui-agent.service loaded active running
qubes-meminfo-writer.service loaded active running
qubes-mount-dirs.service loaded active exited
qubes-qrexec-agent.service loaded active running
qubes-sysinit.service loaded active exited
meminfo-writer is checked in the gui
MOUNTS:
xen on /proc/xen type xenfs (rw,relatime)
/dev/xvdb on /rw type ext4 (rw,relatime,discard)
/dev/xvdb on /home type ext4 (rw,relatime,discard)
/dev/xvdb on /usr/local type ext4 (rw,relatime,discard)
#/dev/xvdc I somehow messed up, but I'm guessing isn't relevant to qvm-run
MODULES:
# lsmod
Module Size Used by
tmem 16384 -2
u2mfn 16384 -2
All the other modules, e.g., evtchn are built-in; this is a compact kernel with what I felt were the very bare minimum, but ultimately will be replaced with the PVH kernel. I attempted to use qubes-linux-kernel, but had issues with the dependency on rpms.
TESTS FOR OPERABILITY:
dmesg https://pastebin.com/1WwLZFKL
guest-lfsgo.log https://pastebin.com/fnhw3Tc7
guest-lfsgo-dm.log https://pastebin.com/bhhL76pQ
qrexec.lfsgo.log : is empty
guid.lfsgo.log : contains one line "Icon size: 128x128"
[will@dom0 ~] qrexec-client -d lfsgo lfs:bash
[will@dom0 ~] qrexec-client -d lfsgo user:bash
[will@dom0 ~] qvm-run lfsgo "touch /tmp/dummy"
Running 'touch /tmp/dummy' on lfsgo
[will@dom0 ~] qvm-run --pass-io lfsgo whoami
[will@dom0 ~]
(domU)
root [ ~ ]# uname -a
Linux lfsgo 4.18.9 #4 SMP Fri Oct 5 08:56:34 MST 2018 x86_64 GNU/Linux
root [ ~ ]# su - user
user [ ~ ]$ sudo whoami #passwordless sudo working
root
user [ ~ ]$ exit
logout
root [ ~ ]# xenstore-read name
lfsgo
`ls -la /tmp` shows no sign of '/tmp/dummy', same as with `sudo touch /tmp/dummy`
Are there other logs I can provide to give a clearer picture of my install?
Are there any more-verbose way I can check why qrexec isn't working as expected?
root [ ~ ]# ls -lah /dev/xen/privcmd
crw-rw---- 1 root qubes 10, 58 Oct 24 2018 /dev/xen/privcmd
The line "read(4," is really where the trace ends--stays there indefinitely. No change to this behavior even if qubes-qrexec-agent.service is stopped/running.
I've also then rebuilt qubes-vmm-xen, qubes-linux-utils, and qubes-core-agent-linux for good measure, with the same strace results.
Perfect! It seems like that put me on the right track and a few new leads have emerged.
So stracing qrexec-agent led me to qrexec-fork-server:
> connect(8, {sa_family=AF_FILE, path="/var/run/qubes/qrexec-server.user.sock"}, 40) = -1 ENOENT (No such file or directory)
Running `/usr/bin/qrexec-fork-server` as `user` fails with "bind() failed". I `chmod g+w /var/run/qubes`'ed and it let me run the process.
ls -la /var/run/qubes
...
srwxrwxr-x 1 user user 0 Oct 25 10:14 qrexec-server.user.sock
--
`qvm-run lfsgo 'touch /tmp/hi'` strace output from dom0: https://pastebin.com/5FrnAhUg
`journalctl -u qubes-qrexec-agent` output: https://pastebin.com/nKxLxidx
--
Feels like I'm doing some really hacky, non-systematic approaches to get things going, so I know I must've missed a few things.
1) how does qrexec-fork-server typically init'ed?
2) qvm-run still doesn't run the command, but attempting it does push this to "user"'s terminal: "executed QUBESRPC qubes.WaitForSession dom0 pid 679" and the dom0 side process stalls indefinitely.
control-C killing the process pushes "pid 679 exited with -1" on domU's side.
Looking at qubes.WaitForSession, I'm now suspicious that this is now an X issue--maybe surrounding the X session owner--is this a possibility?
--
Other details
# ls -la /tmp
prw-rw-r-- 1 user user 0 Oct 25 10:37 qrexec-rpc-stderr-return.679
prw-rw-r-- 1 user user 0 Oct 25 10:37 qrexec-rpc-stderr.679
-rw-rw-r-- 1 user user 4 Oct 25 10:37 qubes-session-waiter
None of the rpc files have any content in them.
Any guidance would be greatly appreciated!
Will
Tweaking pam.d/qrexec allowed me to ultimately execute commands from dom0. It appears they ONLY work via --no-gui calls, as anything without that flag stalls/hangs indefinitely.
So in the meantime, it looks like *most* of this is solved; I just have to finally get a grasp of self-installed X that can work to provide an X11 session for qubes.WaitForSession.
It looks like my barebones pam.d configuration was stifling the qubes-gui-agent process. I went ahead and made the /etc/pam.d/qubes-gui-agent more permissive and it seems I'm making huge strides toward it working. The newest issue is now coming up in Dom0(!).
Here's my ~/.xsession-errors: https://pastebin.com/Cf0mfGz3
Xorg.0.log looks pretty clean, start to finish, but here's the paste anyway: https://pastebin.com/BEj9tHm0
----
There's an early error, which I believe to be dbus. Not sure if this is a huge obstacle, but I notice that `dbus-launch --autolaunch` is already running before the system's dbus-daemon:
642 user 20 0 6.2m 2.2m 0.0 0.1 0:00.00 S `- dbus-launch --autolaunch f8bd609a74b7407db64dfe82f04a8fc6 --binary-syntax --close-stderr
644 user 20 0 4.9m 2.4m 0.0 0.1 0:00.00 S `- /usr/bin/dbus-daemon --syslog-only --fork --print-pid 5 --print-address 7 --session
----
Otherwise my biggest concerns are with qubes-session:
Starting qubes-session...
Failed to issue method call: Caller does not belong to any known session
----
And lastly, this is what happens when I attempt to open Xterm from a shortcut from dom0:
executed QUBESRPC qubes.WaitForSession dom0 pid 712
send exit code 0
pid 712 exited with 0
executed (nowait) QUBESRPC qubes.StartApp+xterm dom0 pid 753
send exit code 0
and this from a dom0 terminal via "qvm-run lfsgo xterm":
executed QUBESRPC qubes.WaitForSession dom0 pid 787
send exit code 0
pid 787 exited with 0
executed QUBESRPC qubes.VMShell dom0 pid 809
pid 809 exited with -1
One thing to note is that the first time I attempt qvm-run, I get a modal in dom0 saying:
"The domain lfsgo attempted to perform an invalid or suspicious GUI request. This might be a sign that the domain has been compromised and is attempting to compromise the GUI daemon (Dom0 domain). In rare cases, however, it might be possible that a ligitimate application trigger such condition (check the guid logs for more information).
Do you allow this VM to continue running? [Ignore/Terminate]
Here's the guid.lfsgo.log:
Icon size: 128x128
Verify failed: untrusted_hdr.type > MSG_MIN && untrusted_hdr.type < MSG_MAX
Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged.
got unknown msg type 147
Thanks for your help again!
Of course, I'll need a lot more work to get everything from the firewall a couple of other minor issues that have cropped up--or PVH--but now I have a fully working LFS system in Qubes!
Going to see if I can fully reproduce this environment with the saved instructions I took and if I'm able, I hope to share this here for any of the other Qubes users who....evidently want to build 100% of their appvms from source lol.
Thanks, thanks, thanks again for your help~