Installing qubes tools to BLFS - qrexec doesn't work

114 views
Skip to first unread message

Will Dizon

unread,
Oct 10, 2018, 1:34:26 PM10/10/18
to qubes-devel
Hi, I have recently completed an installation of BLFS 8.3 (systemd) in an HVM and am attempting to install Qubes tools on it, ultimately with the idea of making it PVH.

As far as BLFS is concerned, the installation seems to have gone without issue, and I have successfully built a kernel that matches the hardware required for XEN and a working lightdm X11 environment.

Where I'm running into the issue is that qvm-run from dom0 does not seem to have any effect on the HVM itself. Here are the basic tests for functionality I have confirmed:

SERVICES RUNNING:

qubes-db.service loaded active running
qubes-early-vm-config.service loaded active exited
qubes-gui-agent.service loaded active running
qubes-meminfo-writer.service loaded active running
qubes-mount-dirs.service loaded active exited
qubes-qrexec-agent.service loaded active running
qubes-sysinit.service loaded active exited

meminfo-writer is checked in the gui

MOUNTS:

xen on /proc/xen type xenfs (rw,relatime)
/dev/xvdb on /rw type ext4 (rw,relatime,discard)
/dev/xvdb on /home type ext4 (rw,relatime,discard)
/dev/xvdb on /usr/local type ext4 (rw,relatime,discard)

#/dev/xvdc I somehow messed up, but I'm guessing isn't relevant to qvm-run

MODULES:

# lsmod
Module Size Used by
tmem 16384 -2
u2mfn 16384 -2

All the other modules, e.g., evtchn are built-in; this is a compact kernel with what I felt were the very bare minimum, but ultimately will be replaced with the PVH kernel. I attempted to use qubes-linux-kernel, but had issues with the dependency on rpms.

TESTS FOR OPERABILITY:

dmesg https://pastebin.com/1WwLZFKL
guest-lfsgo.log https://pastebin.com/fnhw3Tc7
guest-lfsgo-dm.log https://pastebin.com/bhhL76pQ
qrexec.lfsgo.log : is empty
guid.lfsgo.log : contains one line "Icon size: 128x128"

[will@dom0 ~] qrexec-client -d lfsgo lfs:bash
[will@dom0 ~] qrexec-client -d lfsgo user:bash
[will@dom0 ~] qvm-run lfsgo "touch /tmp/dummy"
Running 'touch /tmp/dummy' on lfsgo
[will@dom0 ~] qvm-run --pass-io lfsgo whoami
[will@dom0 ~]

(domU)
root [ ~ ]# uname -a
Linux lfsgo 4.18.9 #4 SMP Fri Oct 5 08:56:34 MST 2018 x86_64 GNU/Linux
root [ ~ ]# su - user
user [ ~ ]$ sudo whoami #passwordless sudo working
root
user [ ~ ]$ exit
logout
root [ ~ ]# xenstore-read name
lfsgo

`ls -la /tmp` shows no sign of '/tmp/dummy', same as with `sudo touch /tmp/dummy`

Are there other logs I can provide to give a clearer picture of my install?

Are there any more-verbose way I can check why qrexec isn't working as expected?

Message has been deleted

Will Dizon

unread,
Oct 10, 2018, 1:44:33 PM10/10/18
to qubes-devel
Pastebin is being problematic for my logs. Attaching:

dmesg
guest-lfsgo.log
guest-lfsgo-dm.log

Marek Marczykowski-Górecki

unread,
Oct 10, 2018, 1:58:20 PM10/10/18
to Will Dizon, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Oct 10, 2018 at 10:34:26AM -0700, Will Dizon wrote:
> Hi, I have recently completed an installation of BLFS 8.3 (systemd) in an HVM and am attempting to install Qubes tools on it, ultimately with the idea of making it PVH.

Nice!
See journalctl (or wherever stderr of services is logged) for
qubes-qrexec-agent service. It should log every command being executed.
If you have X server running (if qubes-gui-agent works properly), there
is also qrexec-fork-server started as part of X session and it's that
process where processes (started as 'user') are executed and it's that
process stderr to inspect.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlu+PbQACgkQ24/THMrX
1yzpYQf/cGAOrANCHS/TYZS/sezu4vI6lAFieoqDIyFvrC8yCNqsAmpn4te6wORm
ypwlHapQT1KMW5bQd5ER+EjdwSp9GnssGh3xES8e62fV9kMTNzKyaAn7ycoaYoEh
6PGqw/gSX0AZiPrQv6q2Cm5++5V6nEUW2f0ywhgwgihkC8A5YacjslrBpbiFwEvj
/vAHoxNUAl3tQJWo1MYRgRHSGoMGLPNYYvX/T7kY+au/+peKz130qHIEYjk6sEHb
Bg1iJWsOPqi7oWhitX6buda/2qij4KM2/0QNBZEtra3elBmwbt+C08u6iJUiXhR1
P90UmYcsRWv6/+Aw+j+rifORxzrvKA==
=SGz4
-----END PGP SIGNATURE-----

Will Dizon

unread,
Oct 23, 2018, 12:07:57 PM10/23/18
to qubes-devel
Marek,

I have modified my startup scripts so that there's no Xorg invoked, but instead Xvfb. So the server boots and the services start up just fine, it appears.

I have a few questions, since I'm still not at a point where I can control this from dom0.

- Can't connect to xl console without `-t pv`

# xl console lfsgo
xenconsole: Could not read tty from store: No such file or directory
# xl console -t pv lfsgo

lfsgo login: user (automatic login)

This qubes is currently set as HVM. Are qubes scripts, e.g., qvm-run or other support detecting HVM and forcing `xl console` to default to `-t serial`?

I'm suspecting that if I set the HVM to PVH instead that qvm-run might try the other pv console and succeed, though that forces me to figure out how to work with xvdd, which I don't have a good grasp on just yet.

- Regarding the logs, dom0 `qrexec-client -d lfsgo ...` produces these journal entries:

root [ /var/log/qubes ]# systemctl status qubes-qrexec-agent
● qubes-qrexec-agent.service - Qubes remote exec agent
Loaded: loaded (/lib/systemd/system/qubes-qrexec-agent.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2018-10-23 15:36:01 MST; 6h left
Process: 479 ExecStartPre=/bin/sh -c [ -e /dev/xen/evtchn ] || modprobe xen_evtchn (code=exited, stat>
Main PID: 521 (qrexec-agent)
Tasks: 1 (limit: 2372)
Memory: 1.3M
CGroup: /system.slice/qubes-qrexec-agent.service
└─521 /usr/lib/qubes/qrexec-agent

Oct 23 08:55:58 lfsgo qrexec-agent[521]: pid 758 exited with 1
Oct 23 08:55:58 lfsgo qrexec-agent[521]: eintr
Oct 23 08:56:12 lfsgo qrexec-agent[521]: executed user:whoami pid 764
Oct 23 08:56:12 lfsgo qrexec-agent[521]: send exit code 1
Oct 23 08:56:12 lfsgo qrexec-agent[521]: pid 764 exited with 1
Oct 23 08:56:12 lfsgo qrexec-agent[521]: eintr
Oct 23 08:56:17 lfsgo qrexec-agent[521]: executed root:reboot pid 767
Oct 23 08:56:17 lfsgo qrexec-agent[521]: send exit code 1
Oct 23 08:56:17 lfsgo qrexec-agent[521]: pid 767 exited with 1
Oct 23 08:56:17 lfsgo qrexec-agent[521]: eintr

I'm not entirely sure how to check the stderr of these processes that seem to die immediately.

Marek Marczykowski-Górecki

unread,
Oct 23, 2018, 1:42:12 PM10/23/18
to Will Dizon, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Oct 23, 2018 at 09:07:57AM -0700, Will Dizon wrote:
> Marek,
>
> I have modified my startup scripts so that there's no Xorg invoked, but instead Xvfb. So the server boots and the services start up just fine, it appears.
>
> I have a few questions, since I'm still not at a point where I can control this from dom0.
>
> - Can't connect to xl console without `-t pv`
>
> # xl console lfsgo
> xenconsole: Could not read tty from store: No such file or directory
> # xl console -t pv lfsgo
>
> lfsgo login: user (automatic login)
>
> This qubes is currently set as HVM. Are qubes scripts, e.g., qvm-run or other support detecting HVM and forcing `xl console` to default to `-t serial`?

qvm-run or others do not use console, so it's separate issue. xl console
defaults to serial for HVM and have no option to change that (other than
patching its sources).

> I'm suspecting that if I set the HVM to PVH instead that qvm-run might try the other pv console and succeed, though that forces me to figure out how to work with xvdd, which I don't have a good grasp on just yet.
>
> - Regarding the logs, dom0 `qrexec-client -d lfsgo ...` produces these journal entries:
>
> root [ /var/log/qubes ]# systemctl status qubes-qrexec-agent
> ● qubes-qrexec-agent.service - Qubes remote exec agent
> Loaded: loaded (/lib/systemd/system/qubes-qrexec-agent.service; enabled; vendor preset: enabled)
> Active: active (running) since Tue 2018-10-23 15:36:01 MST; 6h left
> Process: 479 ExecStartPre=/bin/sh -c [ -e /dev/xen/evtchn ] || modprobe xen_evtchn (code=exited, stat>
> Main PID: 521 (qrexec-agent)
> Tasks: 1 (limit: 2372)
> Memory: 1.3M
> CGroup: /system.slice/qubes-qrexec-agent.service
> └─521 /usr/lib/qubes/qrexec-agent
>
> Oct 23 08:55:58 lfsgo qrexec-agent[521]: pid 758 exited with 1
> Oct 23 08:55:58 lfsgo qrexec-agent[521]: eintr
> Oct 23 08:56:12 lfsgo qrexec-agent[521]: executed user:whoami pid 764
> Oct 23 08:56:12 lfsgo qrexec-agent[521]: send exit code 1
> Oct 23 08:56:12 lfsgo qrexec-agent[521]: pid 764 exited with 1
> Oct 23 08:56:12 lfsgo qrexec-agent[521]: eintr
> Oct 23 08:56:17 lfsgo qrexec-agent[521]: executed root:reboot pid 767
> Oct 23 08:56:17 lfsgo qrexec-agent[521]: send exit code 1
> Oct 23 08:56:17 lfsgo qrexec-agent[521]: pid 767 exited with 1
> Oct 23 08:56:17 lfsgo qrexec-agent[521]: eintr
>
> I'm not entirely sure how to check the stderr of these processes that seem to die immediately.

You should get stderr back to qrexec-client (unless you pass -e option).
If not, probably no output is produced at all - maybe it die before even
starting the process?
You can also strace -f qrexec-agent process to see what it's doing.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlvPXW0ACgkQ24/THMrX
1yynSwf/ZheB/bl1TLat0baCyNQK3k6a+VgzIimaV/KuOdhsZFBqsE35gShQyEhm
aLFRV/SAP71A1WAXDruCkr2OwHpdK6+oFHr1Q6NjXyLvty2jm798Rr7nyLXrVaKw
xX5ckVF/xWaSD/hzQNvoSFg8G8lCPz9aZQNDyJBeKSiKo/T5tTE81uo3WNpBmTe3
FPaJtP54OK3UhZDSpa5WSqKsCpSA57jwS3B7aK/OFu+Bgc+0YIMwiW8Dft1BVXNa
gFZGyblPJ/F7GnjyJ/aqt6SjrLKwASJXvJHPCRbhRiFHcMOBPwcm+6UKi8JU253B
swQltS1nXrevqDDs8ZQtXpq/29ftOA==
=oP9c
-----END PGP SIGNATURE-----

Will Dizon

unread,
Oct 24, 2018, 11:45:25 AM10/24/18
to qubes-devel
Thanks for your continued help; I'm afraid I'm running into a real block here when it comes to making sense of this strace.

https://pastebin.com/a8WD93Na

root [ ~ ]# ls -lah /dev/xen/privcmd
crw-rw---- 1 root qubes 10, 58 Oct 24 2018 /dev/xen/privcmd

The line "read(4," is really where the trace ends--stays there indefinitely. No change to this behavior even if qubes-qrexec-agent.service is stopped/running.


I've also then rebuilt qubes-vmm-xen, qubes-linux-utils, and qubes-core-agent-linux for good measure, with the same strace results.

Marek Marczykowski-Górecki

unread,
Oct 24, 2018, 12:12:16 PM10/24/18
to Will Dizon, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Oct 24, 2018 at 08:45:25AM -0700, Will Dizon wrote:
> Thanks for your continued help; I'm afraid I'm running into a real block here when it comes to making sense of this strace.
>
> https://pastebin.com/a8WD93Na
>
> root [ ~ ]# ls -lah /dev/xen/privcmd
> crw-rw---- 1 root qubes 10, 58 Oct 24 2018 /dev/xen/privcmd
>
> The line "read(4," is really where the trace ends--stays there indefinitely. No change to this behavior even if qubes-qrexec-agent.service is stopped/running.

Try doing that on existing process (strace -fp `pidof qrexec-agent`)
instead of new one. This one looks like waiting for qrexec-daemon to
connect to it (there is no automatic reconnect if you restart
qrexec-agent).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlvQmdoACgkQ24/THMrX
1yxeEAf8C+A4zWWYWYa/Bo08Dj2Bbjs2lFvYiJk2OVPXUAO7lMhsxhK9eVz4h9cI
QEafEBRDsQrGnzSfZiTOpA+uhev/xbJbQIrOHXT/zkNTrDhMAT132P0w+uJ0OQD6
jVxZ2AJuHZWXlpXv0hJrvnabWSnfyIPgZdjiF8fotoN0BDfqq32UNTPoaxSHhXPQ
lVEn8cD54jZLOej9yADdGrCV/1iUOCakDPytDgHEnaNiNAI0tfBPEYXJWeoR7llz
EH+r/Vp2/J8qnFKvhJiVcswm+6MVRtWH+ACwI5eieTFVdS6JOvSARmpENGWtiaxe
GIoCNVWTwInfB38seCDH/jw/gT2Yxw==
=cZxh
-----END PGP SIGNATURE-----

Will Dizon

unread,
Oct 25, 2018, 2:01:58 PM10/25/18
to qubes-devel
> Try doing that on existing process (strace -fp `pidof qrexec-agent`)
> instead of new one. This one looks like waiting for qrexec-daemon to
> connect to it (there is no automatic reconnect if you restart
> qrexec-agent).

Perfect! It seems like that put me on the right track and a few new leads have emerged.

So stracing qrexec-agent led me to qrexec-fork-server:

> connect(8, {sa_family=AF_FILE, path="/var/run/qubes/qrexec-server.user.sock"}, 40) = -1 ENOENT (No such file or directory)

Running `/usr/bin/qrexec-fork-server` as `user` fails with "bind() failed". I `chmod g+w /var/run/qubes`'ed and it let me run the process.

ls -la /var/run/qubes
...
srwxrwxr-x 1 user user 0 Oct 25 10:14 qrexec-server.user.sock

--

`qvm-run lfsgo 'touch /tmp/hi'` strace output from dom0: https://pastebin.com/5FrnAhUg

`journalctl -u qubes-qrexec-agent` output: https://pastebin.com/nKxLxidx

--

Feels like I'm doing some really hacky, non-systematic approaches to get things going, so I know I must've missed a few things.

1) how does qrexec-fork-server typically init'ed?

2) qvm-run still doesn't run the command, but attempting it does push this to "user"'s terminal: "executed QUBESRPC qubes.WaitForSession dom0 pid 679" and the dom0 side process stalls indefinitely.

control-C killing the process pushes "pid 679 exited with -1" on domU's side.

Looking at qubes.WaitForSession, I'm now suspicious that this is now an X issue--maybe surrounding the X session owner--is this a possibility?

--

Other details

# ls -la /tmp
prw-rw-r-- 1 user user 0 Oct 25 10:37 qrexec-rpc-stderr-return.679
prw-rw-r-- 1 user user 0 Oct 25 10:37 qrexec-rpc-stderr.679
-rw-rw-r-- 1 user user 4 Oct 25 10:37 qubes-session-waiter

None of the rpc files have any content in them.

Any guidance would be greatly appreciated!

Will

Marek Marczykowski-Górecki

unread,
Oct 25, 2018, 5:01:15 PM10/25/18
to Will Dizon, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Thu, Oct 25, 2018 at 11:01:58AM -0700, Will Dizon wrote:
> > Try doing that on existing process (strace -fp `pidof qrexec-agent`)
> > instead of new one. This one looks like waiting for qrexec-daemon to
> > connect to it (there is no automatic reconnect if you restart
> > qrexec-agent).
>
> Perfect! It seems like that put me on the right track and a few new leads have emerged.
>
> So stracing qrexec-agent led me to qrexec-fork-server:
>
> > connect(8, {sa_family=AF_FILE, path="/var/run/qubes/qrexec-server.user.sock"}, 40) = -1 ENOENT (No such file or directory)

That's ok. qrexec-fork-server is an optional component. It's purpose is
to start user processes as part of user X11 session to inherit
environment, make systemd-logind happy etc. If it isn't there,
qrexec-agent will start the process itself.

> Running `/usr/bin/qrexec-fork-server` as `user` fails with "bind() failed". I `chmod g+w /var/run/qubes`'ed and it let me run the process.
>
> ls -la /var/run/qubes
> ...
> srwxrwxr-x 1 user user 0 Oct 25 10:14 qrexec-server.user.sock
>
> --
>
> `qvm-run lfsgo 'touch /tmp/hi'` strace output from dom0: https://pastebin.com/5FrnAhUg

This looks like strace from the VM. Which is fine, lets see:


connect(8, {sa_family=AF_FILE, path="/var/run/qubes/qrexec-server.user.sock"}, 40) = -1 ECONNREFUSED (Connection refused)

Ok, no qrexec-fork-server this time. That's fine.

[pid 1675] socket(PF_FILE, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3
[pid 1675] connect(3, {sa_family=AF_FILE, path="/dev/log"}, 110) = 0
[pid 1675] sendto(3, "<85>Oct 25 10:21:33 qrexec-agent"..., 185, MSG_NOSIGNAL, NULL, 0) = 185
(...)
[pid 1675] exit_group(1) = ?

If only that output was a little longer (see strace -s option)... But I
guess it's one of "exited with 1" errors. So, lets scroll a little up:

[pid 1675] openat(AT_FDCWD, "/etc/pam.d/qrexec", O_RDONLY) = 3
(...)
[pid 1675] openat(AT_FDCWD, "/etc/pam.d/system-auth", O_RDONLY) = 4
(...)
[pid 1675] openat(AT_FDCWD, "/etc/pam.d/other", O_RDONLY) = 3
(...)
[pid 1675] openat(AT_FDCWD, "/lib/security/pam_deny.so", O_RDONLY|O_CLOEXEC) = 4


/etc/pam.d/other looks like this:

#%PAM-1.0
auth required pam_deny.so
account required pam_deny.so
password required pam_deny.so
session required pam_deny.so

So, if that's being loaded, this is the reason for the failure. The
question is why, but for that you need to look into your pam
configuration. Start with /etc/pam.d/qrexec.

> Feels like I'm doing some really hacky, non-systematic approaches to get things going, so I know I must've missed a few things.
>
> 1) how does qrexec-fork-server typically init'ed?

- From /usr/bin/qubes-session, started under X11 session.

> 2) qvm-run still doesn't run the command, but attempting it does push this to "user"'s terminal: "executed QUBESRPC qubes.WaitForSession dom0 pid 679" and the dom0 side process stalls indefinitely.
>
> control-C killing the process pushes "pid 679 exited with -1" on domU's side.
>
> Looking at qubes.WaitForSession, I'm now suspicious that this is now an X issue--maybe surrounding the X session owner--is this a possibility?

Normally, qvm-run waits for X11 session using qubes.WaitForSession
service (otherwise things like qvm-run vmname xterm would fail). You can
avoid that with --no-gui option.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAlvSLxUACgkQ24/THMrX
1yw6dgf/Z9nVB06FTnFDZB3SQ2Zto7iP3HDR5Gq4u3mRtzRGCpxObgLpW0FntlaB
uXwH87V1dTzjZYW0E3VhyVHfSP8Nc2KGszGuCdyb633QqYL3RtJ0Jqll0gg1mAIE
PosqB4We646MAxth08kBaECLFkcj5ImYasLZ+zCuZIFu88cSP6uKAg7c3YHNUybk
TeGLo3jxIQ2MIflbIDYlFMI/HnXfPLAI40EXczIZaTEeJEP+4RcKEJncFB+relnV
7deRjiOW+2O92uEz0pigKs1W7FSDNwrgXTX82xrZ0ytC+HilWaZRqharPAbhY5TA
UbHOHu7Dd/zZFS/yce17trsOAPkhMw==
=3njx
-----END PGP SIGNATURE-----

Will Dizon

unread,
Oct 30, 2018, 10:43:33 AM10/30/18
to qubes-devel
Just following up:

Tweaking pam.d/qrexec allowed me to ultimately execute commands from dom0. It appears they ONLY work via --no-gui calls, as anything without that flag stalls/hangs indefinitely.

So in the meantime, it looks like *most* of this is solved; I just have to finally get a grasp of self-installed X that can work to provide an X11 session for qubes.WaitForSession.

Will Dizon

unread,
Jun 18, 2019, 3:16:52 PM6/18/19
to qubes-devel
I failed at this previously, but I'm dead-set on getting this to work.

Right now, almost all the services work--almost everything seems to run in terms of the systemctl services executing and staying alive.

However, the ongoing issue is that I still can't seem to get the X11 session going, so I'm curious what is the best way to test this:

Right now, I can get qubes-gui-agent started and staying alive:

root [ ~ ]# systemctl status qubes-gui-agent
● qubes-gui-agent.service - Qubes GUI Agent
Loaded: loaded (/lib/systemd/system/qubes-gui-agent.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2019-06-18 11:57:54 MST; 21s ago
Process: 645 ExecStartPre=/bin/touch /var/run/console/user (code=exited, status=0/SUCCESS)
Process: 642 ExecStartPre=/bin/mkdir -p /var/run/console (code=exited, status=0/SUCCESS)
Main PID: 646 (qubes-gui)
Tasks: 2 (limit: 2372)
Memory: 3.5M
CGroup: /system.slice/qubes-gui-agent.service
└─646 /usr/bin/qubes-gui

Jun 18 11:57:54 lfsgo systemd[1]: Starting Qubes GUI Agent...
Jun 18 11:57:54 lfsgo systemd[1]: Started Qubes GUI Agent.
Jun 18 11:57:54 lfsgo qubes-gui[646]: Waiting on /var/run/xf86-qubes-socket socket...

---

Despite this, the qubes-gui binary seems to zombie out:

661 root 20 0 3.6m 2.4m 0.0 0.1 0:00.00 R `- top
646 root 20 0 6.0m 3.1m 0.0 0.2 0:00.00 S `- /usr/bin/qubes-gui
647 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 Z `- [qubes-gui-runus] <defunct>

---

I can run the underlying command manually, as root (after sudo -i from user):

exec /usr/bin/qubes-gui-runuser user /bin/sh -l -c "exec /usr/bin/xinit /etc/X11/xinit/xinitrc -- /usr/libexec/Xorg :0 -nolisten tcp vt07 -wr -config xorg-qubes.conf > ~/.xsession-errors 2>&1

It seems to run without error but it drops me back into the parent shell for `user`. Opening xterm from the shortcuts still doesn't work, and xterm from dom0 is no-go, either:

[will@dom0 ~]$ qvm-run -v lfsgo xterm
Running 'xterm' on lfsgo

(waits here forever without opening it).

---

The process it opens looks like this in top:

769 root 20 0 68.7m 2.4m 0.0 0.1 0:00.00 S `- /usr/lib/qubes/qrexec-agent
771 root 20 0 68.9m 2.4m 0.0 0.1 0:00.00 S `- /usr/lib/qubes/qrexec-agent
772 user 20 0 3.9m 2.7m 0.0 0.1 0:00.00 S `- /bin/sh /etc/qubes-rpc/qubes.WaitForSession
781 user 20 0 3.9m 0.3m 0.0 0.0 0:00.00 S `- /bin/sh -l /usr/lib/qubes/qubes-rpc-multiplexer qubes.WaitForSession dom0
783 user 20 0 2.4m 0.7m 0.0 0.0 0:00.00 S `- cat
782 user 20 0 3.9m 1.3m 0.0 0.1 0:00.00 S `- /bin/sh -l /usr/lib/qubes/qubes-rpc-multiplexer qubes.WaitForSession dom0
784 user 20 0 2.2m 0.7m 0.0 0.0 0:00.00 S `- tee /tmp/qrexec-rpc-stderr-return.772
785 user 20 0 2.2m 0.7m 0.0 0.0 0:00.00 S `- logger -t qubes.WaitForSession-dom0
789 user 20 0 2.2m 0.7m 0.0 0.0 0:00.00 S `- sleep inf


There also appears to be no output to /tmp/qrexec-rpc-stderr-return.772

---

What might be the best way to diagnose where qubes-gui is failing to create the Xserver that dom0 can attach to?

Marek Marczykowski-Górecki

unread,
Jun 18, 2019, 3:27:42 PM6/18/19
to Will Dizon, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tue, Jun 18, 2019 at 12:16:52PM -0700, Will Dizon wrote:
> I failed at this previously, but I'm dead-set on getting this to work.
>
> Right now, almost all the services work--almost everything seems to run in terms of the systemctl services executing and staying alive.
>
> However, the ongoing issue is that I still can't seem to get the X11 session going, so I'm curious what is the best way to test this:
>
> Right now, I can get qubes-gui-agent started and staying alive:
>
> root [ ~ ]# systemctl status qubes-gui-agent
> ● qubes-gui-agent.service - Qubes GUI Agent
> Loaded: loaded (/lib/systemd/system/qubes-gui-agent.service; disabled; vendor preset: enabled)
> Active: active (running) since Tue 2019-06-18 11:57:54 MST; 21s ago
> Process: 645 ExecStartPre=/bin/touch /var/run/console/user (code=exited, status=0/SUCCESS)
> Process: 642 ExecStartPre=/bin/mkdir -p /var/run/console (code=exited, status=0/SUCCESS)
> Main PID: 646 (qubes-gui)
> Tasks: 2 (limit: 2372)
> Memory: 3.5M
> CGroup: /system.slice/qubes-gui-agent.service
> └─646 /usr/bin/qubes-gui
>
> Jun 18 11:57:54 lfsgo systemd[1]: Starting Qubes GUI Agent...
> Jun 18 11:57:54 lfsgo systemd[1]: Started Qubes GUI Agent.
> Jun 18 11:57:54 lfsgo qubes-gui[646]: Waiting on /var/run/xf86-qubes-socket socket...
>
> ---
>
> Despite this, the qubes-gui binary seems to zombie out:
>
> 661 root 20 0 3.6m 2.4m 0.0 0.1 0:00.00 R `- top
> 646 root 20 0 6.0m 3.1m 0.0 0.2 0:00.00 S `- /usr/bin/qubes-gui
> 647 root 20 0 0.0m 0.0m 0.0 0.0 0:00.00 Z `- [qubes-gui-runus] <defunct>

Looks like a problem with starting X server. Check ~/.xsession-errors
and ~/.local/share/xorg/Xorg.0.log (or other xorg log location).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl0JOygACgkQ24/THMrX
1yyKZAgAi2ksSzd+V2hWfCe6JB14f1bjDR64g+XXZBkpRhIRPoXZhi4D0IGGqyZo
zXmyGrJj+sR334NsDYnz0gjMAec+OwbyBOsXggXy8PL2uQCciVgboTe0AFxbjZnO
sGcgM2KS4TkEzxbTlSqBsm19jn/mhaLx4woGgf7zVl1CdbkOFH6xlDYZyvNp7TH+
UaEH+G/IL385qL4QW881k5VGBna3MgwhCW0mepTPHgdBTXgcsSM4myzWUnZRlDrl
0QBLcMDgMU+ftkcUTaQiyOf5ojcDjCIsWqEZFpGSqU4m/gwF6wVUJWfUr63d/rB8
C3p1feyn4SLdRAbL/492I6Q6SW4mgA==
=9NfP
-----END PGP SIGNATURE-----

Will Dizon

unread,
Jun 19, 2019, 3:16:06 PM6/19/19
to qubes-devel
Thanks Marek,

It looks like my barebones pam.d configuration was stifling the qubes-gui-agent process. I went ahead and made the /etc/pam.d/qubes-gui-agent more permissive and it seems I'm making huge strides toward it working. The newest issue is now coming up in Dom0(!).

Here's my ~/.xsession-errors: https://pastebin.com/Cf0mfGz3

Xorg.0.log looks pretty clean, start to finish, but here's the paste anyway: https://pastebin.com/BEj9tHm0

----

There's an early error, which I believe to be dbus. Not sure if this is a huge obstacle, but I notice that `dbus-launch --autolaunch` is already running before the system's dbus-daemon:

642 user 20 0 6.2m 2.2m 0.0 0.1 0:00.00 S `- dbus-launch --autolaunch f8bd609a74b7407db64dfe82f04a8fc6 --binary-syntax --close-stderr
644 user 20 0 4.9m 2.4m 0.0 0.1 0:00.00 S `- /usr/bin/dbus-daemon --syslog-only --fork --print-pid 5 --print-address 7 --session

----

Otherwise my biggest concerns are with qubes-session:

Starting qubes-session...
Failed to issue method call: Caller does not belong to any known session

----

And lastly, this is what happens when I attempt to open Xterm from a shortcut from dom0:

executed QUBESRPC qubes.WaitForSession dom0 pid 712
send exit code 0
pid 712 exited with 0
executed (nowait) QUBESRPC qubes.StartApp+xterm dom0 pid 753
send exit code 0

and this from a dom0 terminal via "qvm-run lfsgo xterm":

executed QUBESRPC qubes.WaitForSession dom0 pid 787
send exit code 0
pid 787 exited with 0
executed QUBESRPC qubes.VMShell dom0 pid 809
pid 809 exited with -1

One thing to note is that the first time I attempt qvm-run, I get a modal in dom0 saying:

"The domain lfsgo attempted to perform an invalid or suspicious GUI request. This might be a sign that the domain has been compromised and is attempting to compromise the GUI daemon (Dom0 domain). In rare cases, however, it might be possible that a ligitimate application trigger such condition (check the guid logs for more information).

Do you allow this VM to continue running? [Ignore/Terminate]

Here's the guid.lfsgo.log:

Icon size: 128x128
Verify failed: untrusted_hdr.type > MSG_MIN && untrusted_hdr.type < MSG_MAX
Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged.
got unknown msg type 147


Thanks for your help again!

Marek Marczykowski-Górecki

unread,
Jun 19, 2019, 5:27:15 PM6/19/19
to Will Dizon, qubes-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, Jun 19, 2019 at 12:16:05PM -0700, Will Dizon wrote:
> Thanks Marek,
>
> It looks like my barebones pam.d configuration was stifling the qubes-gui-agent process. I went ahead and made the /etc/pam.d/qubes-gui-agent more permissive and it seems I'm making huge strides toward it working. The newest issue is now coming up in Dom0(!).
>
> Here's my ~/.xsession-errors: https://pastebin.com/Cf0mfGz3
>
> Xorg.0.log looks pretty clean, start to finish, but here's the paste anyway: https://pastebin.com/BEj9tHm0
>
> ----
>
> There's an early error, which I believe to be dbus. Not sure if this is a huge obstacle, but I notice that `dbus-launch --autolaunch` is already running before the system's dbus-daemon:
>
> 642 user 20 0 6.2m 2.2m 0.0 0.1 0:00.00 S `- dbus-launch --autolaunch f8bd609a74b7407db64dfe82f04a8fc6 --binary-syntax --close-stderr
> 644 user 20 0 4.9m 2.4m 0.0 0.1 0:00.00 S `- /usr/bin/dbus-daemon --syslog-only --fork --print-pid 5 --print-address 7 --session
>
> ----
>
> Otherwise my biggest concerns are with qubes-session:
>
> Starting qubes-session...
> Failed to issue method call: Caller does not belong to any known session

Those things are probably related. Do you have pam_systemd in
/etc/pam.d/qubes-gui-agent? Also, it may be some kind of issue with
environment variables propagation - namely DBUS_SESSION_BUS_ADDRESS and
that could explain two dbus instances.

> ----
>
> And lastly, this is what happens when I attempt to open Xterm from a shortcut from dom0:
>
> executed QUBESRPC qubes.WaitForSession dom0 pid 712
> send exit code 0
> pid 712 exited with 0
> executed (nowait) QUBESRPC qubes.StartApp+xterm dom0 pid 753
> send exit code 0

Looks fine...

> and this from a dom0 terminal via "qvm-run lfsgo xterm":
>
> executed QUBESRPC qubes.WaitForSession dom0 pid 787
> send exit code 0
> pid 787 exited with 0
> executed QUBESRPC qubes.VMShell dom0 pid 809
> pid 809 exited with -1
>
> One thing to note is that the first time I attempt qvm-run, I get a modal in dom0 saying:
>
> "The domain lfsgo attempted to perform an invalid or suspicious GUI request. This might be a sign that the domain has been compromised and is attempting to compromise the GUI daemon (Dom0 domain). In rare cases, however, it might be possible that a ligitimate application trigger such condition (check the guid logs for more information).
>
> Do you allow this VM to continue running? [Ignore/Terminate]
>
> Here's the guid.lfsgo.log:
>
> Icon size: 128x128
> Verify failed: untrusted_hdr.type > MSG_MIN && untrusted_hdr.type < MSG_MAX
> Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged.
> got unknown msg type 147

Oh, I think you compiled gui-agent from master branch (R4.1), but run it
with with R4.0 dom0. Use release4.0 branch of gui-agent (and
gui-common), if you run it on R4.0 dom0.

You may hit similar issue with core-agent-linux (in R4.1 it is split
into core-agent-linux and core-qrexec, and there are qrexec protocol
changes too).

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAl0KqKkACgkQ24/THMrX
1ywzpAf/bk9h399/clUbNbtt3vxc+nHVIHBYYCNqBNwb4tiGMcuLUcF6jeA82QOE
4UH1GOQikOUD6HdalrY7QhLDQrtfmZ/ZWExUSPe17R8l7EZyhr+19BZGrIAwNt11
xv0/il6XBs6YIhzhv3rz9/XYlIJIsxR2Ky+eBaxJZUJ5r0OZMU4dIyuB9Wl1NuyE
oWhE4i/PCCOouj9l8GKXeAH7U6w9KEriys8n+2Rq/PqNu8NZkcoUMquCrv5JUhxa
q3YUCaeG0uUQFFu4reIYdQ/vTHhU7fwTpStl4ZUJSacuJmCSLLCICy/IjE+2MUjM
yfvUXuTerYojacF7robxEuPXKXpL+w==
=oaRi
-----END PGP SIGNATURE-----

Will Dizon

unread,
Jun 19, 2019, 5:58:52 PM6/19/19
to qubes-devel
I am over the moon right now--I definitely built with too recent a qubes-gui-agent-linux and once I rebuilt with R4.0, Xterm immediately opened and worked!


Of course, I'll need a lot more work to get everything from the firewall a couple of other minor issues that have cropped up--or PVH--but now I have a fully working LFS system in Qubes!

Going to see if I can fully reproduce this environment with the saved instructions I took and if I'm able, I hope to share this here for any of the other Qubes users who....evidently want to build 100% of their appvms from source lol.


Thanks, thanks, thanks again for your help~

Reply all
Reply to author
Forward
0 new messages