Strange Chrome crashes under Xen (Qubes) 4.1 (R2) and 4.4 (R3)

[Joanna Rutkowska]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've been experiencing strange Chrome browser crashes under Linux PVs (Qubes
AppVMs) for some time now. This is tracked by the following ticket:

https://github.com/QubesOS/qubes-issues/issues/1003

This happens to me on two different machines, one running Qubes R2 with Xen 4.1
and on Fedora 20 template, the other R3.0 with FC21 template. I could not
reproduce the same bug when Chrome runs on baremetal... It's really hard for me
to believe I'm the only one experiencing this issue. Am I?

Thanks,
joanna.
-----BEGIN PGP SIGNATURE-----

iQIcBAEBAgAGBQJVbN5wAAoJEDOT2L8N3GcYFFYP/37iaFQxWrhR75bNNbfbffcD
zGcyd8Gq0ubAe57LeOfwfpPJFmyEPWjNL3H6RsyITeW3fjZ3lIi3Fw9thGYJOUWh
NxPUZAs2mJyokKsGpPVIvrzHlpJQ993bmt34T7LPBNho+JageCJLSFnacwCn9y2g
I2IvO9Hz5/y0lsile+RbV2TCAZsAk2hXHRZr+TUWjXayhiYcb5ztmQg+duUyZ34T
SdclCzZrR9/RsOW8VISEKrrQR9qLzS1pSIUyAEui64L3UWwD3H35yzVFFDo2YxSh
ad2fvTTtWpqdOC1Em75CAjV2xcHGo/PBpc7RIpnN9ozy/Lvmey5AagGShyKwEkDW
oqbiEvjHE3u4KgtITiedvUPTvbcpIP5yps/hlzRmufzUvCST+KbL6UI8iWCTpcog
or79bTF9q60bdnSz2VP3+4FUWfLdchi8UryDQpz211vLNKPKlvqpY9SYoo9ddV1z
JuCbM3/mMd0slhKyR1BqU8SNOX72g8RLP/yEF+vDZVMzIIU2MNkVVeMI5pcyFqLY
WlXgaAntorEsTjBe2xk+waJ4cvuPdH3/UUK1R47IZU6zOIs/juvtTPjuV3g0JVEF
ZSsUusoOiKtShxSphaBurMZVq6Sd7irqBnSAXbntT8lP2lMGd/b6QZc3bD6NlMIE
sD+eYElb38k9/Qy94uyt
=k4Yd
-----END PGP SIGNATURE-----

[7v5w7go9ub0o]

On 06/01/15 18:36, Joanna Rutkowska wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I've been experiencing strange Chrome browser crashes under Linux PVs (Qubes
> AppVMs) for some time now. This is tracked by the following ticket:
>
> https://github.com/QubesOS/qubes-issues/issues/1003
>
> This happens to me on two different machines, one running Qubes R2 with Xen 4.1
> and on Fedora 20 template, the other R3.0 with FC21 template. I could not
> reproduce the same bug when Chrome runs on baremetal... It's really hard for me
> to believe I'm the only one experiencing this issue. Am I?
>

FWIW, I could not replicate the crash described in 1003.

R2; Xen 4.1.6.1; Fedora 20 dispvm

I did get "plugin failures?" a few weeks ago, but none after recent
application software updates.

[Zrubi]

On 06/02/15 00:36, Joanna Rutkowska wrote:

> I've been experiencing strange Chrome browser crashes under Linux PVs (Qubes
> AppVMs) for some time now. This is tracked by the following ticket:
>
> https://github.com/QubesOS/qubes-issues/issues/1003
>

> It's really hard for me
> to believe I'm the only one experiencing this issue. Am I?

I have no chrome installed can't try this.

May be a lame question but this may be because in PV env you have much
lower amount of RAM than you have in a physical machine.


May be not related - but sometimes my firefox freeze in a DVM when the
only solution is to kill that VM - but never investigated this issue
because only happens in a DVM when I visiting a bloated news portal..

--
Zrubi

[bow...@gmail.com]

Happy to try it out. Let me know what do do to set it up

Alex

> --
> You received this message because you are subscribed to the Google Groups "qubes-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users...@googlegroups.com.
> To post to this group, send email to qubes...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/556D4918.5010903%40zrubi.hu.
> For more options, visit https://groups.google.com/d/optout.

[JPL]

Chrome is very unstable in Qubes in my experience, and is worse in RC3.0. It crashes when I use the mouse wheel to scroll down quite often, and also when using the flash plugin quite often. And the flash plugin itself crashes quite often. All of these happen quite often but not always so it's hard to pinpoint a cause. I don't really use it any more - waiting for it to be debugged. Would it help to start sharing logs after crashes?

[Olivier Médoc]

On 06/02/15 00:36, Joanna Rutkowska wrote:

I've been experiencing strange Chrome browser crashes under Linux PVs (Qubes
AppVMs) for some time now. This is tracked by the following ticket:

https://github.com/QubesOS/qubes-issues/issues/1003

This happens to me on two different machines, one running Qubes R2 with Xen 4.1
and on Fedora 20 template, the other R3.0 with FC21 template. I could not
reproduce the same bug when Chrome runs on baremetal... It's really hard for me
to believe I'm the only one experiencing this issue. Am I?

Thanks,
joanna.

>

I had a similar problem with googleearth.

The crash stack trace pointed out a bug a llvm / mesa drivers which where triggered only on one of my coleague laptop (which used very similar hardware). The reason was a bug causing a call to specific processor instruction that was not available/vurtualized/implemented in a xen appvm context. I had to recompile llvm and mesa libraries from their git to fix it.

Just to conclude that such bugs may not be reproducible with different hardware.

P.S. Do you have any idea of the libraries that make google chrome crashing ?

[Raymond Rizzuto]

I use Chrome with Qubes R2 and have not noticed this behavior.  I'm running on a HP Elitebook 8460p with a i5-2520M, 8gig of ram, 180GB SSD. I'm using whatever is the latest stable version of Chrome.

[Valko]

I have this problem since less then 1month or last 2-3 chrome updates.

Tried to uninstall chrome, deleted all chrome files from appvm and template.

I'm using Qubes R2 with fc20 with latest updates every time.

The problem happens everytime i play slot games on betvictor.com or betvictor.net (domain .com is restricted in some countries)

How i reproduce the problem:

i load the betvictor.com/net

go to casino section

scroll down move around click here and there and chrome crash.

sometimes it close the windows, sometimes it says "Aw, Snap!" screen and after refresh it works fine until i click to play some game.

When i click "Play Now" on some games new window opens and start to load the game after it load both windows(main casino window and game window) shows screen "Aw, Snap!". Then i refresh game window and leave main casino window with crash screen and the game window loads fine, sometimes from 2-3 refresh. some games don't load at all doesn't matter how many times i refresh the game window.

It's very anoying because i don't have flash installed and open flash sites only with chrome.

Same crashes happens sometimes in gmail or google keep.

Tell me what should I send as report and will reproduce my crash. I think it's the same bug because my friend with windows who play the same site don't have such problem and also I have this problem since maybe 1 month or less i think before 2 months the casino was working fine without crashes with chrome in Qubes. from the casino support they told me to use IE.

my machine is lenovo t430 i5-3320 8gb 180gb ssd, intel video

[Zrubi]

When this happens I still be able to start a terminal and just saw this
in ps output:

root 495 0.0 0.0 209560 500 ? Ss May28 0:00
/usr/bin/abrt-watch-log -F BUG: WARNING: at WARNING: CPU: INFO: possible
recursive locking detected ernel BUG at list_del corruption list_add
corruption do_IRQ: stack overflow: ear stack overflow (cur: eneral
protection fault nable to handle kernel ouble fault: RTNL: assertion
failed eek! page_mapcount(page) went negative! adness at NETDEV WATCHDOG
ysctl table check failed : nobody cared IRQ handler type mismatch
Machine Check Exception: Machine check events logged divide error:
bounds: coprocessor segment overrun: invalid TSS: segment not present:
invalid opcode: alignment check: stack segment: fpu exception: simd
exception: iret exception: /var/log/messages -- /usr/bin/abrt-dump-oops -xtD

But no idea what is this all means - what I see is a frozen firefox



--
Zrubi

[Fredrik Strömberg]

On Tue, Jun 2, 2015 at 12:36 AM, Joanna Rutkowska
<joa...@invisiblethingslab.com> wrote:
> I've been experiencing strange Chrome browser crashes under Linux PVs (Qubes
> AppVMs) for some time now. This is tracked by the following ticket:
>
> https://github.com/QubesOS/qubes-issues/issues/1003
>
> This happens to me on two different machines, one running Qubes R2 with Xen 4.1
> and on Fedora 20 template, the other R3.0 with FC21 template. I could not
> reproduce the same bug when Chrome runs on baremetal... It's really hard for me
> to believe I'm the only one experiencing this issue. Am I?
>
> Thanks,
> joanna.

You are not alone. I experience frequent crashes with Chrome running
in an AppVM under Qubes R3rc1 with the fedora-21 template.

Sometimes only the specific tab crashes, and sometimes the entire
Chrome process. Pages with lots of graphics, such as landing pages,
crash more frequently.

Cheers,
Fredrik

[Randy Jonasz]

On Monday, June 29, 2015 at 9:48:04 AM UTC-4, Fredrik Strömberg wrote:

On Tue, Jun 2, 2015 at 12:36 AM, Joanna Rutkowska
<joa...@invisiblethingslab.com> wrote:
> I've been experiencing strange Chrome browser crashes under Linux PVs (Qubes
> AppVMs) for some time now. This is tracked by the following ticket:
>
> https://github.com/QubesOS/qubes-issues/issues/1003
>

You might consider another browser with the news Google installs a blackbox program in Chrome which listens to sound on the computer's mic and sends the stream to Google's servers for "safe keeping".

Regards,

Randy

[lanythe]

That should be mitigated entirely by not attaching an audio-input device
to the particular VM, right? ;)
However, I see your point. Chrome contains a bunch of "features" that
"phone home" to the proverbial "mothership".

For what it's worth, I can't reproduce the Chrome crashes even with
heavy flash usage. Using: mobile i5, integrated video, debian-testing
VM, R2 and R3.

-Laynthe

[Joanna Rutkowska]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So, any progress with fixing that?

joanna.
-----BEGIN PGP SIGNATURE-----

iQIcBAEBAgAGBQJVw0bZAAoJEDOT2L8N3GcYnysQANwgbULhck1joKSysMFU5Rye
st395o0dTuMdFBkaATynr0OopaRFAtrPGAwIcZxD1LTkHfOavN5aV6kOL4idzHvT
1T4aAzNWUD33gptlNL5fOXf/NFGAPQU+rHD1PAz3Yfo4NWi+Pry0Cg17IKSvJlI8
i32CqZurMP9WC9vydFyjwsuwekedYQBRqEk8GecWWqkrjmYuzjGpwhEYQ+oqkXeh
aJOzXoXRa/Z94M//W+7ztxEKYp9ReOTKl215T2zMvRQaEss/tneBCWpXp6l6QXdj
0k6HZ2JUKOZai/cEOJKfpyjjTjIW7DRP/VpPfhq9rUf9YYr2AC1TZRR7yvc3VGhA
uncziW2QCRpR0zbuTCRZAEGWJMD7UjWOb0H+r3lRHFph6jmN6TM0jgFbSrokl/y8
Ge2vV/Fo91iP1FFD6yqTXwaKTOpjkYnzLSByQUb9CcNTVquuOS9WlIcW6dwW4pY3
6/nproSKQcS/3iHPVDxwjas8U+0/FbMwJ2WySdIb46FvaNknFwxrcKCBAUmCytw9
fKZrvnEqpSCC0vpI5lPqJasII9DsJTGz9/Qq/wQkn0kCUs87j32AQTFVAvUSQAbo
ceZKVu6N7A0fgmyJm/HlrDdyxowiOP8zvi40Ce4ld78s91TlrLmPpxQRqOgx2yP+
8SUfz/YOsZsmnnwVAkc7
=naH2
-----END PGP SIGNATURE-----

[Fredrik Strömberg]

On Thu, Aug 6, 2015 at 1:36 PM, Joanna Rutkowska
<joa...@invisiblethingslab.com> wrote:
> On Mon, Jun 29, 2015 at 03:48:02PM +0200, Fredrik Strömberg wrote:
>> On Tue, Jun 2, 2015 at 12:36 AM, Joanna Rutkowska
>> <joa...@invisiblethingslab.com> wrote:
>> > I've been experiencing strange Chrome browser crashes under Linux PVs (Qubes
>> > AppVMs) for some time now. This is tracked by the following ticket:
>> >
>> > https://github.com/QubesOS/qubes-issues/issues/1003
>> >
>> > This happens to me on two different machines, one running Qubes R2 with Xen 4.1
>> > and on Fedora 20 template, the other R3.0 with FC21 template. I could not
>> > reproduce the same bug when Chrome runs on baremetal... It's really hard for me
>> > to believe I'm the only one experiencing this issue. Am I?
>> >
>> > Thanks,
>> > joanna.
>>
>> You are not alone. I experience frequent crashes with Chrome running
>> in an AppVM under Qubes R3rc1 with the fedora-21 template.
>>
>> Sometimes only the specific tab crashes, and sometimes the entire
>> Chrome process. Pages with lots of graphics, such as landing pages,
>> crash more frequently.
>>
>
> So, any progress with fixing that?
>
> joanna.

No, I switched to Firefox because it was unusable.

I'll admit I haven't taken the time to debug it at all since I guessed
from your qubes-issue #1003 that this was outside my area of
expertise. I haven't touched C in 10 years and my Xen-fu is weak.

Please let me know if I can contribute somehow.

Cheers,
Fredrik

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Mon, Aug 31, 2015 at 03:34:38PM -0700, Eric Shelton wrote:

> Since this involves AVX and SSE instructions, maybe Xen
> commit 3af450fd2d9403f208d3ac6459716f027b8597ad is worth checking out? It
> was described as follows:
>
> The two MMX/SSE/AVX code blocks failed to update IP properly, and these
> as well as get_reg_refix(), which "manually" updated IP so far, failed
> to do the TF and RF processing needed at the end of successfully
> emulated instructions.

This commit is already included Xen 4.4 (last commit just before 4.4.1
release). This might be some regression introduced by it... Eric, are
you able to reproduce this problem?

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJV6OlGAAoJENuP0xzK19csu5MIAIzffv/vQnR9OBlkxRQfxsaQ
kgdhGRAz2aAQilhR0FRjnmc0i8r6zl0o66eO4O0f66+Ir61eebpThBhoyiArHEn8
JKL/SCaO7LBZyATVaKDxYSnNPHEPL2KfH12RafB4g6Gw70gwaNOZOHDLFScoeXhI
FesElSIwiVTXxhLEJ/K9f0vhWNpJZUL/m9bzZdS7DiEkm1jNrPXjtH5m6Wd79LdR
pEd0py9YeuDePES6E5Jtwrn89tkAo+wq/mfxSbUlPwhkgNgz55UTzudNgG9pf0re
UlS0l9ZmLzUkmz4+0UTSdwEbuXrk1+PvME9NHro97gfp30YiJeRV2cU5IzZvTl4=
=CSH3
-----END PGP SIGNATURE-----

[Eric Shelton]

Yeah, I ended up realizing the change was included about 5 minutes after I posted - I forgot to consider it could have found its way into the maintenance releases.

I have been seeing chrome crashes very frequently.  I thought upgrading chrome to the latest version had helped a little bit, but it still crashes.  Running Qubes R3 rc2.  Anything I can do to help track it down?

Eric

[Eric Shelton]

I turned on logging in the hypervisor, and crashed chrome a few times via running Google Maps (reliably causes tab or browser to crash).  There are a number of lines that report "page_alloc.c:1468:dX Over-allocation for domain."  I have attached a portion of the output of 'xl dmesg'.  Domain 3 was the domain in which chrome was being run.

Eric

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Can you correlate this with qmemman log? Both don't have timestamps, but
doing single tail -f on both of them should give some results.
Hypervisor log is written to /var/log/xen/console/hypervisor.log.
Maybe also some correlation with the crash would be helpful (just use VM
console log, with journalctl -f or sth running there).

BTW Honestly I think the whole issue have nothing to do with AVX, IMHO
its just about wrong address for memmove. This seems probable especially
in context of [1].

Maybe it is about some lack of memory? It worth trying to stop qmemman
and manually assigning some large amount of RAM there.

[1]
https://github.com/QubesOS/qubes-issues/issues/1003#issuecomment-104895531

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJV6PYJAAoJENuP0xzK19cs7L4H/isaepcPthJwGYO7UkaLD17R
Oejb5KR8amOT2x6xBV8xQt78iytBkPvmLOyMdyZuH8H3UIQ/T7mBHPyxCSX+kM79
j2t4o45bZvD1s1RJVpj51JRaBmwWRT3X39F+K65DQlT/khzlzgZKVIdabZvOgf5x
FSO2k43k721uUdfs6P3egoYGK5Af99NKCY6AMSSlqjX+t22mwX1w4NUMEDAJWEBF
hvr9KMiFWYFPtCQ8kir7yDUFeF8RYq8zzc0P3u688/WC6cWfbpjb8mEMR3B/SnF9
ZFJMN9cL6WKqmHmVKKcBqIozPheCOlpG6HurZBEIl1uob8iJG703Wsr4s9TQMyA=
=AynH
-----END PGP SIGNATURE-----

[Eric Shelton]

So, although I don't have time right now to try out what you've set out above, I did try another quick experiment.  In Qubes Manager, under the Advanced tab I cleared the "Include in memory balancing" checkbox, and set the "Initial memory" field to a large value (3500 MB).  With these settings, I was no longer able to get Google Maps to crash Chrome.  So, it does look like memory ballooning is at fault here - perhaps qmemman is unable to respond quickly enough to Chrome's demands for memory.

Eric

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Take a look at latest comment on github - it looks like it is about
/dev/shm being too small. Still needs to be confirmed.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJV6a8JAAoJENuP0xzK19csTVIH/0MAf6iCOOkDQ7zfQ4La/4WF
Owbi9ALBJImk2IWM8EV84NRt/2gceMZE6//9XuEFIIBHyUHjjidEXTKTIhd1+Rt/
8xubkrIYbQ8lsyo2SvYyHTTXI+u8S6vedl+WISaCtg/HfFAeMyYnw+hXoANm5mQ6
+eiKxcJlJVyqujnIChttk9a/bsXBiisczzHBhV0Lz2YPlCC7X/wWGfKplx1is+YP
3BGpjUJxwGqEcGduSlAvumDJpylGuNEixitdvRr5+Vn4Fwweg43rxXs/oRVILR6m
Hdwy7GBLkP807sGr/T/YrDd5TIfvOOWLTxyodiKZNBzb6J8yc4PZA+us0lBiPdY=
=oJqL
-----END PGP SIGNATURE-----

[Eric Shelton]

On Friday, September 4, 2015 at 10:47:47 AM UTC-4, Marek Marczykowski-Górecki wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Sep 04, 2015 at 07:45:02AM -0700, Eric Shelton wrote:
> On Thursday, September 3, 2015 at 9:38:24 PM UTC-4, Marek
> Marczykowski-Górecki wrote:
> > Maybe it is about some lack of memory? It worth trying to stop qmemman
> > and manually assigning some large amount of RAM there.
> >
> > [1]
> > https://github.com/QubesOS/qubes-issues/issues/1003#issuecomment-104895531
> >
>
> So, although I don't have time right now to try out what you've set out
> above, I did try another quick experiment.  In Qubes Manager, under the
> Advanced tab I cleared the "Include in memory balancing" checkbox, and set
> the "Initial memory" field to a large value (3500 MB).  With these
> settings, I was no longer able to get Google Maps to crash Chrome.  So, it
> does look like memory ballooning is at fault here - perhaps qmemman is
> unable to respond quickly enough to Chrome's demands for memory.

Take a look at latest comment on github - it looks like it is about
/dev/shm being too small. Still needs to be confirmed.

Rather than try to reproduce the failure on bare Fedora (which I do not have set up), is there something I should try to do in Qubes to see if it resolves the problem?

Eric

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

The opposite - increase /dev/shm and /tmp :) According to robkinyon's
comment it isn't enough, but maybe together with /tmp it will do? Or
also some other tmpfs there? Maybe it worth trying to strace or at least
watch lsof to see what files are accessed at the crash time.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJV6bTKAAoJENuP0xzK19csy/QH/jr+Wod50nfSDdnU02+X9gJi
lJj1mmskT3xIaBoPpXYO1RpPW4t175pdP0jlrwe5CSFOFpnC87CUvBbLtAy6UIzi
LkQYlebgAKMDxKavLa1D/eX3JxO6Rs9vbGAHgZVtb0eErYPlwObDxzUvF043KwUE
tOdaJyLukmd02BXR851MJI5jP6Qew8UXLey6IziYK38jVlySilVfTyDh2UrFXC9C
KDSTptOxYgikfFu21f3Ui8uCkq5OzDc/aIZVBiAwgPKWo678nFuRjtbmoRz614iA
+7UYAVWtuQMYJYOWRL/SoHnXVQBIybn1m0DflOYEW3p+yqopTmJZE6Wtx03I0lQ=
=LfWr
-----END PGP SIGNATURE-----

[Eric Shelton]

But I don't have to do any of that to keep Chrome from crashing when ballooning is disabled for the appvm, so I am a little surprised that a fix would lie in that direction.

One thing that has always seemed odd to me about the Qubes ballooning is that "free" memory is pushed back into dom0, which must relinquish the memory again for it to be assigned to an appvm that needs it.  For example, on a system with 16GB of memory, dom0 might have 13GB assigned to it, which is totally unnecessary.

http://wiki.xenproject.org/wiki/Tuning_Xen_for_Performance#Memory suggests:

"If the host has more memory than a typical laptop/desktop system, then do not rely on dom0 ballooning. Instead set the dom0 memory to be something between 1 and 4GB adding dom0_mem=1024M to the Xen command line. ... Also ballooning down busy dom0 might have bad side effects."

I wonder if what we are seeing here is one or those "bad side effects," and that perhaps extra memory should be left unassigned to allow the hypervisor to more effectively respond to the types of memory pressures being created by Chrome and Docker.  However, this is getting into Xen architecture details that I am not familiar with.

Eric

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

tmpfs size is half of memory size _at system startup_ by default. When
you disable memory ballooning, the VM have 4GB at boot time so all
tmpfs are 2GB. Otherwise about 150MB (half of initial 300MB).

> One thing that has always seemed odd to me about the Qubes ballooning is
> that "free" memory is pushed back into dom0, which must relinquish the
> memory again for it to be assigned to an appvm that needs it. For example,
> on a system with 16GB of memory, dom0 might have 13GB assigned to it, which
> is totally unnecessary.

Actually it is distributed proportionally among all the running VMs. It's
simply that dom0 uses the most memory (KDE...) so it get more of it.

https://www.qubes-os.org/doc/Qmemman/

> http://wiki.xenproject.org/wiki/Tuning_Xen_for_Performance#Memory suggests:
>
> "If the host has more memory than a typical laptop/desktop system, then do
> not rely on dom0 ballooning. Instead set the dom0 memory to be something
> between 1 and 4GB adding dom0_mem=1024M to the Xen command line. ... Also
> ballooning down busy dom0 might have bad side effects."

Yes, limiting dom0 memory might have some sense. But choosing the right
value, especially on systems with 4GB would be tricky (memory hungry
desktop environment on the one side, VMs on the other side). You're also
free to choose which desktop environment you use (or which features you
enable), so it is even harder choice.

> I wonder if what we are seeing here is one or those "bad side effects," and
> that perhaps extra memory should be left unassigned to allow the hypervisor
> to more effectively respond to the types of memory pressures being created
> by Chrome and Docker. However, this is getting into Xen architecture
> details that I am not familiar with.

I don't think that chrome crashes are related to dom0 memory assignment.
But you can do a simple test:
1. Start the VM (with enabled memory balancing)
2. Start chrome
3. Stop qmemman in dom0
4. Take some memory from dom0 (or other VMs), so there is more not
assigned memory left (xl mem-set, xl info)
5. Try to reproduce chrome crash

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJV6b8WAAoJENuP0xzK19csrDgH/3N+M/m1fiKuOreQkE9L86Os
gujrJw8B6ZPZuLK/GO7psM5/FHnKJnlOBsXZGhP/VD361h5ztU+jIJVHx5MCEAZ2
P2e4tK3Ktje2x3YOObjdl5o7smIX/TZfGFcE/bIyCZUYY2MtvFM93UIn/JCnEEMT
V0cq4qsEZ8noLYwFp3YttJUNmv1IHb1lcYAueqCDLCFx4CTgIZIZFYHCEOz2WQ0K
sHAnrN9kwF7WnxNiJlCrn6OxNC10X8uiuup2tVtOR8FgORcskM6YHessXsucOsR7
vpbB9NdoPrFM4g+YycwCB2m3o3aWT1UamDAFdxi6VDt+0gsRQHRk2mONCzUn1QE=
=p1E/
-----END PGP SIGNATURE-----

[Marek Marczykowski-Górecki]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Fri, Sep 04, 2015 at 12:27:25PM -0400, Eric Shelton wrote:

> I just tried that, and even did 'sudo xl mem-max <appvm> 4096'.
> However, 'xl top' never shows a change in the memory above 600MB for
> the appvm, even after starting up both chrome and firefox. Instead, I
> get the impression (due to some slowdown) that the appvm is hitting
> swap.

You can confirm that ballooning was requested checking xenstore:
xenstore-ls -f|grep target

Then in the VM you can confirm that balloon driver get the request:
cat /sys/devices/system/xen_memory/xen_memory0/target_kb

Also note that dynamic change of mem-max is not something that works, at
least not straightforward. But using mem-set should just work.

> So, I don't get the impression that ballooning is really taking place
> (assuming that ballooning would show in increase in memory vai 'xl
> list' or 'xl top'). Perhaps something from how qmemman interacts
> across the VMs is interfering with ordinary Xen ballooning behavior?

I assume you've stopped qmemman first, right? Otherwise it would
rollback your changes almost immediately...

PS restoring cc.

- --
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJV6cjGAAoJENuP0xzK19cs81wH/jWyDc6bRbrBVTFZf4bPEStc
YD/3gCzSXb2VYcUKbYAuVwM8M39jML0GHfx3uQf+yZA7TrZOanDfiMBrmOcRtxSD
BUAoXqbFXTgiBbOtHWuhJqBp6vINz+Ac0IVOJ1BaoOCqz4EwOny8l35hoqB4V+fm
14iOYjw05UWpMFFLmTWNEcRcn94T8x/wPgIcgo9NzcBL01O3dfMzQz+IWPr0Vmaf
Sqzo2+DV0hSqPAhb2GzOKv1QoTwVp2K8TmaKQw739nXQhsRPHO9T4KWuOyJ3LMHB
9jVr4UYtFDP5TS9GRdAL4L6exARBkAdL+Q/O84dtkr98g6YY/CWVT3/5VXSt7T0=
=WE3s
-----END PGP SIGNATURE-----

[Rob Kinyon]

On Monday, June 1, 2015 at 10:36:41 PM UTC, Joanna Rutkowska wrote:

I've been experiencing strange Chrome browser crashes under Linux PVs (Qubes
AppVMs) for some time now. This is tracked by the following ticket:

https://github.com/QubesOS/qubes-issues/issues/1003

This happens to me on two different machines, one running Qubes R2 with Xen 4.1
and on Fedora 20 template, the other R3.0 with FC21 template. I could not
reproduce the same bug when Chrome runs on baremetal... It's really hard for me
to believe I'm the only one experiencing this issue. Am I?

Thanks,
joanna. 

 

I replicated this problem using the Debian-8 template. Since I was running this from the CLI, I got the following:

--------

[1327:1327:0909/025028:ERROR:sandbox_linux.cc(345)] InitializeSandbox() called with multiple threads in process gpu-process

xdg-settings: unknown desktop environment

--2015-09-09 02:53:44--  https://clients2.google.com/cr/report

Resolving clients2.google.com (clients2.google.com)... 216.58.216.110, 2607:f8b0:4009:80b::200e

Connecting to clients2.google.com (clients2.google.com)|216.58.216.110|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: unspecified [text/html]

Saving to: ‘/dev/fd/3’

     0K        

 Crash dump id:  4cf44ec82d717b49 

--------

Does this help? Could this be related to the fact that the GPU isn't available through Xen?

 

[Eric Shelton]

For what it is worth, this crash appears to have gone away for me on a Skylake system running a pre-R3.1-rc1 build that I compiled a couple of days ago.  Google Maps does not crash Chrome the way it used to.  I am running:

Xen 4.6.0
Linux 4.1.13
Chrome 48.0.2564.10 dev (64-bit)

I don't know which of the above (if any of them) fixed things.

Eric