Please help me diagnose this crazy VMWare/FreeBSD 8.x crash

[Mark Felder]

Alright guys, I'm at the end of my rope here. For those that haven't seen
my previous emails here's the (not so) quick breakdown:

Overview:

FreeBSD ?? - 7.4 never crash
FreeBSD 8.0 - 8.2 crashes
FreeBSD 8-STABLE, 8.3, and 9.0 are untested (Sorry, not possible in our
production at this time, and we were hoping we could base some stuff on
8.3 for long term stability...)
ESXi: Confirmed ESXi 4.0 - 5.0 has this problem. Haven't tested on others.


History:

Over the course of the last 2 years we've been banging our heads on the
wall. VMWare is done debugging this. They claim it's not a VMWare issue.
They can't identify what the heck happens. We had a glimmer of hope with
ESXi 5.0 fixing it because we never saw any crashes in the handful of
deployments, but our dreams were crushed today -- two days before an
outage to begin migration to ESXi 5.0 -- when a customer's ESXi 5.0 server
and FreeBSD 8.2 guest crashed.


Crash Details:

The keyboard/mouse usually stops responding for input on the console;
normally we can't type in a username or password. However, we can switch
VTs.

If there's a shell on the console and we can type, we can only run things
in memory. Any time we try to access the disk it will hang indefinitely.

The server still has network access. We can ping it without issue. SSH of
course kicks you out because it can't do any I/O.

If we were to serve a lightweight http server off a memory backed
filesystem I'm confident it would run just fine as long as it wasn't
logging or anything.

On ESXi you see that there is a CPU spike of 100% that goes on
indefinitely. No idea what the FreeBSD OS itself thinks it is doing
because we can't run top during the crash.

This crash can affect a server and happen multiple times a week. It can
also not show up for 180 days or more. But it does happen. The server can
be 100% idle and crash. We have servers that do more I/O than the ones
that crash could ever attempt to do and these don't crash at all.
Completely inexplicable.


Things we've looked into:

Nothing about the installed software matters. We've tried cross
referencing the crashed servers by the programs they run but the base OS
is the only common denominator due to the wide variety of servers it has
affected.

Storage doesn't matter. We've tried different iSCSI SANs, we've tried
different switches, we've tried local datastores on the ESXi servers
themselves.

HP servers, Dell servers -- doesn't seem to matter either. (All with
latest firmwares, BIOSes, etc)

VMWare gave us a ton of debugging tasks, and we've given them gigabytes of
debugging info and data; they can't find anything.

VMWare tools -- with, without, using open-vm-tools makes no difference. I
think we've done a fair job ruling out VMWare.


I think we've finally found enough data that this is definitely something
in the FreeBSD world. I'm going to begin prepping some of the known crashy
servers with more debugging. Any suggestions on what I should build the
kernel with? They never do a proper panic, but I definitely want to at
least *try* to get into the debugger the next time it crashes. And when it
crashes, what the heck should I be running? I've never played with the KDB
before...


Thank you for any suggestions and help you can give me....
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

[Adrian Chadd]

Hi,

* have you filed a PR?
* is the crash easily reproducable?
* are you able to boot some ramdisk-only FreeBSD-8.2 images (eg create
a ramdisk image using nanobsd?) and do some stress testing inside
that?

It sounds like you've established it's a storage issue, or at least
interrupt handling for storage issue. So I'd definitely try the
ramdisk-only boot and thrash it using lighttpd/httperf or something.
If that survives fine, I'd look at trying to establish whether there's
something wrong in the disk driver(s) freebsd is using. I'm not that
cluey on ESXi, but there may be some PIC/APIC/ACPI change between 7.x
and 8.0 which has caused this to surface.

2c,


Adrian

[Doug Barton]

On 3/28/2012 1:59 PM, Mark Felder wrote:
> FreeBSD 8-STABLE, 8.3, and 9.0 are untested

As much as I'm sensitive to your production requirements, realistically
it's not likely that you'll get a helpful result without testing a newer
version. 8.2 came out over a year ago, many many things have changed
since then.

Doug

[Mark Felder]

On Wed, 28 Mar 2012 18:31:38 -0500, Adrian Chadd <adr...@freebsd.org>
wrote:

> * have you filed a PR?

No

> * is the crash easily reproducable?

Unfortunately not. It's totally random. Some servers will "get the bug"
and crash daily, some will crash weekly, some might seem to be fine but 3
months later hit this crash.

> * are you able to boot some ramdisk-only FreeBSD-8.2 images (eg create
> a ramdisk image using nanobsd?) and do some stress testing inside
> that?

That's a plan I'd like to execute but my free time for building that
environment is rather short at the moment :(

> I'm not that
> cluey on ESXi, but there may be some PIC/APIC/ACPI change between 7.x
> and 8.0 which has caused this to surface.

Was there a setting to revert ACPI behavior from 8.x to 7.x? I thought I
read about that at one point.... or perhaps this was something available
back in the dev cycle when 8 was -CURRENT. *shrug* I know 9.0 and onward
has even more ACPI changes so assuming it truly is an ACPI bug I guess we
could cross our fingers and hope that the bug has mysteriously vanished?

[Mark Felder]

On Thu, 29 Mar 2012 02:36:49 -0500, Doug Barton <do...@freebsd.org> wrote:

> As much as I'm sensitive to your production requirements, realistically
> it's not likely that you'll get a helpful result without testing a newer
> version. 8.2 came out over a year ago, many many things have changed
> since then.

The sad part is that VMWare's "supported FreeBSD versions" are a joke, and
we've been trying to keep VMWare happy by only running "supported
versions". I honestly don't think they even test. It's so stupid.

[Mark Felder]

Alright, new data. It happened to crash about 10 minutes after I came in
this morning and I ran some stuff in the DDB. I have no idea what
information is useful, but perhaps someone will see something out of the
ordinary?


http://feld.me/freebsd/esx_crash/


Thanks...

[Joe Greco]

> Hi,
>
> * have you filed a PR?
> * is the crash easily reproducable?
> * are you able to boot some ramdisk-only FreeBSD-8.2 images (eg create
> a ramdisk image using nanobsd?) and do some stress testing inside
> that?
>
> It sounds like you've established it's a storage issue, or at least
> interrupt handling for storage issue. So I'd definitely try the
> ramdisk-only boot and thrash it using lighttpd/httperf or something.
> If that survives fine, I'd look at trying to establish whether there's
> something wrong in the disk driver(s) freebsd is using. I'm not that
> cluey on ESXi, but there may be some PIC/APIC/ACPI change between 7.x
> and 8.0 which has caused this to surface.

We've seen this. Or something that seems really like it.

We run dozens of FreeBSD VM's, many of which are 8.mumble. We have a
scripted build environment dating back many years, so generally servers
come out in a fairly reproducible form.

After several months of smooth running, we had need to shuffle some
things around, and migrated some servers to a different datastore.
Suddenly, one particular VM, our corp Jabber server, started randomly
disconnecting people every morning. Some inspection showed that the
machine was running, but disk I/O in the VM was freezing up.
Subsequent inspection suggested that it was happening during the
periodic daily, though we never managed to get it to happen by manually
forcing periodic daily, so that's only a theory. Given that several
times it appeared that one of the find commands was running, I was
guessing that something in the thin provisioned disk image for the
system had gone bad, but reading the entire disk with dd didn't cause
a hang, running the periodic daily by hand didn't cause a hang, etc.

Migrating the VM to a different host and datastore did not fix the
issue. Migrating the VM from an Opteron to a Xeon host with all the
latest ESXi 4 patches also didn't make any difference. Migrating the
disk image from thin to full seemed to fix it, but I only gave it a
day or two, then decided there were other good reasons to reload the
VM, so I nuked the VM, which, of course, fixed it.

In the meantime, a dozen other similar VM's alongside it run just
fine. My conclusion was that it was something specific that had gone
awry in the virtual machine, probably in the disk image, but I could
not identify it without significant digging that I had no particular
reason or inclination to do; since it appeared to be a VMware problem,
the "reload it and be done with it" seemed the quickest path to
resolution.

That having been said, if anyone has any brilliant ideas about what
would constitute useful further steps to isolate this, I can look at
recovering the faulty VM from backup and seeing if it still exhibits
the problem.

.. JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

[Hans Petter Selasky]

On Thursday 29 March 2012 15:42:42 Joe Greco wrote:
> > Hi,

Do both 32- and 64-bit versions of FreeBSD crash?

--HPS

[Joe Greco]

> On 3/28/2012 1:59 PM, Mark Felder wrote:
> > FreeBSD 8-STABLE, 8.3, and 9.0 are untested
>

> As much as I'm sensitive to your production requirements, realistically
> it's not likely that you'll get a helpful result without testing a newer
> version. 8.2 came out over a year ago, many many things have changed
> since then.
>

> Doug

So you're saying that he should have been using 8.3-RELEASE, then.

If you'll kindly go over to http://www.freebsd.org and look under
"Latest Releases", please note that "8.2" is a production release.
If you don't want it to be a production release, then find a way
to make it so, but please don't snipe at people who are using the
code that the FreeBSD project has indicated is a current production
offering.

There are many good reasons not to run arbitrary snapshots on your
production gear. It's unrealistic to expect people to run non-
RELEASE non-production code on their production gear. We can have
that discussion if you don't understand that, drop me a note off-
list and I'll be happy to explain it.

Otherwise, you've told him to run a "newer version," of which NONE
IS AVAILABLE, unless you're thinking 9.0, but FreeBSD has a rather
catastrophic history of "point zero" releases, and most clueful
admins won't run those in production without carefully measuring
the risks and benefits. So you've basically told him to run a
newer version without any such version being realistically
available.

WTF?

You want people not to use releases that "came out over a year
ago"? The generally sensible solution to that is to release
RELEASEs more than once every fourteen or fifteen months.

.. JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

[Mark Felder]

On Thu, 29 Mar 2012 09:58:16 -0500, Hans Petter Selasky <hsel...@c2i.net>
wrote:

> Do both 32- and 64-bit versions of FreeBSD crash?

Correct, we see both i386 and amd64 flavors crash in the same way.

[Eduardo Morras]

At 16:03 29/03/2012, you wrote:
>Alright, new data. It happened to crash about 10 minutes after I came in
>this morning and I ran some stuff in the DDB. I have no idea what
>information is useful, but perhaps someone will see something out of the
>ordinary?
>
>
>http://feld.me/freebsd/esx_crash/

Don't know about ESXi but on others VM Managers i can change the
chipset emulation from ICH10 to ICH4. Can you change it to an older
chipset too?

[Joe Greco]

> On Thursday 29 March 2012 15:42:42 Joe Greco wrote:
> > > Hi,
>
> Do both 32- and 64-bit versions of FreeBSD crash?

We've only seen it happen on one virtual machine. That was a 32-bit
version. And it's not so much a crash as it is a "disk I/O hang".

The fact that it was happening regularly to that one VM, while a
bunch of other similar VM's were running alongside it without any
incident, along with the problem moving with the VM as it is moved
from host to host and from Opteron to Xeon, strongly points at
something being wrong with the VM itself. Our systems are built
mostly by script; I rebuilt the VM a few months ago and the
problem vanished. The rebuilt system "should" have been virtually
identical to the original. I never actually compared them though.

My working theory was that something bad had happened to the VM
during a migration from one datastore to another. We have a really
slow-writing iSCSI server that it had been migrated onto for a little
bit, which was where the problem first appeared, I believe. At
first I thought it was the nightly cron jobs just exceeding the iSCSI
server's capacity to cope, so we migrated the VM onto a host with
local datastores, and it remained broken thereafter.

So my conclusion was that it seemed likely that somehow VMware's
thin provisioned disk image had gotten fouled up, and under some
unknown use case, it could be teased into locking up further I/O
on the VM. I wasn't able to prove it. I tried a read-dd of the
entire disk - passed, flying. I tried several things to duplicate
the nightly periodic tasks where it seemed so prone to locking up.
They all ran fine. But if I left the machine run, it'd do it
again eventually.

I explained it at the time to one of my VMware friends:

> But here's where it gets weird. Three times, now, one VM - our Jabber
> server - has gone wonky in the wee early AM hours. Disk I/O on the VM
> just locks up. You can type at the console until it does I/O, so you
> can put in "root" at the login: prompt but never get a pw prompt. My
> systems all run "top" from /etc/ttys and I can see that a whole bunch
> of processes are stopped in "getblk". It's like the iSCSI disk has gone
> away, except it hasn't, since the other VM's are all happily churning
> away, on the same datastore, on the same VMware host.

http://www.sol.net/tmp/freebsd/freebsd-esxi-lockup.gif

> Now it's *possible* that the problem actually happens after the 3AM cron
> run (note slight CPU/memory drop) but the Jabber implosion actually
> happens around 0530, see drop in memory%. But the root problem at the
> VM level seems to be that disk I/O has frozen. I can't tell for sure when
> that happens. All three instances are similar to this.
>
> I can't explain this or figure out how to debug it. Since it's locked up
> right now, thought I'd ping you for ideas before resetting it.

Now that was actually before we migrated it back to local datastore,
but when we did, the problem remained, suggesting that whatever has
happened to the VM, it is contained within the VM's vmdk or other
files.

.. JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

[Mark Felder]

On Thu, 29 Mar 2012 10:31:24 -0500, Eduardo Morras <nec...@retena.com>
wrote:

>
> Don't know about ESXi but on others VM Managers i can change the chipset
> emulation from ICH10 to ICH4. Can you change it to an older chipset too?

Unfortunately there's no setting in the GUI for that but I'll keep looking
to see if there's a hidden option -- perhaps in the VM's config file.

[Mark Felder]

On Thu, 29 Mar 2012 10:55:36 -0500, Hans Petter Selasky <hsel...@c2i.net>
wrote:
>
> It almost sounds like the lost interrupt issue I've seen with USB EHCI
> devices, though disk I/O should have a retry timeout?
>
> What does "wmstat -i" output?
>
> --HPS


Here's a server that has a week uptime and is due for a crash any hour now:

root@server:/# vmstat -i
interrupt total rate
irq1: atkbd0 34 0
irq6: fdc0 9 0
irq15: ata1 34 0
irq16: em1 778061 1
irq17: mpt0 19217711 31
irq18: em0 283674769 460
cpu0: timer 246571507 400
Total 550242125 892

[Mark Felder]

On Thu, 29 Mar 2012 10:49:30 -0500, Joe Greco <jgr...@ns.sol.net> wrote:

> I explained it at the time to one of my VMware friends:

This is 100% identical to what we see, Joe! And we're so unlucky that we
have this happen on probably a dozen servers, but a handful are the really
bad ones. We've rebuilt them from scratch many times with no improvement.

[Jim Bryant]

This sounds just like a race condition that happens under Windows 7 on
this laptop. The race condition, as far as I can tell involves heavy
disk access and heavy network access, and usually leaves the drive light
on, while all activity monitors (alldisk, allcpu, allnetwork) are still
active, although on this laptop disk takes priority, and network slows
to a crawl. occasionally, the mouse will stop working, along with
everything else, but usually not. keyboard is lower priority, and
doesn't do anything.

You might want to check with mickeysoft, this might just be their
problem. This sounds so freaking similar to the issue I get, and I
think it's a race condition (shared interrupts??).

This laptop is a Compaq Presario C300 series, with the 945GM chipset and
a T7600 Core2 Duo CPU, with 3G of RAM.

[Alan Cox]

On Thu, Mar 29, 2012 at 11:27 AM, Mark Felder <fe...@feld.me> wrote:

> On Thu, 29 Mar 2012 10:55:36 -0500, Hans Petter Selasky <hsel...@c2i.net>
> wrote:
>
>>
>> It almost sounds like the lost interrupt issue I've seen with USB EHCI
>> devices, though disk I/O should have a retry timeout?
>>
>> What does "wmstat -i" output?
>>
>> --HPS
>>
>
>
> Here's a server that has a week uptime and is due for a crash any hour now:
>
> root@server:/# vmstat -i
> interrupt total rate
> irq1: atkbd0 34 0
> irq6: fdc0 9 0
> irq15: ata1 34 0
> irq16: em1 778061 1
> irq17: mpt0 19217711 31
> irq18: em0 283674769 460
> cpu0: timer 246571507 400
> Total 550242125 892
>
>

Not so long ago, VMware implemented a clever scheme for reducing the
overhead of virtualized interrupts that must be delivered by at least some
(if not all) of their emulated storage controllers:

http://static.usenix.org/events/atc11/tech/techAbstracts.html#Ahmad

Perhaps, there is a bad interaction between this scheme and FreeBSD's mpt
driver.

Alan

[Mark Atkinson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/29/2012 07:03, Mark Felder wrote:
> Alright, new data. It happened to crash about 10 minutes after I
> came in this morning and I ran some stuff in the DDB. I have no
> idea what information is useful, but perhaps someone will see
> something out of the ordinary?
>
>
> http://feld.me/freebsd/esx_crash/

If this is an interrupt problem with disk i/o, then you might want to
look into (DDB(4))

show intr
show intrcount

maybe

show allrman
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk90lloACgkQrDN5kXnx8yaCZACbBamQksNyWC26PUsOn5N9LJLV
ql0AoJwYCFDfXhCpZIN735V9qg0VepFf
=fCLN
-----END PGP SIGNATURE-----

[Mark Felder]

On Thu, 29 Mar 2012 12:24:30 -0500, <je...@seibercom.net> wrote:

>
> I just started reading this tread, but I am wondering if I missed
> something here. What does this have to do with "Windows 7"?

I emailed him off-list but I'm guessing he thought this was on VMWare
Workstation or another product that would virtualize FreeBSD on top of
Windows as the host OS.

[Mark Felder]

On Thu, 29 Mar 2012 11:53:02 -0500, Alan Cox <alan....@gmail.com> wrote:

>
> Not so long ago, VMware implemented a clever scheme for reducing the
> overhead of virtualized interrupts that must be delivered by at least
> some
> (if not all) of their emulated storage controllers:
>
> http://static.usenix.org/events/atc11/tech/techAbstracts.html#Ahmad
>
> Perhaps, there is a bad interaction between this scheme and FreeBSD's mpt
> driver.
>
> Alan

If we assume mpt is the culprit how can I go about diagnosing this more
accurately? Is there something I should be looking for in vmstat -i? Too
many interrupts? Not enough? Rate too high or too low? Or is this
something that is much harder to track down because we're dealing with
emulated hardware?

If any BSD devs are interested in access to our environment I think we
could comply. I might even be able to get authorization to give you an
account on the most crash-prone server which doesn't have any sensitive
customer data on it. I think at this point we'd even be willing to pay
someone to look at a server in this state just so we (and hopefully
others) can benefit.... and hopefully we end up with a more reliable
FreeBSD-on-VMWare for everyone.

I know Doug mentioned running newer OS versions and that is definitely
tempting but because it's not 100% reproducible on demand it's hard to
prove it fixes it without waiting 6 months. We're fighting internally here
with "trust 9.0 fixes it" vs "jump back to 7.4 because we KNOW it doesn't
happen there". Having someone look at this and say "oh, yes, that's a
deficiency in mpt that appears to be fixed in the newer driver that was
MFC'd to 8-STABLE and you'll find in 8.3-RELEASE and 9.0-RELEASE" would be
more comforting.

Thanks to everyone for their time on this!

[Joe Greco]

> On Thursday 29 March 2012 17:49:30 Joe Greco wrote:
> > > On Thursday 29 March 2012 15:42:42 Joe Greco wrote:
> > > > > Hi,
> > >
> > > Do both 32- and 64-bit versions of FreeBSD crash?
> >
> > We've only seen it happen on one virtual machine. That was a 32-bit
> > version. And it's not so much a crash as it is a "disk I/O hang".
>

> It almost sounds like the lost interrupt issue I've seen with USB EHCI
> devices, though disk I/O should have a retry timeout?

That doesn't seem to fit. Why would a perfectly functional VM suddenly
develop this problem when given a slow underlying datastore (fits so far)
but then the problem *remains* when returned to a fast local datastore,
even on a different host and architecture? And why wouldn't the other
VM's running alongside develop the same problem?

> What does "wmstat -i" output?

No idea, we reloaded the VM months ago.

.. JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

[Dieter BSD]

> FreeBSD ?? - 7.4 never crash
> FreeBSD 8.0 - 8.2 crashes

Obvious short term workaround is to run production on 7.4 (assuming you can)
until you figure out what is wrong with 8.x.

What filesystem(s) are you running? UFS? ZFS? other?

> started randomly disconnecting people every morning

Due to timeouts? Something might be keeping interrupts disabled
too long.

> there were other good reasons to reload the
> VM, so I nuked the VM, which, of course, fixed it.

> I can look at recovering the faulty VM from backup

Sounds like corruption.  Can you compare the bad VM against a good
one?  If you find corruption, the question then becomes what is causing
the corruption?  Sounds like the same thing is getting corrupted
every time, rather than something at random.

Sounds like the corruption is causing a deadlock in something
common, like the buffer cache, or filesystem, or...

Is it possible to have root be a ramdisk?  This might give you
access to the utilities, depending on where the problem is.

I have vague memories that the sticky bit used to lock a program in
memory, but sticky(8) indicates that this is no longer the case.
Is there a way to lock a program in memory? (So that it will be available
when the system can't do disk i/o.)  If not, you could keep some
windows open with things like top and systat -vmstat running.

Some of the utilities have options to look at a disk file rather than
the live system, if you can get a core dump (swap to NFS?).

[Mark Felder]

On Thu, 29 Mar 2012 12:53:49 -0500, Dieter BSD <diet...@engineer.com>
wrote:

>> FreeBSD ?? - 7.4 never crash
>> FreeBSD 8.0 - 8.2 crashes
>
> Obvious short term workaround is to run production on 7.4 (assuming you
> can)
> until you figure out what is wrong with 8.x.

We're moving our most critical servers to 7.4 this weekend. After finally
seeing the crash on ESXi 5.0 we have run out of viable options and we
haven't had time to test FreeBSD 9.0 or 8.3 pre-release code.

>
> What filesystem(s) are you running? UFS? ZFS? other?
>

Only UFS.

[Adam Vande More]

On Thu, Mar 29, 2012 at 1:22 PM, Mark Felder <fe...@feld.me> wrote:

>
> If we assume mpt is the culprit
>

Doesn't VMWare offer different types of emulated disk controllers? If so,
that might be the easiest way to narrow the field. Another thing maybe to
try would be to backport the mpt

Also, it's not VMWare's place to claim "not our problem" when you are
paying for support. If this doesn't happen on bare metal, it's a VMWare
issue, or they need to demonstrate it's not their issue. At least that
would be the expectation I have.

There is also a comment on this post indicating someone else with the issue
and who has received unofficial vmware feedback.

http://www.hailang.me/tech/virtual/freebsd-vmware-esx-a-weird-error-with-san-storage/

And then there is this one with similar symptoms and a workaround:

http://forums.freebsd.org/showthread.php?t=27899

--
Adam Vande More

[Mark Felder]

On Thu, 29 Mar 2012 15:53:52 -0500, Adam Vande More
<amvan...@gmail.com> wrote:

>
> Doesn't VMWare offer different types of emulated disk controllers? If
> so,
> that might be the easiest way to narrow the field. Another thing maybe
> to
> try would be to backport the mpt

Yes, they offer Paravirtual (not applicable for FreeBSD), LSI Parallel
(default option), LSI SAS, and Buslogic (not available for 64bit).

Both LSI SAS and LSI Parallel use the mpt driver.

>
> Also, it's not VMWare's place to claim "not our problem" when you are
> paying for support. If this doesn't happen on bare metal, it's a VMWare
> issue, or they need to demonstrate it's not their issue. At least that
> would be the expectation I have.

You're right, but we've thrown a ton of money at their support and had
direct phone access to their engineers. The best we can get out of them is
"no indication this is a VMWare problem". It's easy for them to blow
people off when they're as big as they've grown to be.

> There is also a comment on this post indicating someone else with the
> issue
> and who has received unofficial vmware feedback.
>
> http://www.hailang.me/tech/virtual/freebsd-vmware-esx-a-weird-error-with-san-storage/

I found that post ages ago and that's me, "mf", as the only person to
comment on it. Unfortunately our problem does not align with what he's
describing.

>
> And then there is this one with similar symptoms and a workaround:
>
> http://forums.freebsd.org/showthread.php?t=27899
>

I'm now investigating those loader.conf options. I have my crashy machine
set to use them on next boot so we'll see if it crashes now that I'm using
LSI SAS emulated controller. If it still crashes, we'll see what happens
after that with those loader.conf options enabled.

[Doug Barton]

On 3/29/2012 7:01 AM, Joe Greco wrote:
>> On 3/28/2012 1:59 PM, Mark Felder wrote:
>>> FreeBSD 8-STABLE, 8.3, and 9.0 are untested
>>
>> As much as I'm sensitive to your production requirements, realistically
>> it's not likely that you'll get a helpful result without testing a newer
>> version. 8.2 came out over a year ago, many many things have changed
>> since then.
>>
>> Doug
>
> So you're saying that he should have been using 8.3-RELEASE, then.

That isn't what I said at all, sorry if I wasn't clear. The OP mentioned
9.0-RELEASE, and in the context of his message (which I snipped) he
mentioned 8-stable. That's what I was referring to.

> If you'll kindly go over to http://www.freebsd.org and look under
> "Latest Releases", please note that "8.2" is a production release.
> If you don't want it to be a production release, then find a way
> to make it so, but please don't snipe at people who are using the
> code that the FreeBSD project has indicated is a current production
> offering.
>
> There are many good reasons not to run arbitrary snapshots on your
> production gear. It's unrealistic to expect people to run non-
> RELEASE non-production code on their production gear. We can have
> that discussion if you don't understand that, drop me a note off-
> list and I'll be happy to explain it.

I can see that you're upset about something, sorry if my message caused
you additional stress. I actually understand the realities of production
environments quite well, and believe it or not I agree with some of your
frustration about how we handle support for our "supported" releases.
We've had various public threads about these issues, which have sparked
some quite-lively private discussions amongst our committers, and I'm
hoping that once the long-overdue 8.3-RELEASE is out we'll be able to
buckle down and start putting some of those ideas into action.

Meanwhile, this is still a volunteer project, and as a result sometimes
the best way to get attention to a problem is to verify that it hasn't
already been fixed. You've been around more than long enough to
understand this Joe. We can spend time arguing about what *should* be
(actually we can't ...) but my point was in trying to help the OP get
the most/best help the fastest way possible.

Doug

[Joe Greco]

> > And then there is this one with similar symptoms and a workaround:
> >

> > http://forums.freebsd.org/showthread.php?t=3D27899

>
> I'm now investigating those loader.conf options. I have my crashy machine
> set to use them on next boot so we'll see if it crashes now that I'm using
> LSI SAS emulated controller. If it still crashes, we'll see what happens
> after that with those loader.conf options enabled.

Um, if I may, that's something completely different.

VMDirectPath, or PCIe passthru, is making a hardware device on a VMware
host available directly to a guest. It'll take your LSI controller, in
the example cited, and make it unavailable to VMware ESXi, and present
it instead inside the guest environment. You do this when you have an
app whose performance would suffer greatly when made to operate through
the indirection that a VM naturally lives in; for example, it is quite
common for FreeNAS users to pass a disk controller through to a VM guest
in order to allow a virtualized FreeNAS instance to directly manage the
physical disks.

In that case, there are some issues with ESXi and interrupt delivery to
the guest VM; virtualization doesn't actually get rid of the possibility
of ESXi problems, since the hypervisor is still ultimately involved. It
is certainly possible that there's some common issue involving interrupt
delivery somehow, but I wouldn't get my hopes up.

It also doesn't explain the experience here, where one VM basically
crapped out but only after a migration - and then stayed crapped out.
It would be interesting to hear about your datastore, how busy it is,
what technology, whether you're using thin, etc. I just have this real
strong feeling that it's some sort of corruption with the vmfs3 and thin
provisioned disk format, but it'd be interesting to know if that's
totally off-track.

.. JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

[Mark Felder]

On Thu, 29 Mar 2012 19:27:31 -0500, Joe Greco <jgr...@ns.sol.net> wrote:

> It also doesn't explain the experience here, where one VM basically
> crapped out but only after a migration - and then stayed crapped out.
> It would be interesting to hear about your datastore, how busy it is,
> what technology, whether you're using thin, etc. I just have this real
> strong feeling that it's some sort of corruption with the vmfs3 and thin
> provisioned disk format, but it'd be interesting to know if that's
> totally off-track.

We've ruled out SAN, but we haven't ruled out VMFS. Even FreeBSD Guests on
standalone ESXi servers with no SAN exhibit this crash.

For the record, we only use thick provisioning and if it was corruption
I'm not sure what layer the corruption could be at. The crashy servers
show no abnormalities when I run either `freebsd-update IPS` or
`pkg_libchk` to confirm checksums of all installed programs. Now the other
data on there... it's not exactly verified, but our backups via rsnapshot
seem to prove there is no issue there or we'd have lots of new files each
run.

[Adrian Chadd]

Again, it's starting to sound like an interrupt handling issue which
may or may not be limited to the storage device.

You'll have to engage someone who knows those device drivers and
likely have them add some debugging to the driver which can be easily
flipped on (via binaries in a ramdisk - very important if you can't
run sysctl because your disk IO has locked up!) to see what the
current state of things.

It's likely that the BSD mpt(4) and other storage drivers, and/or our
interrupt handling code, is just slightly different enough to confuse
the snot out of VMWare. I'd first look at the obvious - (eg, if you've
just stopped receiving interrupts, even if new IO is scheduled). I'd
also ask VMware if they have any tools that they can run on a VM to
get the state of the internal emulated driver. For example, register
dumps of the device to see if it's in a hung state, register dumps of
the PIC/APIC to see what state they're in, etc.

Maybe pull in someone like ixsystems and see if they can help debug
this kind of stuff? If you're paying vmware for support, you could
pull them into things with ixsystems and see if the two of them can
help you sort this out?

Thanks,



Adrian

[Joe Greco]

> On 3/29/2012 7:01 AM, Joe Greco wrote:
> >> On 3/28/2012 1:59 PM, Mark Felder wrote:
> >>> FreeBSD 8-STABLE, 8.3, and 9.0 are untested
> >>
> >> As much as I'm sensitive to your production requirements, realistically
> >> it's not likely that you'll get a helpful result without testing a newer
> >> version. 8.2 came out over a year ago, many many things have changed
> >> since then.
> >>
> >> Doug
> >
> > So you're saying that he should have been using 8.3-RELEASE, then.
>
> That isn't what I said at all, sorry if I wasn't clear. The OP mentioned
> 9.0-RELEASE, and in the context of his message (which I snipped) he
> mentioned 8-stable. That's what I was referring to.

And since both the poster and I made it clear that this doesn't seem
to be a case of "it fails reliably on a machine of your choosing",
just installing random other versions and hoping that it's going to
cause a fail ... well, let's just say that doesn't make a whole lot
of sense. Or at least it's a recipe for a hell of a lot of busywork,
busywork not guaranteed to return any sort of useful result.

What you suggest is a fine solution for "My ASUS Sempron box fails
when I do X!" -- in such a case, "Try a different version of FreeBSD"
makes lots of sense. The problem is, in a virtualization environment,
theoretically the virtual hosts are all the same sort of hardware
(modulo any specific configuration changes of course), so when someone
presents a problem that afflicts only a percentage of their VM's, it
is important to keep in mind that you are not interacting with physical
hardware, and that reinstalling an OS on a "problem" VM...?

Well, let's just say I like real hardware better for many reasons.

In the meantime, it's unrealistic to tell people to use supported
releases, to wait fifteen months between releases, and then to criticize
people complaining about problems with a supported release for "using
old code".

.. JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

[Joe Greco]

> On Thu, 29 Mar 2012 19:27:31 -0500, Joe Greco <jgr...@ns.sol.net> wrote:
>
> > It also doesn't explain the experience here, where one VM basically
> > crapped out but only after a migration - and then stayed crapped out.
> > It would be interesting to hear about your datastore, how busy it is,
> > what technology, whether you're using thin, etc. I just have this real
> > strong feeling that it's some sort of corruption with the vmfs3 and thin
> > provisioned disk format, but it'd be interesting to know if that's
> > totally off-track.
>
> We've ruled out SAN, but we haven't ruled out VMFS. Even FreeBSD Guests on
> standalone ESXi servers with no SAN exhibit this crash.
>
> For the record, we only use thick provisioning and if it was corruption
> I'm not sure what layer the corruption could be at. The crashy servers
> show no abnormalities when I run either `freebsd-update IPS` or
> `pkg_libchk` to confirm checksums of all installed programs. Now the other
> data on there... it's not exactly verified, but our backups via rsnapshot
> seem to prove there is no issue there or we'd have lots of new files each
> run.

Crud, there goes part of my theory :-)

Have you migrated these hosts, or were they installed in-place and
never moved?

fwiw the apparent integrity of things on the VM is consistent with
our experience too.

.. JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

[Mark Felder]

On Fri, 30 Mar 2012 09:44:47 -0500, Joe Greco <jgr...@ns.sol.net> wrote:

> Have you migrated these hosts, or were they installed in-place and
> never moved?
> fwiw the apparent integrity of things on the VM is consistent with
> our experience too.

VMMotion and StorageVMMotion does not seem to affect the stability. Even
deleting the VM, rebuilding from scratch, re-installing all packages from
scratch, copying over a few configs and then copying in any other data
(perhaps website data) does not solve the problem.

However, our two most notorious for crashing happen to be webservers. We
moved one to hardware. We simply rsync'd the exact data (entire OS and
files) off the VM onto hardware, made a few config changes (fstab, network
interface) and it's been running for 4+ months now with zero crashes.

I don't think it's corruption :/

[Joe Greco]

> On Fri, 30 Mar 2012 09:44:47 -0500, Joe Greco <jgr...@ns.sol.net> wrote:
> > Have you migrated these hosts, or were they installed in-place and
> > never moved?
> > fwiw the apparent integrity of things on the VM is consistent with
> > our experience too.
>
> VMMotion and StorageVMMotion does not seem to affect the stability. Even
> deleting the VM, rebuilding from scratch, re-installing all packages from
> scratch, copying over a few configs and then copying in any other data
> (perhaps website data) does not solve the problem.

On the same vmdk files? "Deleting the VM" makes it sound like not.

> However, our two most notorious for crashing happen to be webservers. We
> moved one to hardware. We simply rsync'd the exact data (entire OS and
> files) off the VM onto hardware, made a few config changes (fstab, network
> interface) and it's been running for 4+ months now with zero crashes.

That part doesn't shock me at all.

> I don't think it's corruption :/

Then it is hard to see what it is.

>From my perspective:

We had a perfectly functional, nearly zero-traffic VM, since Jabber
traffic averages no more than a few messages per hour. It was working
for quite some time.

We moved it from a local datastore to an iSCSI datastore that ended up
getting periodically crushed by the load (in particular during the
periodic daily load imposed by a bunch of VM's all running at once).
At this point, this one VM started hanging on I/O. We expected that
this would clear up upon return to a host with a local datastore. It
did not.

This ended up as a broken VM, one that would hang up overnite, maybe
not every night, but several times a week at least.

But wait:

None of the other VM's, even the VM's that had been abused in this
horribly insensitive manner of being placed on intolerably slow iSCSI,
developed this condition.

There are dozens of other VM's running on these hosts, alongside the
one that was exhibiting this behaviour.

The VM continued to exhibit this behaviour even after having been moved
onto a different ESXi platform and architecture (Opteron->Xeon).

For the problem to "follow" the VM in this manner, and afflict *only*
the one VM, strongly suggests that it is something that is contained
within the VM files that constitute this VM. That is consistent with
the observation that the problem arose at a point where the VM is
known to have had all those files moved from one location to a dodgy
location.

That's why I believe the evidence points to corruption of some sort.

Of course, your case makes this all interesting.

.. JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

[Mark Felder]

On Fri, 30 Mar 2012 11:53:10 -0500, Joe Greco <jgr...@ns.sol.net> wrote:

> On the same vmdk files? "Deleting the VM" makes it sound like not.

Fresh new VMDK files every time, and always thick provisioned.

> None of the other VM's, even the VM's that had been abused in this
> horribly insensitive manner of being placed on intolerably slow iSCSI,
> developed this condition.

We've seen similar results. Baffling how VMs you know are worse off never
develop this condition.

> There are dozens of other VM's running on these hosts, alongside the
> one that was exhibiting this behaviour.
> The VM continued to exhibit this behaviour even after having been moved
> onto a different ESXi platform and architecture (Opteron->Xeon).
> For the problem to "follow" the VM in this manner, and afflict *only*
> the one VM, strongly suggests that it is something that is contained
> within the VM files that constitute this VM. That is consistent with
> the observation that the problem arose at a point where the VM is
> known to have had all those files moved from one location to a dodgy
> location.

We were hoping that was the explanation as well, but rebuilding the VM
entirely from scratch on a new host and seeing the crash come back was a
big blow to morale. :(

> That's why I believe the evidence points to corruption of some sort.
> Of course, your case makes this all interesting.

For the last year I've been convinced it's something hidden inside ESXi's
I/O virtualization layer that happens to trigger on only certain VMs.

[Jim Bryant]

Mark Felder wrote:
> On Thu, 29 Mar 2012 12:24:30 -0500, <je...@seibercom.net> wrote:
>
>>
>> I just started reading this tread, but I am wondering if I missed
>> something here. What does this have to do with "Windows 7"?
>
> I emailed him off-list but I'm guessing he thought this was on VMWare
> Workstation or another product that would virtualize FreeBSD on top of
> Windows as the host OS.
> _______________________________________________
>

Correct...My bad.

jim

[Dieter BSD]

> Subsequent inspection suggested that it was happening during the
> periodic daily, though we never managed to get it to happen by manually
> forcing periodic daily, so that's only a theory.

Perhaps due to a bunch of VMs all running periodic daily at the same time?

> We had a perfectly functional, nearly zero-traffic VM, since Jabber
> traffic averages no more than a few messages per hour.  It was working
> for quite some time.
>
> We moved it from a local datastore to an iSCSI datastore that ended up
> getting periodically crushed by the load (in particular during the
> periodic daily load imposed by a bunch of VM's all running at once).
> At this point, this one VM started hanging on I/O.  We expected that
> this would clear up upon return to a host with a local datastore.  It
> did not.
>
> This ended up as a broken VM, one that would hang up overnite, maybe
> not every night, but several times a week at least.

..

> For the problem to "follow" the VM in this manner, and afflict *only*
> the one VM, strongly suggests that it is something that is contained
> within the VM files that constitute this VM.  That is consistent with
> the observation that the problem arose at a point where the VM is
> known to have had all those files moved from one location to a dodgy
> location.
>

> That's why I believe the evidence points to corruption of some sort.

Compare a backup of the VM before it broke to a backup of the same VM
after it broke.  Hopefully the haystack of insignificant differences
isn't too large, or the significant difference needle might be a
lot of "fun" to find.

[Adrian Chadd]

There's no guarantee that upgarding a VM or rebooting it won't change
the config of said VM. Don't forget to diff the vm config file..


Adrian

[Doug Barton]

On 03/30/2012 07:41, Joe Greco wrote:
>> On 3/29/2012 7:01 AM, Joe Greco wrote:
>>>> On 3/28/2012 1:59 PM, Mark Felder wrote:
>>>>> FreeBSD 8-STABLE, 8.3, and 9.0 are untested
>>>>
>>>> As much as I'm sensitive to your production requirements, realistically
>>>> it's not likely that you'll get a helpful result without testing a newer
>>>> version. 8.2 came out over a year ago, many many things have changed
>>>> since then.
>>>>
>>>> Doug
>>>
>>> So you're saying that he should have been using 8.3-RELEASE, then.
>>
>> That isn't what I said at all, sorry if I wasn't clear. The OP mentioned
>> 9.0-RELEASE, and in the context of his message (which I snipped) he
>> mentioned 8-stable. That's what I was referring to.
>
> And since both the poster and I made it clear that this doesn't seem
> to be a case of "it fails reliably on a machine of your choosing",
> just installing random other versions and hoping that it's going to
> cause a fail ... well, let's just say that doesn't make a whole lot
> of sense. Or at least it's a recipe for a hell of a lot of busywork,
> busywork not guaranteed to return any sort of useful result.

And since you can't reliably reproduce the problem, how do you expect us
to? I understand that these sorts of bugs are difficult/annoying, etc.
Been there, done that.

> In the meantime, it's unrealistic to tell people to use supported
> releases, to wait fifteen months between releases, and then to criticize
> people complaining about problems with a supported release for "using
> old code".

Just to be clear, I didn't criticize anyone. And I share your
frustration with the length of the 8.3 release cycle. I really wish I
had a better answer, but as much as you and I may wish that things were
different, "Try a newer version" is the best answer we have atm.

Doug

--

This .signature sanitized for your protection

[Joe Greco]

> On 03/30/2012 07:41, Joe Greco wrote:
> >> On 3/29/2012 7:01 AM, Joe Greco wrote:
> >>>> On 3/28/2012 1:59 PM, Mark Felder wrote:
> >>>>> FreeBSD 8-STABLE, 8.3, and 9.0 are untested
> >>>>
> >>>> As much as I'm sensitive to your production requirements, realistically
> >>>> it's not likely that you'll get a helpful result without testing a newer
> >>>> version. 8.2 came out over a year ago, many many things have changed
> >>>> since then.
> >>>>
> >>>> Doug
> >>>
> >>> So you're saying that he should have been using 8.3-RELEASE, then.
> >>
> >> That isn't what I said at all, sorry if I wasn't clear. The OP mentioned
> >> 9.0-RELEASE, and in the context of his message (which I snipped) he
> >> mentioned 8-stable. That's what I was referring to.
> >
> > And since both the poster and I made it clear that this doesn't seem
> > to be a case of "it fails reliably on a machine of your choosing",
> > just installing random other versions and hoping that it's going to
> > cause a fail ... well, let's just say that doesn't make a whole lot
> > of sense. Or at least it's a recipe for a hell of a lot of busywork,
> > busywork not guaranteed to return any sort of useful result.
>
> And since you can't reliably reproduce the problem, how do you expect us
> to? I understand that these sorts of bugs are difficult/annoying, etc.
> Been there, done that.

Nobody expected you to. We're trying to figure out any commonalities
that might exist; these may serve to help shed light on where the
problem lies.

The interesting thing is that I took it and looked at it and came to a
conclusion that might have been wrong, though I think the trail of
reasoning I used was itself reasonable, given my exceedingly small (one
example of problem) sample size.

Mark's able to actually *reproduce* the problem on separate installs
and with circumstances that are at least somewhat different than what
my theory involved, though it is not quite possible to rule out some
sort of corruption.

Since I have to *assume* that many sites run some sort of FreeBSD on
their VMware gear, given that VMware actually lists it as a supported
version and VMware generally does things "for profit", I am still kind
of of the opinion that this is some sort of corruption bug, one that I
triggered inadvertently, but one that Mark's environment reproduces
rather more frequently. That just seems so unlikely, but more unlikely
things have come to pass, so I'm holding onto it as my working theory ;-)

I still plan to try to recover my broken VM from backups at some point
if time permits.

But in short, to answer your question: I don't *care* if you can
reproduce the problem. As a user, you can't win. If you don't report
a problem, you get criticized. If you report a problem but can't figure
out how to reproduce it, you get criticized. If you can reproduce it
but you don't submit a workaround, you get criticized. If you submit a
workaround but you don't submit a patch, you get criticized. If you
submit a patch but it's not in the preferred format, you get criticized.

Hm.

.. JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

[Doug Barton]

On 4/2/2012 11:43 AM, Joe Greco wrote:
> As a user, you can't win. If you don't report
> a problem, you get criticized. If you report a problem but can't figure
> out how to reproduce it, you get criticized. If you can reproduce it
> but you don't submit a workaround, you get criticized. If you submit a
> workaround but you don't submit a patch, you get criticized. If you
> submit a patch but it's not in the preferred format, you get criticized.

I'm still not sure what you're taking as criticism. Nothing I've said
was intended that way, nor should it be read that way. If you feel that
you've been criticized by others in the manner you describe, you should
probably take it up with them on an individual basis.

My experience of FreeBSD as a community is that we tend to be both less
critical of users, and less tolerant of it. Especially when compared to
other communities that I've interacted with.

Doug

--

This .signature sanitized for your protection

[Joe Greco]

> On 4/2/2012 11:43 AM, Joe Greco wrote:
> > As a user, you can't win. If you don't report
> > a problem, you get criticized. If you report a problem but can't figure
> > out how to reproduce it, you get criticized. If you can reproduce it
> > but you don't submit a workaround, you get criticized. If you submit a
> > workaround but you don't submit a patch, you get criticized. If you
> > submit a patch but it's not in the preferred format, you get criticized.
>
> I'm still not sure what you're taking as criticism. Nothing I've said
> was intended that way, nor should it be read that way. If you feel that
> you've been criticized by others in the manner you describe, you should
> probably take it up with them on an individual basis.

It certainly seemed to me that

> As much as I'm sensitive to your production requirements, realistically
> it's not likely that you'll get a helpful result without testing a newer
> version. 8.2 came out over a year ago, many many things have changed
> since then.

was an unwarranted criticism for reasons that I've already explained.

Or perhaps this:

> And since you can't reliably reproduce the problem, how do you expect us
> to? I understand that these sorts of bugs are difficult/annoying, etc.
> Been there, done that.

Which would appear to be suggesting that either (or possibly both):

1) The reporter has a duty to be able to "reliably reproduce the problem"
prior to reporting, and/or

2) That there was some unreasonable expectation on the reporter's part
that you were expected to reproduce it.

I consider 1) to be ridiculous, as long as the reporter is reasonably
willing to work to resolve the issue, that should certainly be good
enough, and he's certainly been interactive enough to _my_ comments,
and 2) seems to be nowhere in sight in the reporter's comments, but
is nonetheless present in your response.

Please respect Reply-to. Thanks.

.. JG
--
Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net
"We call it the 'one bite at the apple' rule. Give me one chance [and] then I
won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN)
With 24 million small businesses in the US alone, that's way too many apples.

[Mark Felder]

Guys,

The crash on my machine with debugging has evaded me for a few days. I'm
still looking for further suggestions of things I should grab from the DDB
when it happens again.

Thanks for the help everyone!

[Doug Barton]

On 4/2/2012 3:59 PM, Joe Greco wrote:
>> On 4/2/2012 11:43 AM, Joe Greco wrote:
>>> As a user, you can't win. If you don't report
>>> a problem, you get criticized. If you report a problem but can't figure
>>> out how to reproduce it, you get criticized. If you can reproduce it
>>> but you don't submit a workaround, you get criticized. If you submit a
>>> workaround but you don't submit a patch, you get criticized. If you
>>> submit a patch but it's not in the preferred format, you get criticized.
>>
>> I'm still not sure what you're taking as criticism. Nothing I've said
>> was intended that way, nor should it be read that way. If you feel that
>> you've been criticized by others in the manner you describe, you should
>> probably take it up with them on an individual basis.
>
> It certainly seemed to me that
>
>> As much as I'm sensitive to your production requirements, realistically
>> it's not likely that you'll get a helpful result without testing a newer
>> version. 8.2 came out over a year ago, many many things have changed
>> since then.
>
> was an unwarranted criticism for reasons that I've already explained.

Everything in that paragraph is a fact. If you feel criticized when
people state facts, I'm not sure how much I can help you.

Please note, I didn't say, "You're an idiot for running old stuff." I
even explicitly stated that I understood *why* the OP is running an old
version. Nevertheless, the facts are what they are. The only way we can
deal rationally with the world is to acknowledge reality and deal with
it. Wishing it were otherwise isn't really useful. :)

> Or perhaps this:
>
>> And since you can't reliably reproduce the problem, how do you expect us
>> to? I understand that these sorts of bugs are difficult/annoying, etc.
>> Been there, done that.
>
> Which would appear to be suggesting that either (or possibly both):
>
> 1) The reporter has a duty to be able to "reliably reproduce the problem"
> prior to reporting, and/or
>
> 2) That there was some unreasonable expectation on the reporter's part
> that you were expected to reproduce it.

Quite the contrary, I was responding to your implication that there is
some other answer that we should be able to give the OP, other than "Try
a newer version."

Various people have chimed in on the thread, all have offered
suggestions, none of which (AFAICS) have helped. I continue to maintain
that the best course of action for the OP would be to try the latest
8-stable.

And BTW, there are (at least) 2 reasons for that. First, the bug may
actually be fixed. But second, we're in the middle of a release cycle
for 8.3 right now. If the bug persists in the latest code it will be
easier to get the right eyes onto the problem. That benefits both the OP
and the community at large.

Doug

[Mark Felder]

Quick update:

I have received word last night that this crash has been consistently
happening to someone on FreeBSD 9 and they're looking for more ideas. I
changed the following 41 days ago:

- Video memory to "auto" if it wasn't already
- SCSI controller changed from LSI Logic Parallel to LSI Logic SAS

It uses the same driver (da) but so far has been holding steady for us. As
far as the video memory -- many of our servers somehow had video memory
set to 1MB which seemed strange; newer builds of FreeBSD on ESXi do not
show this option. Perhaps there was a build of ESXi in the past that had a
different setting for video memory when you selected FreeBSD?

Another change people might want to do as suggested to us by VMWare
Support:

- Change CPU/MMU Virtualization to the bottom option -- "Use Intel
VTx/AMD-V for instruction set virtualization and Intel EPT / AMD RVI for
MMU virtualization"

Supposedly there are autodetection issues here with some OSes -- they
named some BSDs and Netware.


I'll provide further updates if anything changes, but this seems to be
working well so far. We won't begin to trust it until we can hit at least
100 days of uptime, though. Unfortunately I was hoping to upgrade these
servers to 8.3 before then...

[Mark Felder]

OK guys I've been talking with another user who can recreate this crash
and the last bit of information we've learned seems to be leaning towards
interrupts/IRQ issues like someone (bz@ perhaps?) suggested.

I'm still trying to test this myself, but the other user was able to
recreate my crash pretty much on demand. The fix was to not use the first
NIC in the VM because it will always share an IRQ with mpt0. Once mpt0 is
on its own the crash does not seem to be reproducible anymore.

Before:

$ vmstat -i
interrupt total rate
irq1: atkbd0 378 0
irq6: fdc0 9 0
irq15: ata1 34 0
irq16: em1 687237 1
irq18: em0 mpt0 319094024 539
cpu0: timer 236770821 400
Total 556552503 940

After:

$ vmstat -i
interrupt total rate
irq1: atkbd0 38 0
irq6: fdc0 9 0
irq15: ata1 34 0
irq16: em1 2811 15
irq17: em2 5 0
cpu0: timer 71013 398
irq256: mpt0 12163 68
Total 86073 483


Is there any other way we can make mpt0 get its own dedicated IRQ without
having to do this? The problem is that it causes us to have to make
rc.conf changes, pf.conf changes, and who knows what other software could
be on these machines that is trying to bind to a specific NIC...


Thanks!

[Andrew Boyer]

You could try switching mpt to MSI. MSI interrupts are never shared. Add this to /boot/device.hints:

> hint.mpt.0.msi_enable="1"


-Andrew

--------------------------------------------------
Andrew Boyer abo...@averesystems.com

[Adrian Chadd]

Hi,

can you please, -please- file a PR? And place all of the above
information in it so we don't lose it?

If this is indeed the problem then I really think we should root cause
why the driver and/or interrupt handling code is getting angry with
the shared interrupt.

I'd also appreciate it if you and the other people who can reproduce
this could work with the em/mpt driver people and root cause why this
is going. I think having FreeBSD on vmware work stable out of the box
without these kinds of tweaks is the way to go - who knows what else
is lurking here..

I'm very very glad you've persisted with this and if I had them, I'd
send you a "FreeBSD persistent bug reporter!" t-shirt.

Thanks,


Adrian

[Mark Felder]

On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd <adr...@freebsd.org>
wrote:

> Hi,
>
> can you please, -please- file a PR? And place all of the above
> information in it so we don't lose it?
>

I'd be glad to post a PR and assist in helping to get it permanently
fixed. I certainly don't want this data to get lost and honestly our
business uses FreeBSD on VMWare so much that we really need a permanent
fix as much as anyone else :-)

The reason I've hesitated to post a PR so far is that I didn't have any
truly useful or concrete evidence of where the problem lies. After Dane
Foster contacted me and told me he could recreate the crash on demand with
his workload it was easier to narrow things down. The suggestion that it
was an interrupts issue (by possibly Bjoern Zeeb?) and Dane's discovery
that his crashes ceased when em0 and mpt0 share an IRQ, but em0 is
completely unused was starting to prove there is some strong evidence here
in favor of the interrupts issue.

Dane, what's the status on your end? Has your fix still been successful?
Is it also stable if you simply set hint.mpt.0.msi_enable="1" ?


Thanks!

[dane foster]

Hey all,

On 25/05/2012, at 1:47 AM, Mark Felder wrote:

> On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd <adr...@freebsd.org> wrote:
>
>> Hi,
>>
>> can you please, -please- file a PR? And place all of the above
>> information in it so we don't lose it?
>>
>
> I'd be glad to post a PR and assist in helping to get it permanently fixed. I certainly don't want this data to get lost and honestly our business uses FreeBSD on VMWare so much that we really need a permanent fix as much as anyone else :-)
>
> The reason I've hesitated to post a PR so far is that I didn't have any truly useful or concrete evidence of where the problem lies. After Dane Foster contacted me and told me he could recreate the crash on demand with his workload it was easier to narrow things down. The suggestion that it was an interrupts issue (by possibly Bjoern Zeeb?) and Dane's discovery that his crashes ceased when em0 and mpt0 share an IRQ, but em0 is completely unused was starting to prove there is some strong evidence here in favor of the interrupts issue.
>
> Dane, what's the status on your end? Has your fix still been successful? Is it also stable if you simply set hint.mpt.0.msi_enable="1" ?
>

The situation I've got that's stable now is:

hw.pci.enable_msi="0"
hw.pci.enable_msix="0"

in /boot/loader.conf

and:

samael:~:% vmstat -i [ 6:31PM]
interrupt total rate
irq1: atkbd0 6 0
irq18: em0 mpt0 3061100 15
irq19: em1 6891706 35
cpu0: timer 166383735 868
cpu1: timer 166382123 868
cpu3: timer 166382123 868
cpu2: timer 166382121 868
Total 675482914 3525

Not using em0. This works for 8 (FreeBSD samael.slush.ca 8.3-STABLE FreeBSD 8.3-STABLE #1: Mon May 7 11:51:03 NZST 2012 ro...@samael.slush.ca:/usr/obj/usr/src/sys/DENE amd64).

Neither of those settings on their own seem to stop it from happening.

The 9 box I've tried this on still hangs almost every time i run handbrake, no matter whether MSI/MSIX is enabled, or I have separate IRQs for mpt0 and em0/1

I can cause the hang mostly on demand, but not quite sure what information to provide from the hung system. If somebody can let me know what they need, including root access, I can make that happen.

Cheers,

Dane

[Bjoern A. Zeeb]

On 24. May 2012, at 13:47 , Mark Felder wrote:

> On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd <adr...@freebsd.org> wrote:
>
>> Hi,
>>
>> can you please, -please- file a PR? And place all of the above
>> information in it so we don't lose it?
>>
>
> I'd be glad to post a PR and assist in helping to get it permanently fixed. I certainly don't want this data to get lost and honestly our business uses FreeBSD on VMWare so much that we really need a permanent fix as much as anyone else :-)
>
> The reason I've hesitated to post a PR so far is that I didn't have any truly useful or concrete evidence of where the problem lies. After Dane Foster contacted me and told me he could recreate the crash on demand with his workload it was easier to narrow things down. The suggestion that it was an interrupts issue (by possibly Bjoern Zeeb?)

Just for the public archives. Interrupts wasn't me. I might have mentioned disabling cdrom and fdc as good as possible but everything else I cannot remember...

> and Dane's discovery that his crashes ceased when em0 and mpt0 share an IRQ, but em0 is completely unused was starting to prove there is some strong evidence here in favor of the interrupts issue.
>
> Dane, what's the status on your end? Has your fix still been successful? Is it also stable if you simply set hint.mpt.0.msi_enable="1" ?

--
Bjoern A. Zeeb You have to have visions!
It does not matter how good you are. It matters what good you do!

[Adrian Chadd]

Hi,

You guys now absolutely, positively have enough information for a PR.

It's still not clear whether it's a device/interrupt layer issue in
FreeBSD, or whether vmware is doing something wrong with how it
implements shared interrupts, or a bit of both..

Adrian

[John Baldwin]

On Thursday, May 24, 2012 9:47:46 am Mark Felder wrote:
> On Wed, 23 May 2012 17:30:40 -0500, Adrian Chadd <adr...@freebsd.org>
> wrote:
>
> > Hi,
> >
> > can you please, -please- file a PR? And place all of the above
> > information in it so we don't lose it?
> >
>
> I'd be glad to post a PR and assist in helping to get it permanently
> fixed. I certainly don't want this data to get lost and honestly our
> business uses FreeBSD on VMWare so much that we really need a permanent
> fix as much as anyone else :-)
>
> The reason I've hesitated to post a PR so far is that I didn't have any
> truly useful or concrete evidence of where the problem lies. After Dane
> Foster contacted me and told me he could recreate the crash on demand with
> his workload it was easier to narrow things down. The suggestion that it
> was an interrupts issue (by possibly Bjoern Zeeb?) and Dane's discovery
> that his crashes ceased when em0 and mpt0 share an IRQ, but em0 is
> completely unused was starting to prove there is some strong evidence here
> in favor of the interrupts issue.
>
> Dane, what's the status on your end? Has your fix still been successful?
> Is it also stable if you simply set hint.mpt.0.msi_enable="1" ?

Hmm, so the set of ps output you have from DDB shows a lot of runnable
processes and swi6 (Giant taskq) as the only running thread (all consistent
with your hang). (And that is from your Ctrl-Alt-Esc)

Do you only have one CPU in this VM? If not, do you know which threads
the other CPUs were running (e.g. do you have ps7.png, etc.)?

--
John Baldwin

[Mark Felder]

On Wed, 30 May 2012 10:06:13 -0500, John Baldwin <j...@freebsd.org> wrote:

>
> Do you only have one CPU in this VM? If not, do you know which threads
> the other CPUs were running (e.g. do you have ps7.png, etc.)?

correct, only one CPU in the VM

[John Baldwin]

On Wednesday, May 30, 2012 12:07:50 pm Mark Felder wrote:
> On Wed, 30 May 2012 10:06:13 -0500, John Baldwin <j...@freebsd.org> wrote:
>
> >
> > Do you only have one CPU in this VM? If not, do you know which threads
> > the other CPUs were running (e.g. do you have ps7.png, etc.)?
>
> correct, only one CPU in the VM

Humm, can you test it with 2 CPUs?

--
John Baldwin

[John Baldwin]

On Thursday, May 31, 2012 11:11:11 am Mark Felder wrote:
> So when this hang happens, there never is a real panic. It just sits in a
> state which I describe as like being in a deadlock. How would I go about
> getting a crashdump if it never panics? Is it possible to do the dump over
> a network or something because I don't believe it can write through the
> controller at all.

You can break into ddb and run 'call doadump'. It should use polled IO, so
there is a slight chance of it working.

> Also, thank you for the KTR_SCHED tip. This is the type of info I was
> looking for. Unfortunately I've only ever seen this crash once on a kernel
> with debugging enabled. The machine which is currently prepared to do this
> work used to crash a few times a week and now it has 70 days uptime...
> however, it is an example of a machine with mpt0 and em0 sharing an IRQ so
> I might be able to trigger it using Dane's method.
>
> $ vmstat -i
> interrupt total rate
> irq1: atkbd0 392 0
> irq6: fdc0 9 0
> irq14: ata0 34 0
> irq18: em0 mpt0 1189748491 218
> cpu0: timer 2174263198 400
> Total 3364012124 619
>
>
> I'm doing my best to get you guys the info you need, but this is one heck
> of a Heisenbug...

Thanks.

[Mark Felder]

Hi guys I'm excitedly posting this from my phone. Good news for you guys, bad news for us -- we were building HA storage on vmware for a client and can now replicate the crash on demand. I'll be posting details when I get home to my PC tonight, but this hopefully is enough to replicate the crash for any curious followers:

ESXi 5
9 or 9-STABLE
HAST
1 cpu is fine
1GB of ram
UFS SUJ on HAST device
No special loader.conf, sysctl, etc
No need for VMWare tools
Run Bonnie++ on the HAST device

We can get the crash to happen on the first run of bonnie++ right now. I'll post the exact specs and precise command run in the PR. We found an old post from 2004 when we looked up the process state obtained from CTRL+T -- flswai -- which describes the symptoms nearly perfectly.

http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2004-02/0250.html

Hopefully this gets us closer to a fix...

[John Baldwin]

On Wednesday, June 06, 2012 9:34:02 pm Mark Felder wrote:
> Hi guys I'm excitedly posting this from my phone. Good news for you guys,
bad news for us -- we were building HA storage on vmware for a client and can
now replicate the crash on demand. I'll be posting details when I get home to
my PC tonight, but this hopefully is enough to replicate the crash for any
curious followers:
>
> ESXi 5
> 9 or 9-STABLE
> HAST
> 1 cpu is fine
> 1GB of ram
> UFS SUJ on HAST device
> No special loader.conf, sysctl, etc
> No need for VMWare tools
> Run Bonnie++ on the HAST device
>
> We can get the crash to happen on the first run of bonnie++ right now. I'll
post the exact specs and precise command run in the PR. We found an old post
from 2004 when we looked up the process state obtained from CTRL+T -- flswai
-- which describes the symptoms nearly perfectly.
>
> http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2004-02/0250.html
>
> Hopefully this gets us closer to a fix...

Sorry, I just now saw this. :(

Are you still seeing this, and if so can you get a crashdump? Also, I'm
curious if you only see this with SUJ or if plain UFS+SU works fine?

--
John Baldwin

[Mark Felder]

On Wed, 12 Sep 2012 14:20:26 -0500, John Baldwin <j...@freebsd.org> wrote:

> Are you still seeing this, and if so can you get a crashdump? Also, I'm
> curious if you only see this with SUJ or if plain UFS+SU works fine?

The crash on demand right now is producable on 8.x and 9.x, so SUJ isn't a
requirement. Also, there is no crashdump available. The OS just hangs and
stops taking input. There's no panic or coredump or anything of the like.
You just have to nuke the VM and re-boot it back up.

And for the record we can't reproduce this crash in Xen...

[Andriy Gapon]

on 13/09/2012 17:50 Mark Felder said the following:

> On Wed, 12 Sep 2012 14:20:26 -0500, John Baldwin <j...@freebsd.org> wrote:
>
>> Are you still seeing this, and if so can you get a crashdump? Also, I'm
>> curious if you only see this with SUJ or if plain UFS+SU works fine?
>
> The crash on demand right now is producable on 8.x and 9.x, so SUJ isn't a
> requirement. Also, there is no crashdump available. The OS just hangs and stops
> taking input. There's no panic or coredump or anything of the like. You just have
> to nuke the VM and re-boot it back up.
>
> And for the record we can't reproduce this crash in Xen...

Just curious - does VMWare provide a remote debugger support (gdb stub)?

--
Andriy Gapon

[Mark Felder]

On Thu, 13 Sep 2012 10:11:28 -0500, Andriy Gapon <a...@freebsd.org> wrote:

> Just curious - does VMWare provide a remote debugger support (gdb stub)?

I'm not aware of one. What I have been able to successfully do is break
into the debugger during the hang but the info I've posted so far has not
been relevant to anyone. I'm hoping someone on the core team will
eventually be able to follow my guide and figure out what went wrong.

[Kurt Lidl]

On Thu, Sep 13, 2012 at 11:14:49AM -0500, Mark Felder wrote:
> On Thu, 13 Sep 2012 10:11:28 -0500, Andriy Gapon <a...@freebsd.org> wrote:
>
> > Just curious - does VMWare provide a remote debugger support (gdb stub)?
>
> I'm not aware of one. What I have been able to successfully do is break
> into the debugger during the hang but the info I've posted so far has not
> been relevant to anyone. I'm hoping someone on the core team will
> eventually be able to follow my guide and figure out what went wrong.

Isn't this what you want?

http://stackframe.blogspot.com/2007/04/debugging-linux-kernels-with.html

-Kurt

[Mark Felder]

On Thu, 13 Sep 2012 11:28:15 -0500, Kurt Lidl <li...@pix.net> wrote:

> Isn't this what you want?
>
> http://stackframe.blogspot.com/2007/04/debugging-linux-kernels-with.html
>
> -Kurt

Interesting -- it looks like that's an option on ESX as well. The only
question is: what do I do with that? It's going to give me the debugging
entire VM, not the kernel inside. Without being a VMWare developer I
imagine its data will be a bit useless :-(

[Andriy Gapon]

on 13/09/2012 22:57 Mark Felder said the following:

> On Thu, 13 Sep 2012 11:28:15 -0500, Kurt Lidl <li...@pix.net> wrote:
>
>> Isn't this what you want?
>>
>> http://stackframe.blogspot.com/2007/04/debugging-linux-kernels-with.html
>>
>> -Kurt
>
> Interesting -- it looks like that's an option on ESX as well. The only question
> is: what do I do with that? It's going to give me the debugging entire VM, not
> the kernel inside. Without being a VMWare developer I imagine its data will be a
> bit useless :-(

No, gdb stub is for debugging what is running inside the VM.
E.g. look here for an example of how to do that with qemu:
http://andriygapon.wikispaces.com/QemuSetup
VMWare with gdb stub enabled should not be any different.

--
Andriy Gapon

[Mark Saad]

---

On Sep 13, 2012, at 5:13 PM, Andriy Gapon <a...@FreeBSD.org> wrote:

> on 13/09/2012 22:57 Mark Felder said the following:
>> On Thu, 13 Sep 2012 11:28:15 -0500, Kurt Lidl <li...@pix.net> wrote:
>>
>>> Isn't this what you want?
>>>
>>> http://stackframe.blogspot.com/2007/04/debugging-linux-kernels-with.html
>>>
>>> -Kurt
>>
>> Interesting -- it looks like that's an option on ESX as well. The only question
>> is: what do I do with that? It's going to give me the debugging entire VM, not
>> the kernel inside. Without being a VMWare developer I imagine its data will be a
>> bit useless :-(
>
> No, gdb stub is for debugging what is running inside the VM.
> E.g. look here for an example of how to do that with qemu:
> http://andriygapon.wikispaces.com/QemuSetup
> VMWare with gdb stub enabled should not be any different.
>
> --
> Andriy Gapon
> _____________________________

Mark did you try changing the time counter choice sysctl from hpet to Acpi-safe/fast or tsc ? In esx 4.1 an newer using hpet timers causes issues for bsd and linux vms in some cases .

The sysctl is kern.timecounter.choice . This sounds like another VMware issues I have seen and read about . See this post http://forums.freebsd.org/archive/index.php/t-32104.html

---
Mark saad | mark...@longcount.org

[Mark Felder]

Changing timer source has not been tested. It doesn't crash in 7.x, so did something timer related change in 8.x?

[Mark Saad]

---

On Sep 13, 2012, at 7:45 PM, Mark Felder <fe...@feld.me> wrote:

> Changing timer source has not been tested. It doesn't crash in 7.x, so did something timer related change in 8.x?
>

Mark
Yes the time counter choice priority changed , in 8 favoring higher precision hardware like hpet over acpi-fast/safe . I am not sure why or when thus was done; or if this is a side effect of another change. Interestingly centos/rhel/suse has made similar changes and VMware has odd issues with them as well.

Can you boot up a 7 environment and get us the value sysctl kern.timecounter . Then get that from an a 8 and 9 environment . Then if the 7 environment uses a different time counter can you try using that value on your crashing setup and report back what the result is .

---
Mark saad | mark...@longcount.org

[Mark Felder]

Hi Mark,

Here's the output of our VMs running on ESXi 4.1u1

FreeBSD 7.4:
# sysctl kern.timecounter.choice
kern.timecounter.choice: TSC(800) ACPI-safe(850) i8254(0) dummy(-1000000)
# sysctl kern.timecounter.hardware
kern.timecounter.hardware: ACPI-safe

FreeBSD 8.3:
# sysctl kern.timecounter.choice
kern.timecounter.choice: TSC(800) ACPI-safe(850) i8254(0) dummy(-1000000)
# sysctl kern.timecounter.hardware
kern.timecounter.hardware: ACPI-safe

FreeBSD 9.0:
# sysctl kern.timecounter.choice
kern.timecounter.choice: TSC(1000) i8254(0) ACPI-fast(900) dummy(-1000000)
# sysctl kern.timecounter.hardware
kern.timecounter.hardware: TSC


Note that both 8.3 and 9.0 crash, while 7.4 does not.

[Mark Saad]

On Sep 14, 2012, at 8:48 AM, Mark Felder <fe...@feld.me> wrote:

> Hi Mark,
>
> Here's the output of our VMs running on ESXi 4.1u1
>
> FreeBSD 7.4:
> # sysctl kern.timecounter.choice
> kern.timecounter.choice: TSC(800) ACPI-safe(850) i8254(0) dummy(-1000000)
> # sysctl kern.timecounter.hardware
> kern.timecounter.hardware: ACPI-safe
>
> FreeBSD 8.3:
> # sysctl kern.timecounter.choice
> kern.timecounter.choice: TSC(800) ACPI-safe(850) i8254(0) dummy(-1000000)
> # sysctl kern.timecounter.hardware
> kern.timecounter.hardware: ACPI-safe
>
> FreeBSD 9.0:
> # sysctl kern.timecounter.choice
> kern.timecounter.choice: TSC(1000) i8254(0) ACPI-fast(900) dummy(-1000000)
> # sysctl kern.timecounter.hardware
> kern.timecounter.hardware: TSC
>
>
> Note that both 8.3 and 9.0 crash, while 7.4 does not.

How do you have suj on 8.3 ? Are you using a patch ? Also can you retest 9 with the following sysctl

kern.timecounter.hardware=Acpi-fast

Also in esxi what setup options do you have for the vm's ?

Lastly do you have esxi setup to talk to a ntp server ? If so can you confirm that it's working ? I mean the esxi host not the vm .


---
Mark saad | mark...@longcount.org

[Mark Felder]

On Fri, 14 Sep 2012 20:37:40 -0500, Mark Saad <none...@longcount.org>
wrote:

> How do you have suj on 8.3 ? Are you using a patch ?

I don't have suj on 8.3

> Also can you retest 9 with the following
> sysctlkern.timecounter.hardware=Acpi-fast

Yes, I'll attempt that as soon as possible. We're under a tight deadline
to migrate critical resources off of VMWare now so I don't know how soon I
can test.

> Also in esxi what setup options do you have for the vm's ?

I'm not sure what ones I have off the top of my head, but VMWare support
has previously poured over ever option to make sure nothing was
misconfigured.

> Lastly do you have esxi setup to talk to a ntp server ? If so can you
> confirm that it's working ? I mean the esxi host not the vm .

Yes, and yes. We've got a reliable NTP infrastructure at work and ESXi is
definitely using it.

[Mark Saad]

On Sep 15, 2012, at 11:36 AM, Mark Felder <fe...@feld.me> wrote:

> On Fri, 14 Sep 2012 20:37:40 -0500, Mark Saad <none...@longcount.org> wrote:
>
>> How do you have suj on 8.3 ? Are you using a patch ?
>
> I don't have suj on 8.3

I misread the prior emails

>
>> Also can you retest 9 with the following sysctlkern.timecounter.hardware=Acpi-fast
>
> Yes, I'll attempt that as soon as possible. We're under a tight deadline to migrate critical resources off of VMWare now so I don't know how soon I can test.
>
>> Also in esxi what setup options do you have for the vm's ?
>
> I'm not sure what ones I have off the top of my head, but VMWare support has previously poured over ever option to make sure nothing was misconfigured.
>

It's not that I doubt that , in my experience their support is not equipped to answer questions that don't start with " in my windows vm I have this issue ..."

>> Lastly do you have esxi setup to talk to a ntp server ? If so can you confirm that it's working ? I mean the esxi host not the vm .
>
> Yes, and yes. We've got a reliable NTP infrastructure at work and ESXi is definitely using it.

Just checking this can cause odd issues .

What hardware is the esxi host server ? What are you using for the vm disks? Is the storage pool on local disks , iscsi , fiber channel , or nfs .

---
Mark saad | mark...@longcount.org

[John Baldwin]

On Thursday, September 13, 2012 12:14:49 pm Mark Felder wrote:
> On Thu, 13 Sep 2012 10:11:28 -0500, Andriy Gapon <a...@freebsd.org> wrote:
>
> > Just curious - does VMWare provide a remote debugger support (gdb stub)?
>
> I'm not aware of one. What I have been able to successfully do is break
> into the debugger during the hang but the info I've posted so far has not
> been relevant to anyone. I'm hoping someone on the core team will
> eventually be able to follow my guide and figure out what went wrong.

So the last e-mail I sent before this week I asked if you could get a
crashdump from DDB? The "flswai" case you pointed to was a lock deadlock, and
having a crashdump would be really helpful for figuring out which threads were
deadlocked.

Barring a crashdump, capturing 'ps' output from DDB would be a good first step
(you can sanitize process names if you need to). However, a crashdump that
you can use kgdb on will make debugging this significantly easier.

--
John Baldwin

[guy.h...@gmail.com]

On Wednesday, June 6, 2012 8:36:04 PM UTC-5, Mark Felder wrote:
> Hi guys I'm excitedly posting this from my phone. Good news for you guys, bad news for us -- we were building HA storage on vmware for a client and can now replicate the crash on demand. I'll be posting details when I get home to my PC tonight, but this hopefully is enough to replicate the crash for any curious followers:
>
>
>
> ESXi 5
>
> 9 or 9-STABLE
>
> HAST
>
> 1 cpu is fine
>
> 1GB of ram
>
> UFS SUJ on HAST device
>
> No special loader.conf, sysctl, etc
>
> No need for VMWare tools
>
> Run Bonnie++ on the HAST device
>
>
>
> We can get the crash to happen on the first run of bonnie++ right now. I'll post the exact specs and precise command run in the PR. We found an old post from 2004 when we looked up the process state obtained from CTRL+T -- flswai -- which describes the symptoms nearly perfectly.
>
>
>
> http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2004-02/0250.html
>
>
>
> Hopefully this gets us closer to a fix...

Is this a crash or a hang? Over the past couple of weeks, I've been working with a FreeBSD 9.1RC1 system under VMware ESXi 5.0 with a 64GB UFS root FS and 2TB ZFS filesystem mounted via a virtual LSI SAS interface. Sometimes during heavy I/O load (rsync from other servers) on the ZFS FS, this shows up in /var/log/messages:

Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 5 ee 60 16 0 1 0 0
Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error
Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy
Sep 21 02:14:55 backups kernel: (da1:mpt0:0:1:0): Retrying command
Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 42 51 0 1 0 0
Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error
Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy
Sep 21 02:18:44 backups kernel: (da1:mpt0:0:1:0): Retrying command
Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 64 51 0 1 0 0
Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error
Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy
Sep 21 02:18:48 backups kernel: (da1:mpt0:0:1:0): Retrying command
Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 3 ef 66 51 0 1 0 0
Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error
Sep 21 02:18:49 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy
...
Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): WRITE(10). CDB: 2a 0 41 f3 94 99 0 1 0 0
Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): CAM status: SCSI Status Error
Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): SCSI status: Busy
Sep 21 05:06:18 backups kernel: (da1:mpt0:0:1:0): Retrying command

These have been happening roughly every other day.

mpt0 and em0 were sharing int 18, so today I put
hint.mpt.0.msi_enable="1"
into /boot/devices.hints and rebooted; now mpt0 is using int 256. I'll see if it helps.

Guy

[Mark Felder]

Sometimes you'll see this before a crash, but not every time.

[rafa...@gmail.com]

I know it is an old topic, but yesterday I spend some time to find why Freenas 9.2.1 hang like in your case. Maybe I will help someone else in the future. I find that in my case it happened after upgrading version of virtual hardware in VSphere 5.5 from v.7 to v.10. After changing it back to v.7 this problem gone. It is not the best solution but it is OK for me at this moment.