GDB stub

[Waldek Kozaczuk]

Currently, it is only possible to connect from gdb to a running instance of OSv on QEMU. But ideally, it would be nice to have a gdb stub in OSv that would let one connect from gdb to OSv running on any hypervisor.

Per this fragment of this Wiki - https://github.com/cloudius-systems/osv/wiki/Debugging-OSv

"The fact that our gdb support works with the unit of cpus, not OSV threads, makes it convenient to debug OSV itself, but less convenient to debug application code running inside OSV. Particularly problematic is single stepping support which isn’t currently supported well: The problem is that when you try to look at a single OSV thread (see the next section), a single-step operation (“s” or “n”) makes one step inside the vcpu, not the osv thread, and this step might include switching to a different thread. One way to work around this problem is to compile OSV with preemption disabled (not recommended, but can be a convenient trick in some debugging situations). A better approach would have been to include a gdb stub into OSV itself, to allow gdb to see OSV threads directly and single-step them. We’ve made some progress in implementing such a stub, but it isn’t ready for prime-time yet."

it looks like somebody may have implemented an initial version of the gdb stub. Is there a working patch of it somewhere?

Regards,

Waldek

[Nadav Har'El]

I have a vague recollection - it was many years ago - that Glauber (CCed)  was the one who tried doing it.

Glauber, if it was you, do you remember what you tried?

If it wasn't you, sorry for the spam :-)

Regards,

Waldek

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/a3ca8d68-5994-4b51-94c4-539e2024f12dn%40googlegroups.com.

[Glauber Costa]

On Mon., Feb. 8, 2021, 3:00 a.m. Nadav Har'El, <n...@scylladb.com> wrote:

On Wed, Feb 3, 2021 at 1:18 AM Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

Currently, it is only possible to connect from gdb to a running instance of OSv on QEMU. But ideally, it would be nice to have a gdb stub in OSv that would let one connect from gdb to OSv running on any hypervisor.

Per this fragment of this Wiki - https://github.com/cloudius-systems/osv/wiki/Debugging-OSv

"The fact that our gdb support works with the unit of cpus, not OSV threads, makes it convenient to debug OSV itself, but less convenient to debug application code running inside OSV. Particularly problematic is single stepping support which isn’t currently supported well: The problem is that when you try to look at a single OSV thread (see the next section), a single-step operation (“s” or “n”) makes one step inside the vcpu, not the osv thread, and this step might include switching to a different thread. One way to work around this problem is to compile OSV with preemption disabled (not recommended, but can be a convenient trick in some debugging situations). A better approach would have been to include a gdb stub into OSV itself, to allow gdb to see OSV threads directly and single-step them. We’ve made some progress in implementing such a stub, but it isn’t ready for prime-time yet."

it looks like somebody may have implemented an initial version of the gdb stub. Is there a working patch of it somewhere?

I have a vague recollection - it was many years ago - that Glauber (CCed)  was the one who tried doing it.

Glauber, if it was you, do you remember what you tried?

If it wasn't you, sorry for the spam :-)

I don't think it was me. I have no memory of ever doing this. 

But no need to be sorry, it's good to hear from you!

[Matias Vara]

Hello everyone, 

I would like to step into this topic, maybe my experience may help and also you may help me too. A couple of months ago, I figured that I can't debug Toro KVM guests by using the QEMU built-in gdbstub. I tried hwbreak but that did not work either. The execution was not stopping on the breakpoints. The QEMU built-in gdbstub works perfectly fine when KVM is not used. I think this is a limitation of the QEMU gdbstub for x86. With a bit of debugging, I figured out that there were actually " KVM_EXIT_DEBUG" when breakpoints were reached but control did not go back to the gdbstub (see QEMU tracepoints). I investigated nothing further. I wanted to avoid writing a gdbstub for Toro but I did not have another choice. At the same time, this allows debugging the unikernel no matters the hypervisor. 

Regards, Matias.   

[Waldek Kozaczuk]

Hi Matias,

Good to hear from you.

Are you saying that Toro has already implemented a gdb stub and we can "steal" it (:-) or are you suggesting we work together to implement one? I am all for that.

Can you point to any documentation on how such a stub can be implemented? Also, does gdb require communication over tcp and then the guest must have the networking working? That would prevent early boot debugging.

Regards,

Waldek 

[Matias Vara]

On Wed, 17 Feb 2021 at 06:04, Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

Hi Matias,

Good to hear from you.

Are you saying that Toro has already implemented a gdb stub and we can "steal" it (:-) or are you suggesting we work together to implement one? I am all for that.

Hahaha, I am currently implementing one but it is still a PoC. I just started one month ago (see branch firfox#421) and I have managed to make work most of the commands like break, continue, step, etc. I just step into the conversation because I was very surprised that the QEMU gdbstub built-in did not work well in a KVM guest so I wanted to warn you before you go that path.  

 

Can you point to any documentation on how such a stub can be implemented? Also, does gdb require communication over tcp and then the guest must have the networking working? That would prevent early boot debugging.

I found the following document very interesting: https://www.embecosm.com/appnotes/ean4/embecosm-howto-rsp-server-ean4-issue-2.html#sec_exchange_target_remote. This explains the remote protocol that the gdbstub has to implement. I also used the project https://github.com/mborgerson/gdbstub to have an idea. The way I implemented is by using the serial port and then telling QEMU to forward the serial port to the tcp:1234 (legacy serial port not virtio-console). Then, you have just to execute gdb with "remote target localhost:1234". However, this approach is very slow so I have to find a way to improve it. The problem with early debugging could be if you want to want to debug before you enable long mode. I think gdb wont be happy with that. If this is not the case, early debugging could be done by just using the legacy serial device.

I hope that helps. 

Matias     

   

Regards,

Waldek 

On Saturday, February 13, 2021 at 9:30:46 AM UTC-5 matia...@gmail.com wrote:

Hello everyone, 

I would like to step into this topic, maybe my experience may help and also you may help me too. A couple of months ago, I figured that I can't debug Toro KVM guests by using the QEMU built-in gdbstub. I tried hwbreak but that did not work either. The execution was not stopping on the breakpoints. The QEMU built-in gdbstub works perfectly fine when KVM is not used. I think this is a limitation of the QEMU gdbstub for x86. With a bit of debugging, I figured out that there were actually " KVM_EXIT_DEBUG" when breakpoints were reached but control did not go back to the gdbstub (see QEMU tracepoints). I investigated nothing further. I wanted to avoid writing a gdbstub for Toro but I did not have another choice. At the same time, this allows debugging the unikernel no matters the hypervisor. 

Regards, Matias.   

On Wed, 3 Feb 2021 at 00:18, Waldek Kozaczuk <jwkoz...@gmail.com> wrote:

Currently, it is only possible to connect from gdb to a running instance of OSv on QEMU. But ideally, it would be nice to have a gdb stub in OSv that would let one connect from gdb to OSv running on any hypervisor.

Per this fragment of this Wiki - https://github.com/cloudius-systems/osv/wiki/Debugging-OSv

"The fact that our gdb support works with the unit of cpus, not OSV threads, makes it convenient to debug OSV itself, but less convenient to debug application code running inside OSV. Particularly problematic is single stepping support which isn’t currently supported well: The problem is that when you try to look at a single OSV thread (see the next section), a single-step operation (“s” or “n”) makes one step inside the vcpu, not the osv thread, and this step might include switching to a different thread. One way to work around this problem is to compile OSV with preemption disabled (not recommended, but can be a convenient trick in some debugging situations). A better approach would have been to include a gdb stub into OSV itself, to allow gdb to see OSV threads directly and single-step them. We’ve made some progress in implementing such a stub, but it isn’t ready for prime-time yet."

it looks like somebody may have implemented an initial version of the gdb stub. Is there a working patch of it somewhere?

Regards,

Waldek

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/a3ca8d68-5994-4b51-94c4-539e2024f12dn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/37acb542-8397-4e0f-8dad-9c7ee9003596n%40googlegroups.com.