How to collect serial console logs .

[Nirmoy]

Hi There, 

I am trying to automate kernel CI and using Zuul,  Kubernetes, and kubevirt to boot a new kernel and check if everything was fine by reading dmesg. The problem is if the new kernel fails to boot properly and is not responding over ssh, I would like to fetch serial console log for Zuul failure output artifacts.

Can someone please guide me to the right docs regarding fetching VM's serial console log?

[Nirmoy]

Ok I managed to hack sidecar example to dump console into file in virt-launcher pod. This should be enough for now! 

Sidecar is great!

Thanks,

Nirmoy

diff --git a/cmd/example-hook-sidecar/smbios.go b/cmd/example-hook-sidecar/smbios.go

index 0ba3cc83b..fea7ac8b8 100644

--- a/cmd/example-hook-sidecar/smbios.go

+++ b/cmd/example-hook-sidecar/smbios.go

@@ -119,6 +119,18 @@ func onDefineDomain(vmiJSON []byte, domainXML []byte) ([]byte, error) {

                panic(err)

        }

 

+       var serialPort uint = 0

+       domainSpec.Devices.Serials = []domainSchema.Serial{

+               {

+                       Type: "file",

+                       Target: &domainSchema.SerialTarget{

+                               Port: &serialPort,

+                       },

+                       Source: &domainSchema.SerialSource{

+                               Path: "/var/run/serial0",

+                       },

+               },

+       }

        domainSpec.OS.SMBios = &domainSchema.SMBios{Mode: "sysinfo"}

 

        if domainSpec.SysInfo == nil {

[dvo...@redhat.com]

On Wednesday, April 14, 2021 at 2:49:22 PM UTC-4 nirmoy...@gmail.com wrote:

Ok I managed to hack sidecar example to dump console into file in virt-launcher pod. This should be enough for now! 

Sidecar is great!

great that you were able to find a work around

dumping serial console to k8s logs is something i've thought about in the past. One solution is that we could create a container within  the VMI pod just for capturing the serial log output. We'd stream the serial console to the stdout of that container which would then make it available through standard kubectl tooling, like 'kubectl logs my-vmi-pod -c console'

[Roman Mohr]

On Fri, Apr 16, 2021 at 11:20 PM dvo...@redhat.com <dvo...@redhat.com> wrote:

On Wednesday, April 14, 2021 at 2:49:22 PM UTC-4 nirmoy...@gmail.com wrote:

Ok I managed to hack sidecar example to dump console into file in virt-launcher pod. This should be enough for now! 

Sidecar is great!

great that you were able to find a work around

dumping serial console to k8s logs is something i've thought about in the past. One solution is that we could create a container within  the VMI pod just for capturing the serial log output. We'd stream the serial console to the stdout of that container which would then make it available through standard kubectl tooling, like 'kubectl logs my-vmi-pod -c console'

Having a consistent solution for that sounds great. I like your idea right away for debugging cases (so not enabled by default). In general it would be great if we could "just" enable this for default. We may need a different solution for that, to not create too much memory overhead per VM.

Best regards,

Roman

 

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/0311a3e3-aa8b-401b-a371-8c4d8c0e8f27n%40googlegroups.com.

[David Vossel]

On Mon, Apr 19, 2021 at 4:48 AM Roman Mohr <rm...@redhat.com> wrote:

On Fri, Apr 16, 2021 at 11:20 PM dvo...@redhat.com <dvo...@redhat.com> wrote:

On Wednesday, April 14, 2021 at 2:49:22 PM UTC-4 nirmoy...@gmail.com wrote:

Ok I managed to hack sidecar example to dump console into file in virt-launcher pod. This should be enough for now! 

Sidecar is great!

great that you were able to find a work around

dumping serial console to k8s logs is something i've thought about in the past. One solution is that we could create a container within  the VMI pod just for capturing the serial log output. We'd stream the serial console to the stdout of that container which would then make it available through standard kubectl tooling, like 'kubectl logs my-vmi-pod -c console'

Having a consistent solution for that sounds great. I like your idea right away for debugging cases (so not enabled by default). In general it would be great if we could "just" enable this for default. We may need a different solution for that, to not create too much memory overhead per VM.

It's unclear to me if we'd want to ever enable this functionality by default. The serial consoles can potentially leak guest info into the cluster. I think it's likely this will always remain an opt-in feature. 

[Nanfang Li]

Hi David and Kubevirter,

Hit similar problems here, if we change the console from pty into file, we lose virt console input access.

I do see in qemu we have the option to mux the serial console into file, but it seems the libvirt does not support it? at least I did not find the config knob. (https://patchwork.kernel.org/project/qemu-devel/patch/1455638581-5912-1-git-s...@linaro.org/)

Is there any suggestions to stream the console log into files, at the same time without losing the console access?

libvirt does support adding a file to a serial console, but that one does not produce anything unless the user connects to the serial port, so unfortunately we don't have the early boot/bios logs etc.

Any suggestions are welcome,

Thanks,

Nanfang

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAPjOJFvnkcwza0UyfKpxN8KH0%2BtQ97skkKKn6--8vQiXDeztqg%40mail.gmail.com.

[Andrea Bolognani]

On Tue, May 04, 2021 at 05:34:03PM -0700, 'Nanfang Li' via kubevirt-dev wrote:
> Hi David and Kubevirter,
>
> Hit similar problems here, if we change the console from pty into file, we
> lose virt console input access.
>
> I do see in qemu we have the option to mux the serial console into file,
> but it seems the libvirt does not support it? at least I did not find the
> config knob. (
> https://patchwork.kernel.org/project/qemu-devel/patch/1455638581-5912-1-git-s...@linaro.org/
> )
>
> Is there any suggestions to stream the console log into files, at the same
> time without losing the console access?
>
> libvirt does support adding a file to a serial console, but that one does
> not produce anything unless the user connects to the serial port, so
> unfortunately we don't have the early boot/bios logs etc.
>
> Any suggestions are welcome,

At the libvirt level, you should be able to use something like

<serial type='pty'>
<log file='/var/log/my.log' append='off'/>
</serial>

which translates to

-chardev pty,id=charserial0,logfile=/var/log/my.log,logappend=off
-device isa-serial,chardev=charserial0,id=serial0

at the QEMU level. As far as I can tell, this knob is not exposed
by KubeVirt.

I agree with David that we'd probably want to have the log exposed
via standard Kubernetes APIs rather than stashed away on the
filesystem though.

--
Andrea Bolognani / Red Hat / Virtualization

[Roman Mohr]

On Fri, Apr 16, 2021 at 11:20 PM dvo...@redhat.com <dvo...@redhat.com> wrote:

On Wednesday, April 14, 2021 at 2:49:22 PM UTC-4 nirmoy...@gmail.com wrote:

Ok I managed to hack sidecar example to dump console into file in virt-launcher pod. This should be enough for now! 

Sidecar is great!

great that you were able to find a work around

dumping serial console to k8s logs is something i've thought about in the past. One solution is that we could create a container within  the VMI pod just for capturing the serial log output. We'd stream the serial console to the stdout of that container which would then make it available through standard kubectl tooling, like 'kubectl logs my-vmi-pod -c console'

Did you consider an extra subresource for this and just go through virt-handler? I know it is sightly less convenient if `virtctl` is not present, but a clear `kubectl virt logs` or `virtctl logs` would not be that bad either and the access would be a standard websocket stream like `kubectl logs` internally.

Best regards,

Roman 

 

--

[Nanfang Li]

Thanks Andrea!

Yes, I indeed tried this implementation but as I mentioned, this solution has a problem of not streaming the logs if the socket is not being used, so if user does not perform virt console xxx, there is no logs, until user connect, it will work, so we miss the early booting logs, in this post (https://github.com/Mirantis/virtlet/issues/249), 2nd solution, seems hitting similar.

What works for instance, in qemu level is something like below, so it will both flash the logs continuously no matter the console is used or not, but it seems this knob is not wired out into libvirt.

-chardev stdio,id=char0,mux=on,logfile=/path/to/my/serial.log,signal=off  -serial chardev:char0

 

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CABJz62N%2BBgHnoB_3%2BGR8OR9q6%3Dag220J7VhycHbu_V4dafcW4g%40mail.gmail.com.

[Nanfang Li]

Thanks Roman! I think once the log is present, either via a socket, or file or any format, we can have better ways to stream it out, the problem I am hitting though is not able to keep both input and logs at the same time.

The solution from this (https://github.com/Mirantis/virtlet/issues/249) ended up disabling the console input and wire it out into regular K8s API, but the loss of access with serial seems significant with respect to logging only, so this is not preferable I guess.

Nanfang

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CALDPj7vpD7JfK0UPnHRkkdtBN892Ew3KFr5aVZRbmj%2B%3D-Vsgnw%40mail.gmail.com.

[Andrea Bolognani]

On Wed, May 05, 2021 at 07:58:56AM -0700, Nanfang Li wrote:
> On Wed, May 5, 2021 at 1:27 AM Andrea Bolognani <abol...@redhat.com> wrote:
> > On Tue, May 04, 2021 at 05:34:03PM -0700, 'Nanfang Li' via kubevirt-dev wrote:
> > > Is there any suggestions to stream the console log into files, at the same
> > > time without losing the console access?
> > >
> > > libvirt does support adding a file to a serial console, but that one does
> > > not produce anything unless the user connects to the serial port, so
> > > unfortunately we don't have the early boot/bios logs etc.
> >

> > At the libvirt level, you should be able to use something like
> >
> > <serial type='pty'>
> > <log file='/var/log/my.log' append='off'/>
> > </serial>
> >
> > which translates to
> >
> > -chardev pty,id=charserial0,logfile=/var/log/my.log,logappend=off
> > -device isa-serial,chardev=charserial0,id=serial0
> >
> > at the QEMU level. As far as I can tell, this knob is not exposed
> > by KubeVirt.
>

> Yes, I indeed tried this implementation but as I mentioned, this solution
> has a problem of not streaming the logs if the socket is not being used, so
> if user does not perform virt console xxx, there is no logs, until user
> connect, it will work, so we miss the early booting logs, in this post (
> https://github.com/Mirantis/virtlet/issues/249), 2nd solution, seems
> hitting similar.

Are you quite sure of this? I've just tried with a local libvirt VM
and the entirety of the serial output was recorded in the log file,
including the GRUB menu. I just started the VM using 'virsh start'
and then stopped it using 'virsh shutdown'; at no point did I connect
to the serial console.

Is your guest OS configured to actually send boot messages to the
serial console? You usually need to have

GRUB_TERMINAL=serial
GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0"

in your /etc/default/grub file when calling grub2-mkconfig for that
to be the case.

[Nanfang Li]

Emmmm, this is the version of the type pty (similar like the version of file with side car), which hijike the socket for the virt console access interactively, so we lose the console access as there is no unix socket to accept the streaming in/out anymore.

It indeed works, but the penalty is the loss of "virt console".

Nanfang

[David Vossel]

On Wed, May 5, 2021 at 6:13 PM Nanfang Li <nan...@google.com> wrote:

Emmmm, this is the version of the type pty (similar like the version of file with side car), which hijike the socket for the virt console access interactively, so we lose the console access as there is no unix socket to accept the streaming in/out anymore.

It indeed works, but the penalty is the loss of "virt console".

If we can't find the arguments for libvirt/qemu that give us the desired behavior, I think this can be solved on our end without losing the ability to access the console 

Let's start with how console works today in KubeVirt.  We're exposing console via a unix file within the virt-launcher pod, then we're creating a tunnelled connection from the client (usually virtctl) that goes through virt-api, virt-handler, and eventually directly to that unix socket in virt-launcher. 

Now, if we are willing to mux the connections ourselves within virt-launcher, we can create a server in front of that console unix sock that can stream the console output to multiple places (a log file, and a console stream connection).  The console stream from virtctl would connect to the mux server which would then forward input/output from the console connection.

[Nanfang Li]

Thanks David.

Yeap, I agree with this, this is indeed one of the solutions we are thinking of, the other table of option is to modify the libvirt to wire in the flag, I am not sure at this stage which one is easier though, do you have some insight or experience here?

Thanks,

Nanfang

--
You received this message because you are subscribed to the Google Groups "kubevirt-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubevirt-dev...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubevirt-dev/CAPjOJFvEuiZZygYbojeRYRewR-A_DsHFYEToJUowm-09rk9fjA%40mail.gmail.com.

[David Vossel]

On Thu, May 6, 2021 at 10:31 AM Nanfang Li <nan...@google.com> wrote:

Thanks David.

Yeap, I agree with this, this is indeed one of the solutions we are thinking of, the other table of option is to modify the libvirt to wire in the flag, I am not sure at this stage which one is easier though, do you have some insight or experience here?

modifying virt-launcher to mux connections will be the path of least resistance. 

[Andrea Bolognani]

On Thu, May 06, 2021 at 10:43:24AM -0400, David Vossel wrote:
> On Thu, May 6, 2021 at 10:31 AM Nanfang Li <nan...@google.com> wrote:
> >>> Emmmm, this is the version of the type pty (similar like the version of
> >>> file with side car), which hijike the socket for the virt console access
> >>> interactively, so we lose the console access as there is no unix socket to
> >>> accept the streaming in/out anymore.
> >>>
> >>> It indeed works, but the penalty is the loss of "virt console".
> >>
> >> If we can't find the arguments for libvirt/qemu that give us the desired
> >> behavior, I think this can be solved on our end without losing the ability
> >> to access the console
> >>
> >> Let's start with how console works today in KubeVirt. We're exposing
> >> console via a unix file within the virt-launcher pod, then we're creating a
> >> tunnelled connection from the client (usually virtctl) that goes through
> >> virt-api, virt-handler, and eventually directly to that unix socket in
> >> virt-launcher.
> >>
> >> Now, if we are willing to mux the connections ourselves within
> >> virt-launcher, we can create a server in front of that console unix sock
> >> that can stream the console output to multiple places (a log file, and a
> >> console stream connection). The console stream from virtctl would connect
> >> to the mux server which would then forward input/output from the console
> >> connection.
>

> > Yeap, I agree with this, this is indeed one of the solutions we are
> > thinking of, the other table of option is to modify the libvirt to wire in
> > the flag, I am not sure at this stage which one is easier though, do you
> > have some insight or experience here?
>
> modifying virt-launcher to mux connections will be the path of least
> resistance.

Was using virDomainOpenConsole() in virt-launcher considered?

https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainOpenConsole

That's what 'virsh console' does, and with this setup interactive
console can be used for <console type='pty'>. libvirtd takes care of
the muxing.

There might be architectural reasons to prefer <console type='unix'>,
but I just want to make sure this option has been evaluated.