Blocked when running docker command in CoreOS alpha image for Arm64

[Kevin Zhao]

Hi All,

    After launching CoreOS image: https://alpha.release.core-os.net/arm64-usr/1109.1.0/coreos_production_image.bin.bz2  with libvirt, I run the command：

$docker ps 

But the command is blocked and I see the output below:

=====================================================================

audit: type=1006 audit(1469589592.992:3): pid=2751 uid=0 old-auid=4294967295 auid=500 old-ses=4294967295 ses=1 res=1

containerd[2883]: syscall 1069

Code: f94023e4 f94027e5 f9400fe8 d4000001 (b13ffc1f) 

CPU: 1 PID: 2883 Comm: containerd Not tainted 4.6.4-coreos #1

Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015

task: ffff80002e721a00 ti: ffff80002e8c8000 task.ti: ffff80002e8c8000

PC is at 0x4ea280

LR is at 0x4ea260

pc : [<00000000004ea280>] lr : [<00000000004ea260>] pstate: 60000000

sp : 00000048200327c0

x29: 0000ffffa2b7e950 x28: 0000004820166780 

x27: 0000000000d7ba68 x26: 0000000000000000 

x25: 0000000000800000 x24: 0000000000000000 

x23: 0000ffffa4f05000 x22: 0000ffffd079470f 

x21: 0000000000000000 x20: 000000000046bc68 

x19: 0000000000439f78 x18: 0000000000000000 

x17: 0000000000b6f970 x16: 0000004820032eb0 

x15: 0000000000000040 x14: 0000000000000000 

x13: 0000000000000000 x12: 0000000000000008 

x11: 0000000000000000 x10: 000000482005c540 

x9 : 0000004820032fa0 x8 : 000000000000042d 

x7 : 00000048200f6180 x6 : 0000000000533518 

x5 : 0000000000000000 x4 : 0000000000000000 

x3 : ffffffffffffffff x2 : 0000000000000080 

x1 : 00000048200328b8 x0 : 0000000000000003 

containerd[2886]: syscall 1069

Code: f94023e4 f94027e5 f9400fe8 d4000001 (b13ffc1f) 

CPU: 1 PID: 2886 Comm: containerd Not tainted 4.6.4-coreos #1

Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015

task: ffff80002e724100 ti: ffff80002e8d4000 task.ti: ffff80002e8d4000

PC is at 0x4ea280

LR is at 0x4ea260

pc : [<00000000004ea280>] lr : [<00000000004ea260>] pstate: 60000000

sp : 00000048200377c0

x29: 0000ffff9888e950 x28: 000000482005d080 

x27: 0000000000d7ba68 x26: 0000000000000000 

x25: 0000000000800000 x24: 0000000000000000 

x23: 0000ffff99415000 x22: 0000fffff70ea36f 

x21: 0000000000000000 x20: 000000000046bc68 

x19: 0000000000439f78 x18: 0000000000000000 

x17: 0000000000b6f970 x16: 0000004820037eb0 

x15: 0000000000000040 x14: 0000000000000000 

x13: 0000000000000000 x12: 0000000000000008 

x11: 0000000000000000 x10: 0000004820066540 

x9 : 0000004820037fa0 x8 : 000000000000042d 

x7 : 0000004820000d80 x6 : 0000000000533518 

x5 : 0000000000000000 x4 : 0000000000000000 

x3 : ffffffffffffffff x2 : 0000000000000080 

x1 : 00000048200378b8 x0 : 0000000000000003 

random: nonblocking pool is initialized

====================================================================

I am not sure what happened. Need your kindly help :-)

[Nick Owens]

hey kevin,

great to hear you're trying CoreOS for arm64.

this issue appears to have been fixed by
https://github.com/docker/containerd/pull/226. however, it looks like
containerd has no tag in the git repo since the commit was merged, so
the upcoming alpha (1122.0.0) will not have the fix. hopefully by the
next alpha they will have a tag that has the fix and we will bump
containerd in the OS image.

i'll be sure to write back if there's any update on this.

nick

[Gary Grebus]

FWIW, CoreOS 1081.3.0 has a working docker on arm64 (presumably the current Beta channel 1081.5.0 does too).   Things broke with docker 1.11. 

I was able to patch the "containerd: syscall 1069"  problem, but docker 1.11.2 still isn't working for me.   Hopefully the docker folks will get it straightened out.

- Gary

[Nick Owens]

hi kevin,

it would appear this was fixed by
https://github.com/docker/containerd/pull/226, however, unfortunately
containerd has no tagged releases which have the fix. hopefully they
will make a tag, and we can bump the containerd version then.

nick

On 07/26/2016 08:25 PM, Kevin Zhao wrote:

[kevin...@linaro.org]

Hi Nick,

    It's great that you can help :-)

    I will wait for the newer version of CoreOS which has newer version of Docker.

    Another question, Geoff said that you can boot CoreOS via PXE. I have tried to 

boot image with PXE but met some problem. 

    I has used the CoreOS 1122 and the scrpit : coreos_production_pxe.sh, and run

 the command:

./coreos_production_pxe.sh -nographic -append 'console=tty0 console=ttyAMA0'

But no response and console no output. 

I use below command, no response either.... and ssh 127.0.0.1 -p 2222 no  response...

./coreos_production_pxe.sh -curses 

     I am not sure what happened ,so kindly need your help. My test environment is AMD 

Seattle ARM64 server chip(SoftIron Machine), ubuntu 16.04 baremetal.

     Really appreciated for your help :-)

[Gary Grebus]

On Thursday, July 28, 2016 at 4:25:59 AM UTC-4, kevin...@linaro.org wrote:

Hi Nick,

    It's great that you can help :-)

    I will wait for the newer version of CoreOS which has newer version of Docker.

    Another question, Geoff said that you can boot CoreOS via PXE. I have tried to 

boot image with PXE but met some problem. 

    I has used the CoreOS 1122 and the scrpit : coreos_production_pxe.sh, and run

 the command:

./coreos_production_pxe.sh -nographic -append 'console=tty0 console=ttyAMA0'

But no response and console no output. 

I use below command, no response either.... and ssh 127.0.0.1 -p 2222 no  response...

./coreos_production_pxe.sh -curses 

Are you getting a blank qemu console?    I found I needed to add a EFI BIOS image to the qemu command in the script.   Google to find a copy of the  QEMU_EFI.fd file, and then add '-bios QEMU_EFI.fd'  to the qemu command in the script.

Note that script will directly boot the kernel/initrd used for PXE into a qemu virtual machine.   If you have a real machine you want to PXE boot, you need to set up your PXE server to deliver coreos_production_pxe_grub.efi, and then use the appropriate grub commands to fetch the kernel and initrd (I think the latest version of that grub supports tftp and http).

[kevin...@linaro.org]

Hi Gary,

     Thanks for your tips~

      I think I should add the uefi parameters to the scripts and try again.

[Nick Owens]

hey kevin,

you need 'coreos_production_qemu_uefi_efi_code.fd' and 'coreos_production_qemu_uefi_efi_vars.fd' from https://alpha.release.core-os.net/arm64-usr/current/. these are the UEFI firmware files for QEMU, which are built from ArmVirtPkg in our copy of edk2. for coreos_production_pxe.sh, you will have to pass them in by hand like so..

./coreos_production_pxe.sh -drive if=pflash,file=coreos_production_qemu_uefi_efi_code.fd,format=raw,readonly -drive if=pflash,file=coreos_production_qemu_uefi_efi_vars.fd,format=raw

 we haven't updated the PXE scripts to use these yet, sorry about that.

[Geoff Levand]

Hi,

On Thu, 2016-07-28 at 01:25 -0700, kevin...@linaro.org wrote:
> I has used the CoreOS 1122 and the scrpit :

> coreos_production_pxe.sh,=

I tested coreos_production_pxe.sh and coreos_production_qemu.sh,
and they don't work.

Download all the files coreos_production_qemu_uefi*, and run
coreos_production_qemu_uefi.sh. This worked when I tested it.

-Geoff

[kevin...@linaro.org]

Hey Geoff and Nick,

     With your help I can boot the qemu image with -enable-kvm parameters. If I don't  add -enable-kvm modules, I got blocked just like below: 

kevzha01@ent-arm-huawei-01:~/qemu-coreos$ sudo ./coreos_production_qemu_uefi.sh -drive if=pflash,file=coreos_production_qemu_uefi_efi_code.fd,format=raw,readonly -drive if=pflash,file=coreos_production_qemu_uefi_efi_vars.fd,format=raw

qemu-system-aarch64: -device virtio-9p-pci,fsdev=conf,mount_tag=config-2: unable to init msix vectors to 2

Warning: Orphaned drive without device: id=pflash2,file=coreos_production_qemu_uefi_efi_code.fd,if=pflash,bus=0,unit=2

Warning: Orphaned drive without device: id=pflash3,file=coreos_production_qemu_uefi_efi_vars.fd,if=pflash,bus=0,unit=3

Welcome to GRUB!

check_signatures=no

cmddevice=hd2,gpt1

cmdpath=(hd2,gpt1)/EFI/BOOT

color_highlight=black/light-gray

color_normal=light-gray/black

feature_200_final=y

feature_all_video_module=y

feature_chainloader_bpb=y

feature_default_font_path=y

feature_menuentry_id=y

feature_menuentry_options=y

feature_nativedisk_cmd=y

feature_ntldr=y

feature_platform_search_hint=y

feature_timeout_style=y

grub_cpu=arm64

grub_platform=efi

lang=

locale_dir=

net_default_ip=(null)

net_default_mac=(null)

net_default_server=

pager=

prefix=(memdisk)

pxe_default_server=

root=hd2,gpt1

secondary_locale_dir=

error: no such device: 00000000-0000-0000-0000-000000000001.

                          GNU GRUB  version 2.02~beta2

 /----------------------------------------------------------------------------\

 |*CoreOS default                                                             | 

 | CoreOS USR-A                                                               |

 | CoreOS USR-B                                                               |

 |                                                                            |

 |                                                                            |

 |                                                                            |

 |                                                                            |

 |                                                                            |

 |                                                                            |

 |                                                                            |

 |                                                                            |

 |                                                                            |

 |                                                                            | 

 \----------------------------------------------------------------------------/

      Use the ^ and v keys to select which entry is highlighted.          

      Press enter to boot the selected OS, `e' to edit the commands       

      before booting or `c' for a command-line.                           

                                                                               

EFI stub: Booting Linux Kernel...

EFI stub: Using DTB from configuration table

EFI stub: Exiting boot services and installing virtual address map...

Then it will be blocked and no response....

[Geoff Levand]

On Mon, 2016-08-08 at 05:40 -0700, kevin...@linaro.org wrote:
> With your help I can boot the qemu image with -enable-kvm

Is -enable-kvm compatable with your environment?

> EFI stub: Booting Linux Kernel...
> EFI stub: Using DTB from configuration table
> EFI stub: Exiting boot services and installing virtual address map...
>
>
> Then it will be blocked and no response....

Maybe adding an earlyprintk or earlycon parameter on the kernel
command line will help you.

-Geoff

[kevin...@linaro.org]

On Tuesday, 9 August 2016 05:22:48 UTC+8, Geoff Levand wrote:

On Mon, 2016-08-08 at 05:40 -0700, kevin...@linaro.org wrote:
>      With your help I can boot the qemu image with -enable-kvm

Is -enable-kvm compatable with your environment?
         

A little strange, in Qemu 2.3.0(Debian 1:2.3+dfsg-5ubuntu9.4), it is not compatable. With enable-kvm the qemu command will be blocked.

But in Qemu 2.6.0(Debian 1:2.6+dfsg-3~linaro1linarojessie1), I can boot the image with enable-kvm properly.

In both conditions, without --enable-kvm , I will be blocked as the previous said...

> EFI stub: Booting Linux Kernel...
> EFI stub: Using DTB from configuration table
> EFI stub: Exiting boot services and installing virtual address map...
>
>
> Then it will be blocked and no response....

Maybe adding an earlyprintk or earlycon parameter on the kernel
command line will help you.

If it convenient, could you tell me the kernel commandline that I need to add for more information?

Thanks~ 

-Geoff

[Geoff Levand]

On Tue, 2016-08-09 at 00:24 -0700, kevin...@linaro.org wrote:
> Maybe adding an earlyprintk or earlycon parameter on the kernel
> > command line will help you.
> >
> If it convenient, could you tell me the kernel commandline that I
> need to add for more information?

earlycon is platform specific, so you need to use something compatable
with your qemu configuration. Maybe 'earlycon=pl011,0x9000000'???


-Geoff

[kevin...@linaro.org]

Thanks Geoff ~

I have another working items last week and now I am working on it.