ppc64le port of CoreOS

[Brandon Philips]

Hello Everyone-

I wanted to introduce Leno and team who are working on a ppc64le port of many of the ebuilds required by CoreOS over here: https://github.com/linux-on-power/gentoo

I talked to them on a hangout for a bit and the overall plan for this port, which they want to do the maintenance of, includes:

- Waiting for all of the "second architecture" stuff to finish for ARM

- Test out rkt/update_engine and some other pieces that are becoming easier to cross compile from the above work

- Continue to work upstream to merge the 12 or so patches required for Gentoo

- Generate and distribute an unofficial PPC64LE CoreOS SDK

- Help get the SDK and build into the CoreOS build system over the next 6 months or so

If anyone can help out on this stuff it would be great but it sounds like Leno and team are making great progress so far.

Cheers,

Brandon

[Leno Hou]

Thanks Brandon, This is really what we want. Anyone who want to join  in ppc64le port of CoreOS would be thanks.

And I'll update CoreOS @ppc64le status in here If anything was made. 

Leno Hou

E-mail :  len...@gmail.com

[Brian Harrington]

Leno,

It's a pleasure! While they're aren't official (ppc isn't currently in the appc/rkt specs) I've already built ACIs just waiting on someone to need them.

--redbeard

[Leno Hou]

Hi all,

This is the updated CoreOS@ppc64le status for you and I need your help about bootloader partition.

1. Gentoo hosted ppc64le patches on Gentoo ppc64le overlay
2. The coreos@ppc64le repo manifest was pushed in github
3. All CoreOS production packages was built successfully.

coreos-build@gentoo /mnt/host/source/src/build/images/ppc64le-usr/developer-1032.0.0+2016-07-18-0550-a1 $ ls

configroot  coreos_developer_image.bin  coreos_developer_qemu.README  coreos_developer_qemu.sh  coreos_developer_qemu_image.img  developer_data  rootfs

4. This coreos_developer_qemu_image.img can't boot. This problem was due to ppc64le doesn't support UEFI and must have PReP partition for boot.  

For example, manually configure boot loader

1) Disk Partition

       device            size              Id Type

      /dev/vda1       8M               PPC PReP Boot

      /dev/vda2       10G             Linux

      /dev/vda3       2G               Linux Swap / Solaris

 2) using genkernel-next build kernel and initrd

    #genkernel --all

 3) installing bootloader

    #grub2-install /dev/vda1

 4) generating grub.cfg

   #grub2-mkconfig /boot/grub/grub.cfg

grub2 platform on ppc64le is powerpc-ieee1275

How to automatically range this disk partition use CoreOS SDK to support boot from qcow2 disk image ? Thanks.

Leno Hou

E-mail :  len...@gmail.com

On Sun, Jun 5, 2016 at 9:44 PM, Liang NS Zhuang <liang....@cn.ibm.com> wrote:

Brandon, it was a nice talk, thanks for the time!

Hello team, if necessary, we could later help to apply resource on SuperVessel (which is a Power/OpenPOWER cloud) for test..
http://www-03.ibm.com/press/us/en/pressrelease/47082.wss


Regards,
Liang ZHUANG (庄亮, Leon), PMP, CSM
Manager, Linux Dev, LTC/IBM System, CSTL/SH
Ext. 86-21-609-28548


Brandon Philips ---06/03/2016 11:46:28---Hello Everyone- I wanted to introduce Leno and team who are working on a ppc64le port of

From: Brandon Philips <brandon...@coreos.com>
To: coreos-dev <coreo...@googlegroups.com>, Leno Hou <len...@gmail.com>, Qing Yuan QQ Hou/China/IBM@IBMCN, Liang NS Zhuang/China/IBM@IBMCN, Qu KK Jiang/China/IBM@IBMCN
Date: 06/03/2016 11:46
Subject: ppc64le port of CoreOS

---

[Michael Marineau]

The file src/scripts/build_library/disk_layout.json will need new entries. For dev images copy the the "devel" layout, adding an entry to override partition 2 (BIOS BOOT), changing its type to the PReP GUID (the partitioning tool, cgpt, doesn't have a built in name for that type right now but the raw guid should work, I hope). You may need to adjust the size too. For prod images add an entry that only contains your new partition 2. Anything not defined is inherited from the "base" layout. Then pass --disk_layout= to build_image and image_to_vm. Actually setting up the bootloader should go in src/scripts/build_library/grub_install.sh

I don't know anything about booting PPC but I'd recommend keeping the PReP partition as simple/small as possible with most things in the larger ESP partition for consistency with other platforms.

[Leno Hou]

On Wed, Sep 7, 2016 at 2:06 PM, Qing Yuan QQ Hou <qingyu...@cn.ibm.com> wrote:

Hi all,

 

I concluded the status for ppc64le port of CoreOS yesterday.  Questions is with colour in red.

 

@Michael, Could you help me the parts that with colour in red ? Thanks

 

 1. make CoreOS working on ppc64le kvm                                                                              WIP

       a) modified cros_sdk working on ppc64le platform                                                      DONE

           As we might known, the SDK must be run on an x86-64 Linux machine.

           For arm64, this cros_sdk cross build packages for arm64 platform.

           For amd64, this cros_sdk local build packages for amd64 platform.

 

           This modified cros_sdk add support for ppc64le platform and lately I’ll

           push these patches for you to review.

 

       a) compiled CoreOS and Gentoo packages                                                                     DONE

           Most packages were written by C/C++/golang languages and these languages

           fully supports ppc64le platform. so all packages were built successfully.

         

       b) making CoreOS booting from KVM                                                                           ONGOING

           This divides three parts:

           a) prepare disk layout for ppc64le platform                                                             DONE

               

              Note: please don't set EFI System partition attributes to LegacyBIOS Bootable.

         

          b) installing default grub2 bootloader to PowerPC PReP boot partition       

             

           #sudo grub-install \

                   --no-nvram --no-bootsect \

                   --boot-directory="${ESP_DIR}/boot" "${LOOP_DEV}p2"

 

           #sudo cp "${BUILD_LIBRARY_DIR}/grub.cfg" \

                "${ESP_DIR}/boot/grub/grub.cfg"

 

         c)  booting from vmlinuz and initrd

           grub2.cfg pass cmdline args to kernel initrd. Then sysroot.mount and sysroot-usr.mount

            detected this cmdline args and mount the disk with LABEL=root to "/sysroot" and disk label with 

            LABEL=USR-A to "/sysroot/usr". Lastly switch-root to /sysroot as CoreOS environment.

          

            See below.  Does any problem with sysroot-usr.mount or grub2.cfg cmdline args?

          

            grub.cfg:

            

menuentry "CoreOS USR-A" --id=coreos-a {

   linux$suf /coreos/vmlinuz-a @@MOUNTUSR@@=PARTLABEL=USR-A $linux_cmdline

   initrd /coreos/initramfs-4.4.6-gentoo.img

}

         

         boot log:

 

[  OK  ] Found device /dev/disk/by-label/ROOT.

[  OK  ] Started dracut initqueue hook.

[  OK  ] Reached target Remote File Systems (Pre).

[  OK  ] Reached target Remote File Systems.

         Starting File System Check on /dev/disk/by-label/ROOT...

[  OK  ] Started File System Check on /dev/disk/by-label/ROOT.

         Starting File System Check on /dev/disk/by-label/ROOT...

         Mounting /sysroot...

[  OK  ] Started File System Check on /dev/disk/by-label/ROOT.

[    0.690207] EXT4-fs (vda9): mounted filesystem with ordered data mode. Opts: (null)

[  OK  ] Mounted /sysroot.

[  OK  ] Reached target Initrd Root File System.

         Starting Reload Configuration from the Real Root...

         Mounting /sysroot/usr...

[  OK  ] Started Reload Configuration from the Real Root.

[FAILED] Failed to mount /sysroot/usr.

See 'systemctl status sysroot-usr.mount' for details.

[DEPEND] Dependency failed for Initrd File Systems.

[  OK  ] Stopped target Basic System.

[  OK  ] Stopped target System Initialization.

         Starting Setup Virtual Console...

         Starting Cleaning Up and Shutting Down Daemons...

[  OK  ] Started Setup Virtual Console.

         Starting Emergency Shell...

[FAILED] Failed to start Cleaning Up and Shutting Down Daemons.

See 'systemctl status initrd-cleanup.service' for details.

 

Generating "/run/initramfs/rdsosreport.txt"

 

 

Entering emergency mode. Exit the shell to continue.

Type "journalctl" to view system logs.

You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot

after mounting them and attach it to a bug report.

 

 

:/# 

         

Attached the sysroot-usr.mount status in here.

:/# systemctl status sysroot-usr.mount

● sysroot-usr.mount - /sysroot/usr

   Loaded: loaded (/proc/cmdline; generated; vendor preset: enabled)

   Active: failed (Result: exit-code) since Wed 2016-09-07 05:58:42 UTC; 27min a

    Where: /sysroot/usr

     What: /dev/disk/by-label/ROOT

     Docs: man:fstab(5)

           man:systemd-fstab-generator(8)

Sep 07 05:58:42 localhost systemd[1]: Mounting /sysroot/usr...

Sep 07 05:58:42 localhost mount[263]: mount: /dev/vda9 is already mounted or /sy

Sep 07 05:58:42 localhost mount[263]:        /dev/vda9 is already mounted on /sy

Sep 07 05:58:42 localhost systemd[1]: sysroot-usr.mount: Mount process exited, c

Sep 07 05:58:42 localhost systemd[1]: Failed to mount /sysroot/usr.

Sep 07 05:58:42 localhost systemd[1]: sysroot-usr.mount: Unit entered failed sta

Warning: sysroot-usr.mount changed on disk. Run 'systemctl daemon-reload' to rel

lines 1-15/15 (END)

 

 

Best Regards,

Leno Hou

[Leno Hou]

Hi all,

Today I've successfully built  CoreOS@ppc64le and booted this CoreOS from KVM. 

Hi @Brandon Philips @Michael Marineau el at,

Could you help me to review the patches/PRs sent by Leno Hou ? 

PRs

[1] https://github.com/coreos/scripts/pull/584

[2] https://github.com/coreos/chromite/pull/22

[3] https://github.com/coreos/coreos-overlay/pull/2174

This is localhost (Linux ppc64le 4.5.2-coreos) 16:18:39

SSH host key: SHA256:ofvwp1DXD6w9KF+CQbM246QR9zVS71wf4oZmyuBbpkk (DSA)

SSH host key: SHA256:oGPzkR/jXLvUzPufWMx/suYGlYgPSEBr46gkdsvrhtc (ECDSA)

SSH host key: SHA256:Oc56kbJjAUXIA0+R+qT07MeY3sWiejYdLgCPSWJAR1E (ED25519)

SSH host key: SHA256:sK80vQ0emdXP/0czIPrQgGpUqX1OJQgtiQ7S3r5Cmfo (RSA)

eth0:  

localhost login: core

Password: 

Last login: Wed Sep  7 16:17:00  2016 on hvc0

CoreOS developer (1032.0.0+2016-09-07-1456)

-bash: wc: command not found

Failed Units: 

  initrd-setup-root.service

  systemd-tmpfiles-setup.service

core@localhost ~ $ cat /etc/os-release 

NAME=CoreOS

ID=coreos

VERSION=1032.0.0+2016-09-07-1456

VERSION_ID=1032.0.0

BUILD_ID=2016-09-07-1456

PRETTY_NAME="CoreOS 1032.0.0+2016-09-07-1456 (MoreOS)"

ANSI_COLOR="1;32"

HOME_URL="https://coreos.com/"

BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

Leno Hou

E-mail :  len...@gmail.com

[Michael Marineau]

Thanks! We've started to take a look and give feedback :)

[Jared White]

We have a newer ppc64le rig at work, and I am itching to try to install CoreOS on it.  

Let me know if you guys need any beta testing done.  

[Leno Hou]

Hi all,

I updated the CoreOS@ppc64le status in here and anyone can help to verify/build packages for ppc64le ?

This modified CoreOS SDK which is working on x86 will not make any impact on existing amd64/arm64 platform.

So If we have verified, build, solved the packages for ppc64le, I think we can emerge this developing ppc64le branch.
We will not merge previously CoreOS SDK working on ppc64le since it's broken amd64/arm64 platform.

1. We've made CoreOS SDK on x86  to support cross building packages for ppc64le platform,

    please download this modified CoreOS SDK as below steps if you want to join/verify our work.
   

    a) download coreos sdk

    #repo init -u https://github.com/houstar/manifest.git -b ppc64le

    #repo sync
   

    b) chroot coreos sdk

    #./chromite/sdk/cros_sdk
   

    Once chrooted in CoreOS SDK environment,

   

    c) set password for board

    #./set_shared_user_password.sh
 

    d) setup ppc64le-usr board

    #./setup_board --board=ppc64le-usr
  

    e) cross building packages for ppc64le-usr board

    #./build_packages --board=ppc64le-usr

2. But We've faced cross building issues in following packages. I want to someone point out

    Which packages is MUST needed and which is optional that we can skip it for CoreOS.

dev-libs/pth-2.0.7-r3
dev-lang/perl-5.20.2
sys-apps/texinfo-5.1
app-crypt/gnupg-2.0.28
sys-apps/kexec-tools-2.0.4-r1
dev-util/boost-build-1.55.0
dev-libs/boost-1.55.0-r2
sys-block/thin-provisioning-tools-0.4.1
app-emulation/rkt-9999
net-dns/bind-tools-9.10.2_p4
dev-libs/gobject-introspection-1.38.0
dev-db/etcd-9999
sys-fs/xfsprogs-3.2.2-r1
dev-libs/dbus-glib-0.102
sys-kernel/coreos-kernel-4.8.4
coreos-base/update_engine-9999
app-emulation/docker-9999
dev-lang/go-1.7.1
dev-lang/go-1.6.3-r1

      Current we known that some packages has no cross build support for ppc64le as same as arm64. e.g.

          So we can skip it.

          dev-lang/perl-5.20.2
          app-crypt/gnupg-2.0.28
          sys-apps/kexec-tools-2.0.4-r1
          dev-util/boost-build-1.55.0
          net-dns/bind-tools-9.10.2_p4

          Or ???

  3. If we have build all packages that needed for CoreOS. Could we

      Using grub running on x86 to install ieee1275 to our ppc64le qcow2 images ?

     

      #grub-install --no-bootsect --no-nvram --target=powerpc-ieee1274 ./qemu-boot-partition

Thanks!

     

Leno Hou

E-mail :  len...@gmail.com

On Thu, Sep 8, 2016 at 12:36 AM, Leno Hou <len...@gmail.com> wrote:

Hi all,

Today I've successfully built  CoreOS@ppc64le and booted this CoreOS from KVM. 

Hi @Brandon Philips @Michael Marineau el at,

Could you help me to review the patches/PRs sent by Leno Hou ? 

PRs

[1] https://github.com/coreos/scripts/pull/584

[2] https://github.com/coreos/chromite/pull/22

[3] https://github.com/coreos/coreos-overlay/pull/2174

Leno Hou

E-mail :  len...@gmail.com

[Leno Hou]

Hi Brandon,

We've spend 5 months to make CoreOS@ppc64le it happened. And I replied he overview of progress in your mail. 

On Fri, Jun 3, 2016 at 11:46 AM, Brandon Philips <brandon...@coreos.com> wrote:

Hello Everyone-

I wanted to introduce Leno and team who are working on a ppc64le port of many of the ebuilds required by CoreOS over here: https://github.com/linux-on-power/gentoo

I talked to them on a hangout for a bit and the overall plan for this port, which they want to do the maintenance of, includes:

- Waiting for all of the "second architecture" stuff to finish for ARM

- Test out rkt/update_engine and some other pieces that are becoming easier to cross compile from the above work

Currently rkt doesn't support ppc64/ppc64le archiecture and we've ignore app-emulation/rkt on ppc64le platform.

 

- Continue to work upstream to merge the 12 or so patches required for Gentoo

1) We've merged the 12 or more patches into Gentoo official overlay -> https://cgit.gentoo.org/repo/proj/ppc64le.git

2) We use this official ppc64le-overlay + coreos-overlay + portage-stable to build PPC64LE CoreOS SDK and production image.

- Generate and distribute an unofficial PPC64LE CoreOS SDK

Yes. We've made PPC64LE CoreOS SDK. Could we deliver this SDK on CoreOS website? e.g. https://alpha.release.core-os.net/index.html

 

- Help get the SDK and build into the CoreOS build system over the next 6 months or so

Could you help to review and merge this github PR#584  if we want to official support CoreOS@ppc64le platform ?

If anyone can help out on this stuff it would be great but it sounds like Leno and team are making great progress so far.

Welcome someone who wants to have a try with this CoreOS@ppc64le. We achieved most functions of CoreOS except grub with gptprio.next and rkt.

Congrats !  See following message on CoreOS@ppc64le. Feel free to use latest version 1248 and ping me.

This is localhost (Linux ppc64le 4.9.0-coreos) 12:28:00

SSH host key: SHA256:8zj5hO94ouSQH/WXTnDHLNtAgEjvVkwf3bcUHOFGjpY (DSA)

SSH host key: SHA256:lAVhp7wmFFvH5EUqkDBN2lan5rSS0K7F/rZS+PZhFiI (ED25519)

SSH host key: SHA256:Ty993NifHZxHFCzVbuEdSElwQjU66u9hFbY5THbQU3c (ECDSA)

SSH host key: SHA256:08Z8r2/vrYDnUl0S8EMhO2vr4klx1iNMTz34HVIUvgM (RSA)

localhost login: core

Password: 

Last login: Wed Dec 14 12:26:51 UTC 2016 on hvc0

CoreOS developer (1248.0.0)

core@localhost ~ $ uname -a

Linux localhost 4.9.0-coreos #1 SMP Wed Dec 14 11:00:08 2016 ppc64le POWER8E (raw), altivec supported CHRP IBM pSeries (emulated by qemu) GNU/Linux

core@localhost ~ $ docker info

[   19.881211] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.

[   19.882677] Bridge firewalling registered

[   19.887969] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)

[   19.933514] Initializing XFRM netlink socket

[   19.961334] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready

Containers: 0

 Running: 0

 Paused: 0

 Stopped: 0

Images: 0

Server Version: 1.12.4

Storage Driver: overlay

 Backing Filesystem: extfs

Logging Driver: json-file

Cgroup Driver: cgroupfs

Plugins:

 Volume: local

 Network: bridge host null overlay

Swarm: inactive

Runtimes: runc

Default Runtime: runc

Security Options: apparmor

Kernel Version: 4.9.0-coreos

Operating System: CoreOS 1248.0.0 (Ladybug)

OSType: linux

Architecture: ppc64le

CPUs: 4

Total Memory: 1.979 GiB

Name: localhost

ID: JOOO:NP7H:3Y5Q:XUY3:7XUU:P54L:VMAM:7GPZ:KDZX:JQY2:K4YQ:ZY4P

Docker Root Dir: /var/lib/docker

Debug Mode (client): false

Debug Mode (server): false

Registry: https://index.docker.io/v1/

WARNING: No swap limit support

Insecure Registries:

 127.0.0.0/8

........

........

--Leno Hou