[Error] unleashed hifive board is freezed in the linux kernel boot stage on keystone dev branch

[Kenta Nakajima]

Hi,

I want to launch the unleashed hifive board(not qemu) on dev branch(e448fa323ae28ec09acea85a3e544aa8f70f2bee), but it's freezed in the linux kernel boot stage.

I read the page, https://docs.keystone-enclave.org/en/dev/Getting-Started/Running-Keystone-on-Hardware.html, and setup the keystone on dev branch almost accordingly.

The only difference is that in the `1.2.3.2.3. Example loading script` stage, I used this following script to load the FSBL and BBL into a card. the fsbl partition number is just fixed.

```

#!/bin/bash

set -e

# Relevant partition type codes

BBL=2E54B353-1271-4842-806F-E436D6AF6985

LINUX=0FC63DAF-8483-4772-8E79-3D69D8477DE4

FSBL=5B193300-FC78-40CD-8002-E86C45580B47

MKE2FS=mke2fs

DISK=$1

echo "Operating on $DISK"

test -b $DISK

echo "$DISK exists, paritioning..."

# Configure the partitions on the disk.

# NOTE: The block ranges given here are for the 8GB card we are using

#       You may wish to use different partition sizes.

sgdisk --clear \

       --new=1:2048:67583  --change-name=1:bootloader --typecode=1:$BBL   \

       --new=2:264192:     --change-name=2:root       --typecode=2:$LINUX \

       --new=4:67584:69631 --change-name=4:siv-fsbl   --typecode=4:$FSBL \

       $DISK

sleep 1

PART_BBL=$DISK"p1"

PART_LINUX=$DISK"p2"

PART_FSBL=$DISK"p4"

echo "COPYING BBL to $PART_BBL"

test -b $PART_BBL

dd if=bbl.bin of=$PART_BBL bs=4096

echo "Copy done"

echo "COPYING FSBL to $PART_FSBL"

test -b $PART_FSBL

$MKE2FS -t ext3 $PART_FSBL

dd if=fsbl.bin of=$PART_FSBL bs=1024

echo "Copy done"

```

Here is the boot log on sifive unleashed board;

```

SiFive FSBL:       2020-11-10-24f612f

Using FSBL DTB

HiFive-U serial #: 00000328

Loading boot payload................................

bbl loader

Relocated DTB from 000000027fe00000 to 0000000083200000

                SIFIVE, INC.

         5555555555555555555555555

        5555                   5555

       5555                     5555

      5555                       5555

     5555       5555555555555555555555

    5555       555555555555555555555555

   5555                             5555

  5555                               5555

 5555                                 5555

5555555555555555555555555555          55555

 55555           555555555           55555

   55555           55555           55555

     55555           5           55555

       55555                   55555

         55555               55555

           55555           55555

             55555       55555

               55555   55555

                 555555555

                   55555

                     5

           SiFive RISC-V Core IP

ininitializiinitializiinitializiitializingng sm

ng initializeng sm

inid sm

ztsiaslmi

nmi

 iedi[tialize sm

.   it 0di aslm

Oi0z00e0d0 0s]m

 F: fdt: Ignoring memory range 0x80000000 - 0x80200000

[    0.000000] Linux version 5.6.0-dirty (root@10606a644613) (gcc version 7.2.0 (GCC)) #2 SMP Fri Nov 13 07:22:59 UTC 2020

[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')

[    0.000000] printk: bootconsole [sbi0] enabled

[    0.000000] initrd not found or empty - disabling initrd

[    0.000000] Zone ranges:

[    0.000000]   DMA32    [mem 0x0000000080200000-0x00000000ffffffff]

[    0.000000]   Normal   [mem 0x0000000100000000-0x000000027fffffff]

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x0000000080200000-0x000000027fffffff]

[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x000000027fffffff]

[    0.000000] cma: Reserved 1024 MiB at 0x00000000c0000000

[    0.000000] software IO TLB: mapped [mem 0xbc000000-0xc0000000] (64MB)

[    0.000000] CPU with hartid=0 is not available

[    0.000000] CPU with hartid=0 is not available

[    0.000000] elf_hwcap is 0x112d

[    0.000000] percpu: Embedded 17 pages/cpu s31000 r8192 d30440 u69632

[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 2067975

[    0.000000] Kernel command line: console=hvc earlycon=sbi cma=1024M@0-4G

[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)

[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)

[    0.000000] Sorting __ex_table...

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off

[    0.000000] Memory: 7128832K/8386560K available (6211K kernel code, 396K rwdata, 2021K rodata, 6922K init, 313K bss, 209152K reserved, 1048576K cma-reserved)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1

[    0.000000] rcu: Hierarchical RCU implementation.

[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.

[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.

[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4

[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0

[    0.000000] plic: mapped 53 interrupts with 4 handlers for 9 contexts.

[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [4]

[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 3526361616960 ns

[    0.000006] sched_clock: 64 bits at 1000kHz, resolution 1000ns, wraps every 2199023255500ns

[    0.008484] Console: colour dummy device 80x25

[    0.012790] printk: console [hvc0] enabled

[    0.012790] printk: console [hvc0] enabled

[    0.020929] printk: bootconsole [sbi0] disabled

[    0.020929] printk: bootconsole [sbi0] disabled

[    0.029988] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=4000)

[    0.040032] pid_max: default: 32768 minimum: 301

[    0.045111] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)

[    0.052517] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)

[    0.061947] rcu: Hierarchical SRCU implementation.

[    0.066402] smp: Bringing up secondary CPUs ...

[    0.072083] smp: Brought up 1 node, 4 CPUs

[    0.076701] devtmpfs: initialized

[    0.081674] random: get_random_u32 called from bucket_table_alloc.isra.26+0x4e/0x15c with crng_init=0

[    0.082018] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns

[    0.100191] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)

[    0.163410] NET: Registered protocol family 16

[    0.183326] vgaarb: loaded

[    0.185569] SCSI subsystem initialized

[    0.189330] usbcore: registered new interface driver usbfs

[    0.194507] usbcore: registered new interface driver hub

[    0.199872] usbcore: registered new device driver usb

[    0.205508] clocksource: Switched to clocksource riscv_clocksource

[    0.216711] NET: Registered protocol family 2

[    0.220896] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)

[    0.229049] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)

[    0.238817] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)

[    0.248750] TCP: Hash tables configured (established 65536 bind 65536)

[    0.254939] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)

[    0.261748] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)

[    0.269138] NET: Registered protocol family 1

[    0.273318] RPC: Registered named UNIX socket transport module.

[    0.278733] RPC: Registered udp transport module.

[    0.283412] RPC: Registered tcp transport module.

[    0.288099] RPC: Registered tcp NFSv4.1 backchannel transport module.

[    0.294529] PCI: CLS 0 bytes, default 64

[    0.360071] workingset: timestamp_bits=62 max_order=21 bucket_order=0

[    0.373442] NFS: Registering the id_resolver key type

[    0.377745] Key type id_resolver registered

[    0.381949] Key type id_legacy registered

[    0.385887] nfs4filelayout_init: NFSv4 File Layout Driver Registering...

[    0.392747] 9p: Installing v9fs 9p2000 file system support

[    0.398358] NET: Registered protocol family 38

[    0.402495] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)

[    0.409843] io scheduler mq-deadline registered

[    0.414354] io scheduler kyber registered

[    0.455662] Serial: 8

```

It's freezed in the middle of the message. The message would be `Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled`.

It's almost the same as v0.4.(I have no problem on v0.3, so currently I have used this version.)

How to fix this error?

Thanks.

[Tianyu Yin]

I yet have used Sifve Unleashed. But I think append "initcall_debug=1" to the kernel command line might be a good idea looking for which call is freezed.

[Kenta Nakajima]

Thank you.

I tried with `initcall_debug=1`, but there were no information in the boot message. The boot log is below.

> I yet have used Sifve Unleashed.

Would you tell me the commit hash of keystone you have used on SiFive unleashed board?

```

initialiinitializiinitializiinitializizing sm

nng sm

inig sm

initng sm

tiainitializeialized sminitializelized sm

d

  d ssmm

[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000

[    0.000000] Linux version 5.6.0-dirty (root@21a54c192a40) (gcc version 7.2.0 (GCC)) #2 SMP Wed Nov 25 05:00:15 UTC 2020

[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')

[    0.000000] printk: bootconsole [sbi0] enabled

[    0.000000] initrd not found or empty - disabling initrd

[    0.000000] Zone ranges:

[    0.000000]   DMA32    [mem 0x0000000080200000-0x00000000ffffffff]

[    0.000000]   Normal   [mem 0x0000000100000000-0x000000027fffffff]

[    0.000000] Movable zone start for each node

[    0.000000] Early memory node ranges

[    0.000000]   node   0: [mem 0x0000000080200000-0x000000027fffffff]

[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x000000027fffffff]

[    0.000000] cma: Reserved 1024 MiB at 0x00000000c0000000

[    0.000000] software IO TLB: mapped [mem 0xbc000000-0xc0000000] (64MB)

[    0.000000] CPU with hartid=0 is not available

[    0.000000] CPU with hartid=0 is not available

[    0.000000] elf_hwcap is 0x112d

[    0.000000] percpu: Embedded 17 pages/cpu s31000 r8192 d30440 u69632

[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 2067975

[    0.000000] Kernel command line: initcall_debug=1 console=hvc earlycon=sbi cma=1024M@0-4G

[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)

[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)

[    0.000000] Sorting __ex_table...

[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off

[    0.000000] Memory: 7128832K/8386560K available (6211K kernel code, 396K rwdata, 2021K rodata, 6922K init, 313K bss, 209152K reserved, 1048576K cma-reserved)

[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1

[    0.000000] rcu: Hierarchical RCU implementation.

[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.

[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.

[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4

[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0

[    0.000000] plic: mapped 53 interrupts with 4 handlers for 9 contexts.

[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [4]

[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 3526361616960 ns

[    0.000006] sched_clock: 64 bits at 1000kHz, resolution 1000ns, wraps every 2199023255500ns

[    0.008490] Console: colour dummy device 80x25

[    0.012791] printk: console [hvc0] enabled

[    0.012791] printk: console [hvc0] enabled

[    0.020930] printk: bootconsole [sbi0] disabled

[    0.020930] printk: bootconsole [sbi0] disabled

[    0.029988] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=4000)

[    0.040032] pid_max: default: 32768 minimum: 301

[    0.045110] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)

[    0.052519] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)

[    0.062019] rcu: Hierarchical SRCU implementation.

[    0.066588] smp: Bringing up secondary CPUs ...

[    0.072158] smp: Brought up 1 node, 4 CPUs

[    0.076781] devtmpfs: initialized

[    0.081749] random: get_random_u32 called from bucket_table_alloc.isra.26+0x4e/0x15c with crng_init=0

[    0.090677] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns

[    0.099959] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)

[    0.163458] NET: Registered protocol family 16

[    0.184274] vgaarb: loaded

[    0.186561] SCSI subsystem initialized

[    0.190358] usbcore: registered new interface driver usbfs

[    0.195523] usbcore: registered new interface driver hub

[    0.200747] usbcore: registered new device driver usb

[    0.206723] clocksource: Switched to clocksource riscv_clocksource

[    0.218070] NET: Registered protocol family 2

[    0.222269] tcp_listen_portaddr_hash hash table entries: 4096 (order: 4, 65536 bytes, linear)

[    0.230410] TCP established hash table entries: 65536 (order: 7, 524288 bytes, linear)

[    0.240180] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)

[    0.250106] TCP: Hash tables configured (established 65536 bind 65536)

[    0.256302] UDP hash table entries: 4096 (order: 5, 131072 bytes, linear)

[    0.263110] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes, linear)

[    0.270541] NET: Registered protocol family 1

[    0.274695] RPC: Registered named UNIX socket transport module.

[    0.280087] RPC: Registered udp transport module.

[    0.284768] RPC: Registered tcp transport module.

[    0.289458] RPC: Registered tcp NFSv4.1 backchannel transport module.

[    0.295908] PCI: CLS 0 bytes, default 64

[    0.361657] workingset: timestamp_bits=62 max_order=21 bucket_order=0

[    0.375467] NFS: Registering the id_resolver key type

[    0.379772] Key type id_resolver registered

[    0.383950] Key type id_legacy registered

[    0.387928] nfs4filelayout_init: NFSv4 File Layout Driver Registering...

[    0.394784] 9p: Installing v9fs 9p2000 file system support

[    0.400507] NET: Registered protocol family 38

[    0.404568] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)

[    0.411883] io scheduler mq-deadline registered

[    0.416396] io scheduler kyber registered

[    0.455433] Serial: 8250/16550 driver, 4 ports, IRQ

```

Kenta Nakajima

2020年11月20日金曜日 22:38:52 UTC+9 yinty...@gmail.com:

[Kenta Nakajima]

I also built linux kernel with `initcall_debug=1 ignore_loglevel=1` in keystone dev branch. The boot.log is attatched in this e-mail.

The sifive unleashed board seems to be freezed in the middle of the `of_platform_serial_driver_init`, and I wonder why the serial8250 probing process failed.

2020年11月25日水曜日 17:20:40 UTC+9 Kenta Nakajima:

[Dayeol Lee]

This is probably the same old issue that bbl had with the device tree.

Did the patches applied to your Linux kernel? The patches are in patches/linux/ (https://github.com/keystone-enclave/keystone/blob/master/patches/linux/linux64.sifive.patch)

This is required because of the change in the Linux made a discrepancy in the way Linux/bbl parse the device tree.

If the patches are there, try these Linux configs.

CONFIG_SOC_VIRT=y

CONFIG_VIRTIO_CONSOLE=y

Hopefully, we can get away from all the device tree issue once we land the OpenSBI port :D

Thanks!

--
You received this message because you are subscribed to the Google Groups "Keystone Enclave Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keystone-enclave-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keystone-enclave-forum/c6bbcd9a-6eef-45fa-bef8-32b440aa49c5n%40googlegroups.com.

[Nakajima Kenta]

Thank you for your reply.

> Did the patches applied to your Linux kernel?

Yes, I built with `cmake .. -DLINUX_SIFIVE=y` in the latest dev branch(e448fa323ae28ec09acea85a3e544aa8f70f2bee), so I think I have no problem.

> If the patches are there, try these Linux configs.

I tried your suggested configuration, but the unleashed board still has been freeze in the middle of the boot stage.

Here is the linux kernel configuration -> https://gist.github.com/knknkn1162/726717569fb8fde4fda5ee5ab290ff74.

Is there anything missing?

Thanks.

[Dayeol Lee]

I created an issue for two duplicated reports:

https://github.com/keystone-enclave/keystone/issues/219

This is indeed a bug, I'll follow up in the GitHub issue thread.