July 09, 2017 posted by Jared McNeill

A new SUNXI evbarm kernel has appeared recently in NetBSD -current with support for boards based on the Allwinner H3 system on a chip (SoC). The H3 SoC is a quad-core Cortex-A7 SoC designed primarily for set-top boxes, but has managed to find its way into many single-board computers (SBC). This is one of the first evbarm ports built from the ground up with device tree support, which helps us to use a single kernel config to support many different boards.

To get these boards up and running, first we need to deal with low-level startup code. For the SUNXI kernel this currently lives in sys/arch/evbarm/sunxi/. The purpose of this code is fairly simple; initialize the boot CPU and initialize the MMU so we can jump to the kernel. The initial MMU configuration needs to cover a few things -- early on we need to be able to access the kernel, UART debug console, and the device tree blob (DTB) passed in from U-Boot. We wrap the kernel in a U-Boot header that claims to be a Linux kernel; this is no accident! This tells U-Boot to use the Linux boot protocol when loading the kernel, which ensures that the DTB (loaded by U-Boot) is processed and passed to us in r2.

Once the CPU and MMU are ready, we jump to the generic ARM FDT implementation of initarm in sys/arch/evbarm/fdt/fdt_machdep.c. The first thing this code does is validate and relocate the DTB data. After it has been relocated, we compare the compatible property of the root node in the device tree with the list of ARM platforms compiled into the kernel. The Allwinner sunxi platform code lives in sys/arch/arm/sunxi/sunxi_platform.c. The sunxi platform code provides SoC-specific versions of code needed early at boot. We need to know how to initialize the debug console, spin up application CPUs, reset the board, etc.

Instead of writing H3-specific code for spinning up application CPUs, I took advantage of U-Boot's Power State Coordination Interface implementation. A psci(4) driver was added and the allwinner,sun8i-h3 platform code was modified to use this code to start up all processors.

With a bit of luck, we're now booting and enumerating devices. Apart from a few devices, almost nothing works yet as we are missing a driver for the CCU. The CCU in the Allwinner H3 SoC controls PLLs and most of the clock generation, division, muxing, and gating. Since there are many similarities between Allwinner SoCs, I opted to write generic CCU code and then SoC-specific frontends. The resulting code lives in sys/arch/arm/sunxi/; generic code as sunxi_ccu.c and H3-specific code in sun8i_h3_ccu.c.

Now we have a CCU driver, we can attach a com(4) and have a valid console device.

After this, it's a matter of writing drivers and/or adapting existing code to attach to fdtbus based on the bindings used in the DTB. For cases where we had a compatible driver in the old Allwinner port, I opted to make a copy of the code and FDT-ize it. A few reasons for this -- 1) the old drivers have CCU-specific code with per-SoC ifdefs scattered throughout, 2) I didn't want to break existing kernels, and 3) long term goal is to move the SoCs supported by the old code over to the new code (this process has already started with the Allwinner A31 port).

So what do we get out of this? This is a step towards being able to ship a GENERIC evbarm kernel. I developed the H3 port on two boards, the NanoPi NEO and Orange Pi Plus 2E, but since then users on port-arm@ have been reporting success on many other H3 boards, all from a single kernel config. In addition, I've added support for other Allwinner SoCs (sun8i-a83t, sun6i-a31) to the kernel and have tested booting the same kernel across all 3 SoCs.

Orange Pi Plus 2E boot log is below.

U-Boot SPL 2017.05 (Jul 01 2017 - 17:11:09) DRAM: 2048 MiB Trying to boot from MMC1 U-Boot 2017.05 (Jul 01 2017 - 17:11:09 -0300) Allwinner Technology CPU: Allwinner H3 (SUN8I 1680) Model: Xunlong Orange Pi Plus 2E I2C: ready DRAM: 2 GiB MMC: SUNXI SD/MMC: 0, SUNXI SD/MMC: 1 In: serial Out: serial Err: serial Net: phy interface7 eth0: ethernet@1c30000 starting USB... USB0: USB EHCI 1.00 USB1: USB OHCI 1.0 USB2: USB EHCI 1.00 USB3: USB OHCI 1.0 USB4: USB EHCI 1.00 USB5: USB OHCI 1.0 scanning bus 0 for devices... 2 USB Device(s) found scanning bus 2 for devices... 1 USB Device(s) found scanning bus 4 for devices... 1 USB Device(s) found scanning usb for storage devices... 0 Storage Device(s) found Hit any key to stop autoboot: 0 reading netbsd.ub 6600212 bytes read in 334 ms (18.8 MiB/s) reading sun8i-h3-orangepi-plus2e.dtb 16775 bytes read in 49 ms (334 KiB/s) ## Booting kernel from Legacy Image at 42000000 ... Image Name: NetBSD/sunxi 8.99.1 Image Type: ARM Linux Kernel Image (uncompressed) Data Size: 6600148 Bytes = 6.3 MiB Load Address: 40008000 Entry Point: 40008000 Verifying Checksum ... OK ## Flattened Device Tree blob at 43000000 Booting using the fdt blob at 0x43000000 Loading Kernel Image ... OK Loading Device Tree to 49ff8000, end 49fff186 ... OK Starting kernel ... [ Kernel symbol table missing! ] Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017 The NetBSD Foundation, Inc. All rights reserved. Copyright (c) 1982, 1986, 1989, 1991, 1993 The Regents of the University of California. All rights reserved. NetBSD 8.99.1 (SUNXI) #304: Sat Jul 8 11:01:22 ADT 2017 jmcneill@undine.invisible.ca:/usr/home/jmcneill/netbsd/cvs-src/sys/arch/evbarm/compile/obj/SUNXI total memory = 2048 MB avail memory = 2020 MB sysctl_createv: sysctl_create(machine_arch) returned 17 armfdt0 (root) fdt0 at armfdt0: Xunlong Orange Pi Plus 2E fdt1 at fdt0 fdt2 at fdt0 cpus0 at fdt0 cpu0 at cpus0: Cortex-A7 r0p5 (Cortex V7A core) cpu0: DC enabled IC enabled WB disabled EABT branch prediction enabled cpu0: 32KB/32B 2-way L1 VIPT Instruction cache cpu0: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache cpu0: 512KB/64B 8-way write-through L2 PIPT Unified cache vfp0 at cpu0: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals cpu1 at cpus0 cpu2 at cpus0 cpu3 at cpus0 gic0 at fdt1: GIC armgic0 at gic0: Generic Interrupt Controller, 160 sources (150 valid) armgic0: 16 Priorities, 128 SPIs, 7 PPIs, 15 SGIs fclock0 at fdt2: 24000000 Hz fixed clock ffclock0 at fdt2: x1 /1 fixed-factor clock fclock1 at fdt2: 32768 Hz fixed clock sunxigates0 at fdt2 sunxiresets0 at fdt1 gtmr0 at fdt0: Generic Timer armgtmr0 at gtmr0: ARMv7 Generic 64-bit Timer (24000 kHz) armgtmr0: interrupting on irq 27 sunxigpio0 at fdt1: PIO gpio0 at sunxigpio0: 94 pins sunxigpio1 at fdt1: PIO gpio1 at sunxigpio1: 12 pins sun8ih3ccu0 at fdt1: H3 CCU fregulator0 at fdt0: vcc3v3 fregulator1 at fdt0: gmac-3v3 fregulator2 at fdt0: vcc3v0 fregulator3 at fdt0: vcc5v0 sunxiusbphy0 at fdt1: USB PHY /soc/dma-controller@01c02000 at fdt1 not configured /soc/codec-analog@01f015c0 at fdt1 not configured /clocks/ir_clk@01f01454 at fdt2 not configured sunxiemac0 at fdt1: EMAC sunxiemac0: interrupting on GIC irq 114 rgephy0 at sunxiemac0 phy 0: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 5 rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto rgephy1 at sunxiemac0 phy 1: RTL8169S/8110S/8211 1000BASE-T media interface, rev. 5 rgephy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto psci0 at fdt0: PSCI 0.1 gpioleds0 at fdt0: orangepi:green:pwr orangepi:red:status gpiokeys0 at fdt0: sw4 sunximmc0 at fdt1: SD/MMC controller sunximmc0: interrupting on GIC irq 92 sunximmc1 at fdt1: SD/MMC controller sunximmc1: interrupting on GIC irq 93 sunximmc2 at fdt1: SD/MMC controller sunximmc2: interrupting on GIC irq 94 ehci0 at fdt1: EHCI ehci0: interrupting on GIC irq 106 ehci0: 1 companion controller, 1 port usb0 at ehci0: USB revision 2.0 ohci0 at fdt1: OHCI ohci0: interrupting on GIC irq 107 ohci0: OHCI version 1.0 usb1 at ohci0: USB revision 1.0 ehci1 at fdt1: EHCI ehci1: interrupting on GIC irq 108 ehci1: 1 companion controller, 1 port usb2 at ehci1: USB revision 2.0 ohci1 at fdt1: OHCI ohci1: interrupting on GIC irq 109 ohci1: OHCI version 1.0 usb3 at ohci1: USB revision 1.0 ehci2 at fdt1: EHCI ehci2: interrupting on GIC irq 110 ehci2: 1 companion controller, 1 port usb4 at ehci2: USB revision 2.0 ohci2 at fdt1: OHCI ohci2: interrupting on GIC irq 111 ohci2: OHCI version 1.0 usb5 at ohci2: USB revision 1.0 /soc/timer@01c20c00 at fdt1 not configured /soc/watchdog@01c20ca0 at fdt1 not configured /soc/codec@01c22c00 at fdt1 not configured com0 at fdt1: ns16550a, working fifo com0: console com0: interrupting on GIC irq 32 sunxirtc0 at fdt1: RTC /soc/ir@01f02000 at fdt1 not configured cpu3: Cortex-A7 r0p5 (Cortex V7A core) cpu3: DC enabled IC enabled WB disabled EABT branch prediction enabled cpu3: 32KB/32B 2-way L1 VIPT Instruction cache cpu3: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache cpu3: 512KB/64B 8-way write-through L2 PIPT Unified cache vfp3 at cpu3: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals cpu1: Cortex-A7 r0p5 (Cortex V7A core) cpu1: DC enabled IC enabled WB disabled EABT branch prediction enabled cpu1: 32KB/32B 2-way L1 VIPT Instruction cache cpu1: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache cpu1: 512KB/64B 8-way write-through L2 PIPT Unified cache vfp1 at cpu1: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals cpu2: Cortex-A7 r0p5 (Cortex V7A core) cpu2: DC enabled IC enabled WB disabled EABT branch prediction enabled cpu2: 32KB/32B 2-way L1 VIPT Instruction cache cpu2: 32KB/64B 4-way write-back-locking-C L1 PIPT Data cache cpu2: 512KB/64B 8-way write-through L2 PIPT Unified cache vfp2 at cpu2: NEON MPE (VFP 3.0+), rounding, NaN propagation, denormals sdmmc0 at sunximmc0 sdmmc1 at sunximmc1 sdmmc2 at sunximmc2 uhub0 at usb0: Generic (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1 uhub1 at usb2: Generic (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1 uhub2 at usb3: Generic (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1 uhub3 at usb1: Generic (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1 uhub4 at usb4: Generic (0000) EHCI root hub (0000), class 9/0, rev 2.00/1.00, addr 1 uhub5 at usb5: Generic (0000) OHCI root hub (0000), class 9/0, rev 1.00/1.00, addr 1 ld2 at sdmmc2: <0x15:0x0100:AWPD3R:0x00:0xec19649f:0x000> sdmmc0: SD card status: 4-bit, C10, U1, V10 ld0 at sdmmc0: <0x27:0x5048:2&DRP:0x07:0x01c828bc:0x109> ld2: 14910 MB, 7573 cyl, 64 head, 63 sec, 512 bytes/sect x 30535680 sectors ld0: 15288 MB, 7765 cyl, 64 head, 63 sec, 512 bytes/sect x 31309824 sectors (manufacturer 0x24c, product 0xf179, standard function interface code 0x7)at sdmmc1 function 1 not configured ld2: mbr partition exceeds disk size ld0: 4-bit width, High-Speed/SDR25, 50.000 MHz ld2: 8-bit width, 52.000 MHz urtwn0 at uhub0 port 1 urtwn0: Realtek (0xbda) 802.11n NIC (0x8179), rev 2.00/0.00, addr 2 urtwn0: MAC/BB RTL8188EU, RF 6052 1T1R, address e8:de:27:16:0c:81 urtwn0: 1 rx pipe, 2 tx pipes urtwn0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps urtwn0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps boot device: ld0 root on ld0a dumps on ld0b root file system type: ffs kern.module.path=/stand/evbarm/8.99.1/modules WARNING: clock lost 6398 days WARNING: using filesystem time WARNING: CHECK AND RESET THE DATE! Sat Jul 8 11:05:42 ADT 2017 Starting root file system check: /dev/rld0a: file system is clean; not checking Not resizing /: already correct size swapctl: adding /dev/ld0b as swap device at priority 0 Starting file system checks: /dev/rld0e: 22 files, 32340 free (8085 clusters) random_seed: /var/db/entropy-file: Not present Setting tty flags. Setting sysctl variables: ddb.onpanic: 1 -> 1 Starting network. Hostname: sunxi IPv6 mode: host Configuring network interfaces:. Adding interface aliases:. Waiting for DAD to complete for statically configured addresses... Starting dhcpcd. Starting mdnsd. Building databases: dev, utmp, utmpx. Starting syslogd. Mounting all file systems... Clearing temporary files. Updating fontconfig cache: done. Creating a.out runtime link editor directory cache. Checking quotas: done. Setting securelevel: kern.securelevel: 0 -> 1 Starting virecover. Checking for core dump... savecore: no core dump Starting devpubd. Starting local daemons:. Updating motd. Starting ntpd. Jul 8 11:05:58 sunxi ntpd[595]: ntp_rlimit: Cannot set RLIMIT_STACK: Invalid argument Starting sshd. Starting inetd. Starting cron. Sat Jul 8 11:06:02 ADT 2017 NetBSD/evbarm (sunxi) (console) login: