Microwatt with Linux on OrangeCrab
Microwatt is an OpenPOWER CPU design to run on relatively small FPGAs. It can run ppc64le Linux. These instructions show how to get it going on an OrangeCrab FPGA dev board.
The OrangeCrab 85F board has a Lattice ECP5 chip with 85k LUTs, 256MB DDR3 RAM, 16MB SPI flash, and a SD card socket. That's plenty of resources to run Linux at 48MHz! Thanks to the efforts of the Yosys developers it is possible to build the whole FPGA bitstream using open source tools - an advantage compared to many other FPGA boards.
This guide is based on three components:
A JTAG interface: a FTDI FT232H/FT2322H-based interface board, like the Adafruit FT232H Breakout. We will use this for uploading the FPGA bitstream and programming the OrangeCrab flash.
A TTL-serial-to-USB interface, like the Adafruit 954, but any TTL-capable serial device should work. This is for accessing the console of the Microwatt system.
There are a few components involved in getting Microwatt running: the FPGA hardware, the Microwatt bitstream, and the kernel + rootfs boot payload. To build these components, we will also need a set of synthesis tools.
The steps below should cover a moderately straightforward case of booting Linux on a Microwatt CPU.
Firstly, we need to connect the OrangeCrab's JTAG pins to the FT232 JTAG interface. The connection is fairly simple - the header pins are the same order on each end, apart from TDI/TDO which cross over:
We also have connections for the TTL-serial interface to the OrangeCrab board. This provides a console for the Microwatt system:
|0||OC TX / UART RX|
|1||OC RX / UART TX|
The UART will run at 115200 baud.
The picture below shows these connections, with the OrangeCrab on the left, FT232 on the right:
It is also possible to flash a FPGA bitstream with the OrangeCrab's USB DFU bootloader, which doesn't require a JTAG interface. However, the DFU interface cannot currently write a to other flash locations, which precludes writing a Linux image. Future developments on Microwatt may enable alternative ways to provide the OS & software components - for example, using the SD card. We will update this guide when those updates become available.
The Microwatt build process uses a set of digital logic design tools that are
open source, but may not be packaged for your distribution. To make installation
easy, the Yosys folks have assembled a suite of tools, called OSS Cad
Suite. Download a recent
release, and add the
bin/ directory to your
Alternatively you can use tools from HDL Containers, or even build the tools from source. See Software Tools below for further details.
For the Linux kernel build, you'll need a cross-compiler for the 64-bit
little-endian powerpc architecture. Most recent Linux distributions provide a
suitable cross compiler - the package will likely be named
Microwatt for ECP5 targets is built with a Makefile. (Other targets like Xilinx use fusesoc but that doesn't support Yosys)
Yosys produces copious amounts of build logs - it's useful to redirect them to a file. For example:
microwatt.bit file has been build, you can either temporarily
load it to the OrangeCrab:
Alternatively, you can flash the image, which makes the bitstream persist across reboots of the FPGA:
After programming, and with the console UART connected, you should see the Microwatt banner appear:
Welcome to Microwatt ! Soc signature: f00daa5500010001 Soc features: UART DRAM SPIFLASH SDCARD DRAM: 256 MB DRAM INIT: 8 KB CLK: 48 MHz SPI FLASH ID: ef4018 SPI FLASH OFF: 0x400000 bytes
However, since we haven't loaded a kernel yet, it will fail to read its boot payload from flash. You'll probably see something like:
Trying flash... Doesn't look like an elf64 HDR: ff ff ff ff ff ff ff ff Copying payload to DRAM... Booting from DRAM...
- and no further output. In the next step, we will build and flash a suitable kernel to run.
make microwatt.dfu will generate a .dfu file that can be
programmed via OrangeCrab USB with dfu-util. This is a slightly different
process, but may be handy if you're only programming the bitstream.
Microwatt runs mostly an upstream kernel but a few additions are yet to be submitted:
- devicetree files
- SPI flash driver
- litesdcard driver
- faster memcpy
- liteuart powerpc early boot console (for valentyusb serial)
We have a branch containing these changes:
matt/orangecrab, which is
based on Joel Stanley's
We'll be building this with an embedded initramfs filesystem, which provides a minimal userspace suitable for the Microwatt core: rootfs.cpio.zst. Details for building your own rootfs are included in the Custom buildroot rootfs section below.
To build and flash this kernel:
# we build the initramfs into the kernel image # build # write it to the SPI flash
Reset the OrangeCrab and you should see Linux boot on the serial console, using the connection details as above.
In the console, you should see the usual Linux boot banner:
[ 0.000000] Linux version 5.16.0-rc1-00020-gdae78cf662a6 [...] [ 0.000000] Using microwatt machine description
Log output will continue until the system starts userspace code, which will end up with a login prompt:
[ 78.033288] Freeing unused kernel image (initmem) memory: 3548K [ 78.173637] Run /init as init process [ 78.182741] with arguments: [ 78.190565] /init [ 78.196245] with environment: [ 78.204436] HOME=/ [ 78.212077] TERM=linux Starting syslogd: OK Starting klogd: OK Running sysctl: OK Saving random seed: OK Starting network: OK Welcome to Buildroot buildroot login:
At this point, your new Microwatt system is ready to use! With this default
rootfs image, you will be able to login as the
root user, no password is
Custom buildroot rootfs (optional)🔗
You may want to customise the embedded root filesystem used in the Microwatt kernel build. The pre-built rootfs image above was built using Joel's buildroot tree, which has some customisations for the Microwatt CPU implementation.
To download and build a new buildroot-based image, clone the tree and configure:
When configuring for Microwatt, the important options to set are:
Target options -> Target Architecture (PowerPC64 (little endian)) -> Target Architecture Variant (microwatt) Filesystem Images -> cpio the root filesystem (for use as an initial RAM filesystem) -> Compression method (zstd)
Then, to build:
This will download and build a suitable toolchain, the target packages, and build a compressed cpio image. This may take a while!
The resulting image will be generated in
build/images/rootfs.cpio.zst - copy
that to your kernel build directory as above, and rebuild.
If you'd like to replicate the build we have used for the above image:
this uses commit
77853c757of the buildroot tree
download the buildroot configuration: buildroot-microwatt.config, and save as
.configin your buildroot object directory.
The following sections contain some further details on the OrangeCrab port of Microwatt, which may be useful when developing or debugging.
The OrangeCrab has 16MB SPI flash.
|0x0||DFU foboot bootloader bitstream. FPGA loads this on initial power on|
|0x80000||Main bitstream. The DFU bootloader loads this to the FPGA then starts it running|
|0x400000||ELF binary loaded by Microwatt (|
|...||Remainder past the end of the ELF binary can be used for other purposes|
The OrangeCrab board uses a Lattice ECP5 FPGA. One advantage of this part is that it can be targetted by entirely open source tools. It isn't necessary to understand what they're all doing, but the main tools involved are:
Yosys - a synthesis tool that translates a Verilog design (or other HDL) into FPGA hardware cells.
NextPNR - takes the output from Yosys, performs place-and-route to layout cells on the target FPGA.
ecppack - takes nextpnr output and produces a
.bitbitstream file to load on the ECP5.
ecpprog - loads an ECP5 bitstream or flashes the OrangeCrab's SPI flash, via a FT232H USB JTAG adapter. The JTAG pins on the OrangeCrab are directly connected to the ECP5 FPGA, it can allow SPI data access to pass through to the SPI flash. (sysCONFIG docs)
dfu-util - loads an ECP5 bitstream to the default boot location, via the OrangeCrab's DFU bootloader (foboot).
LiteX - DRAM and sdcard functionality in Microwatt is from litedram and litesdcard (part of LiteX). These components are written in Python with Migen and generate Verilog source that can integrate with Microwatt's build. The generated source is checked into the Microwatt repository (litedram, litesdcard, it only needs to be regenerated when upstream LiteX components need updating.
Install Alternative 1: OSS Cad Suite🔗
OSS Cad Suite is a tarball of prebuilt binaries
for Yosys and various other tools. Grab the latest
nightly build and put it in your
PATH. Note that it
includes its own
lsusb which may be a surprise. Only x64 Linux builds have the necessary GHDL
Install Alternative 2: Docker🔗
Building Microwatt with
make DOCKER=1 will use tools in Docker images from
HDL Containers. Those tools are updated frequently.
Install Alternative 3: Build from source🔗
The tools can be built from source. Summon-FPGA-Tools is one option.
Yosys and nextpnr print copious log information, most of it can be ignored unless investigating a problem. There are a few parts that are generally useful.
The most important figure is the final
systemclk Max Frequency:
Warning: Max frequency for clock '$glbnet$system_clk': 44.93 MHz (FAIL at 48.00 MHz)
In this example the timing analysis indicates that the layout doesn't meet the desired 48MHz, though in practise a small difference will usually work fine. The Slack Histogram shows the distribution of timing slack - values are in nanoseconds, negative is failing.
Info: Slack histogram: Info: legend: * represents 148 endpoint(s) Info: + represents [1,148) endpoint(s) Info: [ -1424, 696) |+ Info: [ 696, 2816) |**+ Info: [ 2816, 4936) |*********+ Info: [ 4936, 7056) |***************+ Info: [ 7056, 9176) |********************+ Info: [ 9176, 11296) |********************+ Info: [ 11296, 13416) |***********************+ Info: [ 13416, 15536) |*********************************************+ Info: [ 15536, 17656) |***********************************+ Info: [ 17656, 19776) |************************************************************ Info: [ 19776, 21896) |*******+
It is possible to reduce the target frequency though below ~42MHz will prevent DRAM working - this may be board dependent. Note that valentyusb requires a fixed 48MHz if using the USB serial port.
There is a fair amount of random variation in output quality between different
runs of yosys/nextpnr - the achieved MHz may vary between runs. Sometimes a
small change to the design or parameters will randomly gain a few MHz.
There is also a
nextpnr-ecp5 -r option to randomise seed values.
Another parameter to look at is utilisation. The important cell types to look at
TRELLIS_SLICE which are LUTs (lookup up table logic) and flip-flops; and
DP16KD which are block RAM.
Info: Device utilisation: Info: TRELLIS_SLICE: 32038/41820 76% Info: TRELLIS_IO: 79/ 365 21% Info: DCCA: 4/ 56 7% Info: DP16KD: 41/ 208 19% Info: MULT18X18D: 33/ 156 21% Info: ALU54B: 0/ 78 0% ... various other cells
Thanks to Joel for his instructions for Linux on Microwatt.
2022-02-02: Fixed incorrect swapped TX/RX pins