Microwatt is an OpenPOWER CPU design to run on relatively small FPGAs. It can run ppc64le Linux. These instructions show how to get it going on an OrangeCrab FPGA dev board.

The OrangeCrab 85F board has a Lattice ECP5 chip with 85k LUTs, 256MB DDR3 RAM, 16MB SPI flash, and a SD card socket. That's plenty of resources to run Linux at 48MHz! Thanks to the efforts of the Yosys developers it is possible to build the whole FPGA bitstream using open source tools - an advantage compared to many other FPGA boards.

Hardware🔗

This guide is based on three components:

The FPGA: an OrangeCrab 85F, available from 1bitsquared or element14.
A JTAG interface: a FTDI FT232H/FT2322H-based interface board, like the Adafruit FT232H Breakout. We will use this for uploading the FPGA bitstream and programming the OrangeCrab flash.
A TTL-serial-to-USB interface, like the Adafruit 954, but any TTL-capable serial device should work. This is for accessing the console of the Microwatt system.

Steps🔗

There are a few components involved in getting Microwatt running: the FPGA hardware, the Microwatt bitstream, and the kernel + rootfs boot payload. To build these components, we will also need a set of synthesis tools.

The steps below should cover a moderately straightforward case of booting Linux on a Microwatt CPU.

Hardware setup🔗

Firstly, we need to connect the OrangeCrab's JTAG pins to the FT232 JTAG interface. The connection is fairly simple - the header pins are the same order on each end, apart from TDI/TDO which cross over:

OC JTAG	FT232H
GND	GND
TCK	D0
TDO	D2
TDI	D1
TMS	D3

We also have connections for the TTL-serial interface to the OrangeCrab board. This provides a console for the Microwatt system:

OC IO	UART
0	OC TX / UART RX
1	OC RX / UART TX
GND	GND

The UART will run at 115200 baud.

The picture below shows these connections, with the OrangeCrab on the left, FT232 on the right:

OrangeCrab JTAG connection to FT232H board

It is also possible to flash a FPGA bitstream with the OrangeCrab's USB DFU bootloader, which doesn't require a JTAG interface. However, the DFU interface cannot currently write a to other flash locations, which precludes writing a Linux image. Future developments on Microwatt may enable alternative ways to provide the OS & software components - for example, using the SD card. We will update this guide when those updates become available.

Install software🔗

The Microwatt build process uses a set of digital logic design tools that are open source, but may not be packaged for your distribution. To make installation easy, the Yosys folks have assembled a suite of tools, called OSS Cad Suite. Download a recent release, and add the bin/ directory to your PATH.

Alternatively you can use tools from HDL Containers, or even build the tools from source. See Software Tools below for further details.

For the Linux kernel build, you'll need a cross-compiler for the 64-bit little-endian powerpc architecture. Most recent Linux distributions provide a suitable cross compiler - the package will likely be named gcc-powerpc64le-linux-gnu.

Build Microwatt🔗

Microwatt for ECP5 targets is built with a Makefile. (Other targets like Xilinx use fusesoc but that doesn't support Yosys)

git clone https://github.com/antonblanchard/microwatt
cd microwatt
make microwatt.bit   # or "make DOCKER=1 microwatt.bit" to use container tools

Yosys produces copious amounts of build logs - it's useful to redirect them to a file. For example:

make microwatt.bit 2>&1 | tee ~/tmp/build1.txt

Once the microwatt.bit file has been build, you can either temporarily load it to the OrangeCrab:

make ecpprog        # program via JTAG

Alternatively, you can flash the image, which makes the bitstream persist across reboots of the FPGA:

make ecpflash       # flash via JTAG

After programming, and with the console UART connected, you should see the Microwatt banner appear:

Welcome to Microwatt !

 Soc signature: f00daa5500010001
  Soc features: UART DRAM SPIFLASH SDCARD
          DRAM: 256 MB
     DRAM INIT: 8 KB
           CLK: 48 MHz
  SPI FLASH ID: ef4018
 SPI FLASH OFF: 0x400000 bytes

However, since we haven't loaded a kernel yet, it will fail to read its boot payload from flash. You'll probably see something like:

Trying flash...
Doesn't look like an elf64
HDR: ff ff ff ff ff ff ff ff
Copying payload to DRAM...
Booting from DRAM...

- and no further output. In the next step, we will build and flash a suitable kernel to run.

Alternatively make microwatt.dfu will generate a .dfu file that can be programmed via OrangeCrab USB with dfu-util. This is a slightly different process, but may be handy if you're only programming the bitstream.

Build Linux🔗

Microwatt runs mostly an upstream kernel but a few additions are yet to be submitted:

devicetree files
SPI flash driver
litesdcard driver
faster memcpy
liteuart powerpc early boot console (for valentyusb serial)

We have a branch containing these changes: matt/orangecrab, which is based on Joel Stanley's microwatt branch.

We'll be building this with an embedded initramfs filesystem, which provides a minimal userspace suitable for the Microwatt core: rootfs.cpio.zst. Details for building your own rootfs are included in the Custom buildroot rootfs section below.

To build and flash this kernel:

git clone https://github.com/mkj/linux -b matt/orangecrab
cd linux

make CROSS_COMPILE=powerpc64le-linux-gnu- ARCH=powerpc O=build microwatt_defconfig

# we build the initramfs into the kernel image
wget -O build/rootfs.cpio.zst https://codeconstruct.com.au/docs/microwatt-orangecrab/rootfs-20220114.cpio.zst

echo 'CONFIG_INITRAMFS_SOURCE="rootfs.cpio.zst"' >> build/.config

# build
make CROSS_COMPILE=powerpc64le-linux-gnu- ARCH=powerpc O=build -j8

# write it to the SPI flash
ecpprog -o 0x400000 build/arch/powerpc/boot/dtbImage.microwatt-oc02.elf

Reset the OrangeCrab and you should see Linux boot on the serial console, using the connection details as above.

In the console, you should see the usual Linux boot banner:

[    0.000000] Linux version 5.16.0-rc1-00020-gdae78cf662a6 [...]
[    0.000000] Using microwatt machine description

Log output will continue until the system starts userspace code, which will end up with a login prompt:

[   78.033288] Freeing unused kernel image (initmem) memory: 3548K
[   78.173637] Run /init as init process
[   78.182741]   with arguments:
[   78.190565]     /init
[   78.196245]   with environment:
[   78.204436]     HOME=/
[   78.212077]     TERM=linux
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Saving random seed: OK
Starting network: OK

Welcome to Buildroot
buildroot login:

At this point, your new Microwatt system is ready to use! With this default rootfs image, you will be able to login as the root user, no password is required.

Custom buildroot rootfs (optional)🔗

You may want to customise the embedded root filesystem used in the Microwatt kernel build. The pre-built rootfs image above was built using Joel's buildroot tree, which has some customisations for the Microwatt CPU implementation.

To download and build a new buildroot-based image, clone the tree and configure:

git clone https://github.com/shenki/buildroot -b microwatt
cd buildroot
make O=build menuconfig

When configuring for Microwatt, the important options to set are:

Target options
 -> Target Architecture (PowerPC64 (little endian))
 -> Target Architecture Variant (microwatt)

Filesystem Images
 -> cpio the root filesystem (for use as an initial RAM filesystem)
 -> Compression method (zstd)

Then, to build:

make O=build

This will download and build a suitable toolchain, the target packages, and build a compressed cpio image. This may take a while!

The resulting image will be generated in build/images/rootfs.cpio.zst - copy that to your kernel build directory as above, and rebuild.

If you'd like to replicate the build we have used for the above image:

this uses commit 77853c757 of the buildroot tree
download the buildroot configuration: buildroot-microwatt.config, and save as .config in your buildroot object directory.

Technical details🔗

The following sections contain some further details on the OrangeCrab port of Microwatt, which may be useful when developing or debugging.

Flash Layout🔗

The OrangeCrab has 16MB SPI flash.

Address
0x0	DFU foboot bootloader bitstream. FPGA loads this on initial power on
0x80000	Main bitstream. The DFU bootloader loads this to the FPGA then starts it running
0x400000	ELF binary loaded by Microwatt (`dtbImage.microwatt-oc02.elf`)
...	Remainder past the end of the ELF binary can be used for other purposes

Software Tools🔗

The OrangeCrab board uses a Lattice ECP5 FPGA. One advantage of this part is that it can be targetted by entirely open source tools. It isn't necessary to understand what they're all doing, but the main tools involved are:

Yosys - a synthesis tool that translates a Verilog design (or other HDL) into FPGA hardware cells.
GHDL - a VHDL compiler and tools. The plugin allows using VHDL source as input for Yosys synthesis. Most of Microwatt is written in VHDL.
NextPNR - takes the output from Yosys, performs place-and-route to layout cells on the target FPGA.
ecppack - takes nextpnr output and produces a .bit bitstream file to load on the ECP5.
ecpprog - loads an ECP5 bitstream or flashes the OrangeCrab's SPI flash, via a FT232H USB JTAG adapter. The JTAG pins on the OrangeCrab are directly connected to the ECP5 FPGA, it can allow SPI data access to pass through to the SPI flash. (sysCONFIG docs)
dfu-util - loads an ECP5 bitstream to the default boot location, via the OrangeCrab's DFU bootloader (foboot).
LiteX - DRAM and sdcard functionality in Microwatt is from litedram and litesdcard (part of LiteX). These components are written in Python with Migen and generate Verilog source that can integrate with Microwatt's build. The generated source is checked into the Microwatt repository (litedram, litesdcard, it only needs to be regenerated when upstream LiteX components need updating.

Install Alternative 1: OSS Cad Suite🔗

OSS Cad Suite is a tarball of prebuilt binaries for Yosys and various other tools. Grab the latest nightly build and put it in your PATH. Note that it includes its own lsusb which may be a surprise. Only x64 Linux builds have the necessary GHDL plugin.

Install Alternative 2: Docker🔗

Building Microwatt with make DOCKER=1 will use tools in Docker images from HDL Containers. Those tools are updated frequently.

Install Alternative 3: Build from source🔗

The tools can be built from source. Summon-FPGA-Tools is one option.

Build logs🔗

Yosys and nextpnr print copious log information, most of it can be ignored unless investigating a problem. There are a few parts that are generally useful.

The most important figure is the final systemclk Max Frequency:

Warning: Max frequency for clock      '$glbnet$system_clk': 44.93 MHz (FAIL at 48.00 MHz)

In this example the timing analysis indicates that the layout doesn't meet the desired 48MHz, though in practise a small difference will usually work fine. The Slack Histogram shows the distribution of timing slack - values are in nanoseconds, negative is failing.

Info: Slack histogram:
Info:  legend: * represents 148 endpoint(s)
Info:          + represents [1,148) endpoint(s)
Info: [ -1424,    696) |+
Info: [   696,   2816) |**+
Info: [  2816,   4936) |*********+
Info: [  4936,   7056) |***************+
Info: [  7056,   9176) |********************+
Info: [  9176,  11296) |********************+
Info: [ 11296,  13416) |***********************+
Info: [ 13416,  15536) |*********************************************+
Info: [ 15536,  17656) |***********************************+
Info: [ 17656,  19776) |************************************************************
Info: [ 19776,  21896) |*******+

It is possible to reduce the target frequency though below ~42MHz will prevent DRAM working - this may be board dependent. Note that valentyusb requires a fixed 48MHz if using the USB serial port.

There is a fair amount of random variation in output quality between different runs of yosys/nextpnr - the achieved MHz may vary between runs. Sometimes a small change to the design or parameters will randomly gain a few MHz. There is also a nextpnr-ecp5 -r option to randomise seed values.

Another parameter to look at is utilisation. The important cell types to look at are TRELLIS_SLICE which are LUTs (lookup up table logic) and flip-flops; and DP16KD which are block RAM.

Info: Device utilisation:
Info:          TRELLIS_SLICE: 32038/41820    76%
Info:             TRELLIS_IO:    79/  365    21%
Info:                   DCCA:     4/   56     7%
Info:                 DP16KD:    41/  208    19%
Info:             MULT18X18D:    33/  156    21%
Info:                 ALU54B:     0/   78     0%
... various other cells

Acknowledgements🔗

Thanks to Joel for his instructions for Linux on Microwatt.

Revision History🔗

2022-02-02: Fixed incorrect swapped TX/RX pins

Microwatt with Linux on OrangeCrab