Expanding Deku - Code Construct

Deku is an open-source Rust crate which simplifies the process of serialisation and deserialisation of binary data structures, preventing tedious manual parser-writing and inevitable bugs.

Andrew at Code Construct has contributed to the project, expanding it for use in embedded systems.

Contributing to this project is part of our support for the open-source software community. We're proud to be growing the capacity for ergonomic and reliable code in embedded systems.

The situation🔗

Our client monitors the health and status of their servers, even when those servers are turned off or not working well, without relying on their host or primary system. This is called out-of-band monitoring.

To do this they communicate with their NVMe (Non-Volatile Memory Express) drives using the NVMe Management Interface (NVMe-MI). NVMe-MI as an "upper layer" protocol builds on the Management Component Transport Protocol (MCTP) for its transport. You can read more about our work with this tooling here.

While current NVMe-MI standards typically prescribe the use of I2C as the MCTP physical transport, industry requirements are motivating the use of USB instead. There is no commercially available hardware that can test NVMe-MI over USB at time of writing, so we wrote firmware for a microcontroller to be an NVMe-MI emulator, allowing us to do so before the client invests in new hardware

The NVMe-MI standard specifies a precise structure for messages, that is specific to the standard and not wrapped in a generic serialisation format. Making an emulator in this situation involves managing conversions between the wire- and in-memory- representation for each message, which is detailed and time-consuming. For this reason, we used Deku.

Why did we expand Deku?🔗

Deku automatically derives symmetric read/write methods for structs and enums, allowing developers to focus on how data is represented rather than the underlying parsing code.

It is declarative, allowing users to define the data structures and how each field should be treated, and symmetric, meaning the process to serialise and deserialise should match and be reliable. Together with Rust's safety features, this minimises the bugs that can occur with untrusted inputs, and spares us having to write parsers to handle every edge case.

However, Deku's implementation required runtime memory allocation in many parts, but there's no heap available in our case. So Andrew set about removing Deku's reliance on a heap.

In doing this, he found solutions for problems arising from a common issue when integrating tools into embedded systems - handling dynamic inputs under size constraints.

If you'd like to learn about these problems and solutions, read on. Otherwise, you can skip to How to to get started.

Problems and solutions🔗

This change was about making it clear where Deku relied on a heap, and adding code to handle cases where it was lacking. Andrew added the alloc feature flag, so that users can configure their build for either case - using a heap, or not.

Across the code base, he either removed reliance on allocation or added conditions to make sure anything relying on it wouldn't run when it was disabled. This meant updating use statements to refer to the core crate in place of alloc or std in some cases, and preventing reliance on external crates that themselves use heap allocation, when the alloc feature flag is disabled.

A more significant problem was Deku's error types included descriptive strings for problems encountered during processing. These error strings included values that change with context, and formatting the result required allocation of memory from the assumed heap. Deku also made these error strings efficient by using a copy-on-write (CoW) type - without the heap we don't have this luxury, or the luxury of string formatting.

A related problem was that Deku provides assertion attributes to allow user-defined safety properties during parsing. There is also an assertion error type, returned when assertion attributes fail. The assertion error's descriptive string embeds the failing expression. Careful programming for embedded targets can yield many unique assertion error strings, which significantly increases the final binary size, which we want to avoid in embedded systems.

Andrew fixed both these related problems by adding a new feature, descriptive-errors.

This depends on alloc, and divides the library API in two: When descriptive-errors is enabled, the original copy-on-write dynamic type is retained, but when disabled we fall back to the static string type, removing the need for the heap. Further, by removing string formatting, the descriptive strings for user-defined assertion attributes are no-longer unique; they are coalesced, and we retain the small binary size property for embedded systems.

A further problem was that Deku allows a user-specified number of bits or bytes for fields and padding. To handle padding, Deku's implementation chose to dynamically allocate memory of the requested size, which it used to either discarded data into, or write zeros from, for the purpose of alignment.

Maintaining the ability to accept user-specified sizes without the heap tips the scales on the memory-vs-processing trade-off: Deku's implementation now uses a fixed-sized array on the stack, and loops over this until the specified padding size is reached.

Of course, no contribution to a project is complete until all the tests pass, and to achieve that Andrew edited the library, integration, and documentation tests to make sure they ran (and passed) with and without the alloc feature flag.

How to configure Deku for use in embedded systems🔗

If you're working in embedded systems and you're keen to use Deku in your work, there are a few tweaks to make to your Cargo features list.

To add Deku as a dependency, run the following command from your project directory:

$ cargo add deku --no-default-features

Then check the Cargo.toml file. It should include:

[dependencies]
deku = { version = "0.20", default-features = false }

The default-features attribute is set to false because this includes the alloc feature, and other features that depend on it. alloc is a feature added in this release, which lets users specify whether they will use runtime memory allocation in their build.

You can specify another version if you choose, but note that you need version 0.20 or later to work in environments that lack a heap.

Future changes🔗

Right now the implementation of Deku without runtime memory allocation only works at the byte level, the bits feature is still dependent on the alloc feature.

However, watch this space, as we have work in progress to remove that dependency too.