[RFC] OTTF / test harness binary protocol

[jwnrt]

This RFC proposes changes to how OTTF communicates with test harnesses.

The key points are:

• Introduction of a new binary SPI protocol.
• Deprecation of SPI console.
• Deprecation of UJSON.

The idea is to use UART exclusively for asynchronous logs and SPI for binary data transfers and synchronisation.

Current communication flow

OTTF communicates with the host test harness via a console which may use either the UART or SPI device as its transport.

Logs, data, and synchronisation messages are sent over the console. Data is encoded as JSON and serialized / deserialized at each end of the transport (UJSON). The UART console is asynchronous but uses software flow control to mitigate data loss. The SPI console is synchronous and requires polling from the host in order for the device to send logs. There is an optional TX notification GPIO available for the SPI console.

Problems

Serializing data as JSON is required over the UART console to allow for in-band flow control support. The (de)serialization adds complexity and overheads. The in-band software flow control is not foolproof and has shown reliability problems in the past with lost data. Increasing reliability requires slowing the data rate by "pacing" each byte to give time for flow control bytes to arrive.

The SPI console uses a (subjectively) complex framing protocol to transport data in chunks. Asynchronous logs from the host cannot be sent until the host polls for them. An extra GPIO pin can be used to notify the host of awaiting data, but adds further complexity. Data is still serialized as UJSON over the SPI console potentially unnecessarily.

Binary SPI protocol

This RFC proposes splitting the work of logs and data between two transports.

UART will be used exclusively for informative logs and not for data transfers or synchronisation. Device-side tests should not rely on the host receiving UART logs for test success.

The SPI will be used to transfer binary data between the device and host and for synchronising control flow. The protocol is built on the following SPI commands sent from the host to device:

1. READ <addr> - read bytes from addr on the device.
2. WRITE <addr> - write bytes to addr on the device.
3. SYNC <code> - synchronise with the device using some code.
4. READ_STATUS - poll the busy status of OTTF synchronisation.

Commands 1-3 will interrupt OTTF and cause an upload. Command 4 will be handled by the SPI device hardware.

The host is able to read and write data to any address at any time. The intended use is writing test data and reading test results to/from SRAM. Known SRAM addresses are obtained from the ELF as is currently done with UJSON. Struct layouts are generated for Rust from the C using bindgen unlike the macro generation system from UJSON.

Synchronisation

The SYNC and READ_STATUS commands are used together for synchronisation.

At startup, the device:

1. Configures the SPI device to upload the SYNC command and set the busy bit.
2. When SYNC is uploaded, the interrupt handler writes the code to a global OTTF variable.

To synchronise, the host:

1. Sends a SYNC <code> command.
2. Sends a READ_STATUS command and continuously reads until the device is no longer busy.
3. The host now knows that the device has synchronised with the same code.

To synchronise, the device:

1. Continuously reads the global OTTF variable until it matches the expected code.
2. Clears the busy bit in the SPI device.
3. The device now knows that the host has also synchronised with the same code.

Test results

Instead of reading PASS! and FAIL! messages from the UART, the host should synchronise on a special end-of-test code and then read the test status from SRAM.

Similarly the host may choose to read out coverage data from a known location in SRAM at the end of the test. This replaces the current coverage data transfer mechanism.

Migration plan

The following is the proposed migration plan for the master branch:

1. Introduce the binary SPI protocol to OTTF and OpenTitanLib.
2. Introduce an OTTF API for synchronisation and migrate all tests.
3. Depending on the console that is used:
   • For UART: log over UART but synchronise and send binary data via the SPI.
   • For SPI: log, synchronise, and send UJSON data over the SPI console still.
4. Deprecate and remove the SPI console.
5. Deprecate and remove UJSON.

Benefits

The new flow should be faster as the UART does not need to be paced, the SPI does not need to be framed, and JSON (de)serialization is no longer needed.

The protocol complexity should be reduced as the SPI no longer requires framing or a GPIO to notify of asynchronous data from the device. The new protocol is expected to be simpler than the current SPI console protocol.

We expect improved reliability by using a synchronous protocol for data transfer instead of UART. We have frequently had problems with data loss and parsing by sending both logs and data over the same UART console channel.

Drawbacks

Two channels are now required for testing instead of one. We currently require both UART and SPI for tests anyway in order to bootstrap, so this should not add extra overhead for Earl Grey.

Inspecting ASCII JSON over the console for debugging (as is currently done) could be easier than inspecting binary data (as would be required). The tools used to capture signals often translate binary into ASCII anyway (for example most logic analyzer viewers can do this).

Interrupts must be enabled in order to process SPI commands.

Future possibilities

The binary protocol doesn't necessarily need to use SPI as its transport. Future posibilities include USB, I2C, and JTAG.

[jwnrt]

CC:

• @timothytrippel or @a-will or @sasdf
• @Razer6 or @sameo
• @AlexJones0 or @pamaury or @nbdd0121

I would appreciate your comments

[a-will]

I haven't looked at the protocol yet, but I think you can only get away with making this an option, not a replacement. Unlike UART, the presence of spi_device and its method of bootstrapping is not guaranteed.

At first glance, though, this feels too raw for direct, general use in OTTF. I'll elaborate later ;)

[a-will]

I think this is creating an unnecessarily large change that throws out the useful layering. Because we are already simply using JSON to serialize structs, here are alternative approaches that could effect similar goals:

• The (de)serialization adds complexity and overheads
  • I'll assume you mean there are problems with ASCII to binary, since (de)serialization is always part of transferring a piece of data that is wider than your wires.
  • Alternative: Add a binary encoding for the structs and send those over the wire. Deserialization can become mostly memcpy().
• The in-band software flow control is not foolproof and has shown reliability problems
  • Alternative: Add transports that have more reliable flow control. For our rather limited UART, it's possible you could get away with using a GPIO for CTS, and the abstracted transport would set it high only when software is explicitly attempting to receive a struct. I would expect that on-OT software can keep up with UART line rate for a binary encoding.

Edit: I think the part of the solution I particularly object to is making the host need to take in the device's ELF, know its layout, and direct the device to write to particular addresses. The part about transferring data in binary form could still be useful, but I think we're creating unnecessary churn to throw out our set of commands and structs and reducing the formalized capabilities for little benefit.

[sasdf]

Truncated coverage dumps due to UART data loss have occurred a few times in our experience.
A more robust transport is definitely welcome, but fully adopting this protocol for coverage still poses challenges.

Similarly the host may choose to read out coverage data from a known location in SRAM at the end of the test. This replaces the current coverage data transfer mechanism.

Coverage dumps behave more like unsolicited messages from the device.
During a single test, several coverage reports may be sent to the host whenever the device prepares to invalidate its RAM.
This includes:

1. Completion of a firmware stage (ROM / ROM_EXT / IMM_SECTION / TEST)
2. Device-triggered reset
3. Faults or exceptions followed by a shutdown

Also, if some coverage dumps remain on UART and this protocol is utilized for test data transfer, another thread would likely be necessary to drain and process the UART logs concurrently.

[pamaury]

After thinking about it, I am wondering if this proposal is mixing several things which probably ought to be separated and fixed separately. In my mind, we have three distinct problems:

• The main problem that we have is that we use a single channel (UART usually) for both logging (aka unsolicited message) and control/sync of the DUT. That's not ideal for certains tests (it's fine if one just wants a PASS/FAIL but that's pretty much it). Therefore we should simply look at separating the logging and controls channels? Ideally with proper abstraction: logging is clear, control less so.
• The UART is still unreliable. I think we should once and for all fix it by introducing hardware flow control (which can be handled by GPIOs and interrupts on the OT side) which could be used if one wants UART for a control channel (e.g. using two UARTS). SPI doesn't have this issue and would be fine as a control channel.
• The ujson is definitely not great (very hard to debug/improve macros, generates tons of code, etc) and for those we should just transmit the data in raw form. If we have a proper control channel, then sending a bunch of bytes without serialization will become trivial.

I don't think it's realistic to change all tests so we should probably just accept that a simple PASS/FAIL message on the logging UART is good enough for most tests though. It's really the more complex tests which benefit from a distinction between logging and control I think?

[jwnrt]

Thanks for the comments everyone, I'll respond to a few now but will need to think about the larger concerns some more.

I think the part of the solution I particularly object to is making the host need to take in the device's ELF, know its layout, and direct the device to write to particular addresses.

This part is intended to be a no-op from the current situation if possible. We currently read ELFs for a handful of test harnesses and send read/write mem UJSON commands through the console.

The exception is that I'm proposing to read the test status over the binary transport as well which requires knowing the address it's stored in. To avoid reading the ELF of every single test, I was thinking of either using a fixed address in SRAM, or reading the test status field of the Ibex wrapper's DV_SIM_WINDOW register like DV sim does. I haven't checked whether this register exists on silicon or not.

Coverage dumps behave more like unsolicited messages from the device.

I wasn't aware of this, thanks. The current RFC text assumes coverage will only be read at the end of a test.

@AlexJones0 and I discussed an alternative synchronisation protocol that might support this. Instead of the host sending a synchronisation code and blocking until the device synchronises, we could instead do:

• Host sends a "where are you blocked?" command and polls until the device responds.
• Whenever the device is "blocked" it calls ottf_sync(code) which spins until the host unblocks it.
• Examples of blocks include:
  • Needing to synchronise with the host.
  • Test finished, host should read status.
  • Crash, host should read errors.
  • Coverage report produced, host needs to read it.

This would allow the device to prompt its own events without having to "blindly" send them over the asynchronous UART, not knowing if the host will catch them.

I don't think it's realistic to change all tests so we should probably just accept that a simple PASS/FAIL message on the logging UART is good enough for most tests though.

I am coming over to the opinion of allowing PASS!/FAIL! over the UART, it seems like a way to simplify the 90% of tests that don't need a harness. It also lets you watch for BFVs from the ROM which we can't move over to the SPI device anyway.

[a-will]

I think the part of the solution I particularly object to is making the host need to take in the device's ELF, know its layout, and direct the device to write to particular addresses.

This part is intended to be a no-op from the current situation if possible. We currently read ELFs for a handful of test harnesses and send read/write mem UJSON commands through the console.

The exception is that I'm proposing to read the test status over the binary transport as well which requires knowing the address it's stored in. To avoid reading the ELF of every single test, I was thinking of either using a fixed address in SRAM, or reading the test status field of the Ibex wrapper's DV_SIM_WINDOW register like DV sim does. I haven't checked whether this register exists on silicon or not.

If you wanted a command to read the test status, we could simply add that command. There's no need to add extra requirements for a known address if software needs to handle the command anyway. There can even be common code to handle this command, so the implementation is shared across tests.

[jwnrt]

That's not it, the command to read test status is to allow removing all the other stuff, not the other way around

Maybe I misunderstand though. A command for test status would probably be better than reading from an address, yeah

[sasdf]

Instead of the host sending a synchronisation code and blocking until the device synchronises, ...

Thanks, this variant looks better. It also has the benefit that unexpected sync code can be raised as an error on the host side instead of sync timeout.

Besides, what's your plan on migrating FT perso? It also uses OTTF SPI console + ujson stack currently, but arbitrary SRAM read-write is likely to be an security issue on mission mode device.