MicroZed Chronicles: Running the Debug Laboratory on Hardware
FPGA Horizons London- October 6th and 7th 2026 - get Tickets here.
The $99 Artix UltraScale+ Explorer Board - learn more here
In my last blog, I introduced a debug laboratory: a Spartan-7 design that can be deliberately broken. It allows us to select one of six realistic faults at run time and then find them using ILA cores inserted into the design.
In this blog, we are going to look not only at running the experiments on hardware, but also at identifying the key characteristics we should look for in the ILA captures to diagnose each fault.
For each fault, we will work through the GUI settings, the Tcl commands used to arm and display the capture, and what the resulting waveform shows.
Getting the laboratory running
To get started, the first thing we need to do is program the FPGA on the Arty board and then connect the Python GUI. Once programmed, the LEDs on the board should indicate the status.
LED 3 flashes as a heartbeat.
LED 2 is on if the MMCM has locked.
LED 1 indicates whether a fault has been logged.
LED 0 indicates whether an experiment is running.
With the FPGA configured, we can start the Python GUI and connect it to the FPGA UART. The GUI is available under the host directory in the repository.
To make working with these labs easier, the repository provides a Tcl recipe script that should be sourced in the Vivado Tcl console.
source C:/hdl_projects/debugging_advanced/scripts/ila_recipes.tclBefore we start any experiments, it is worth taking an immediate capture from each of the three cores: the fast native ILA, the slow native ILA, and the System ILA.
lab_capture_now
lab_capture_now ila_slow
lab_capture_now system_ilaThis confirms that the JTAG connection, the clocks, and the probes file are all working before we start chasing bugs that we injected on purpose.
Every exercise then follows the same process:
Select the mode and settings in the GUI.
Click Clear + stop to remove any old sticky errors.
Run the arm commands in the Tcl console.
Click Apply + restart, then run the display commands.
This order is important because arming the ILA before restarting the experiment means that the trigger is already waiting when the fault first occurs.
Unless an exercise says otherwise, the GUI settings are 16 words per packet, 48 stall cycles in every 64, an AXI skew of 16 cycles, and a rare packet number of 8. The recipes capture every clock using 1024 samples, with the trigger at sample 768. This gives us 15.36 microseconds of history on the 50 MHz fast ILA before the trigger point. This is deliberate: several of these faults occur during stalled cycles, and capturing only accepted transfers would throw away exactly the cycles we need to see.
The healthy baseline
Before we inject anything, it is good practice to see what correct behaviour looks like. Select mode 0 in the GUI and click Clear + stop, then arm the System ILA on a source stall and the slow ILA on the CDC completion.
lab_arm_system source_stall
lab_arm_cdc_slowThen click Apply + restart and display both captures.
lab_show_capture system_ila
lab_show_capture ila_slowThe capture signature we are looking for is as follows. In the source stream slot, we can find cycles where TVALID is high and TREADY is low, while TDATA and TLAST hold their values across the entire wait and into the accepting edge. This is a correctly behaved source under backpressure. In the slow trace, the destination event count finishes at 32, matching the 32 source events. The GUI reports increasing accepted stream words and completed AXI writes, 32/32 CDC events, and no errors recorded.
This is important because a stall is perfectly normal on an AXI-Stream interface. Incorrect handling of a stall is our first fault.
Fault one: backpressure
Select mode 1 in the GUI, keep the standard settings, and click Clear + stop. Then arm the fast ILA on the stability check and the System ILA directly on the stalled handshake.
lab_arm_fault 1
lab_arm_system source_stallClick Apply + restart in the GUI. To display the captures, run the commands below.
lab_show_capture
lab_show_capture system_ilaThe capture signature we are looking for is as follows.
In the System ILA capture of the source interface, TVALID stays high and TREADY is low; however, TDATA changes on successive stalled clocks. The faulty source counter is enabled by TVALID alone rather than by TVALID and TREADY. As a result, it advances without a transfer, and words are lost while the FIFO cannot accept them.
The GUI confirms this with Payload changed while stalled, followed later by a data sequence mismatch at the sink.
The first lesson of the laboratory is to trigger as close to the causal violation as possible. The downstream data error arrives many transfers later: it tells us that something is wrong, while the handshake violation tells us where.
Fault two: the missing last word
Select mode 2, keeping 16 words per packet and the 48-in-64 stall pattern. Click Clear + stop, then arm the TLAST check and a System ILA capture around an accepted last word.
lab_arm_fault 2
lab_arm_system sink_lastClick Apply + restart in the GUI. To display the captures, run the commands below.
lab_show_capture
lab_show_capture system_ilaThe capture signature we are looking for is as follows.
The trick to reading this capture is to count only accepted beats: a stream transfer occurs only on a sampled edge where both TVALID and TREADY are high. With 16-word packets and an incrementing data pattern, every accepted word ending in hexadecimal F should carry TLAST, and no other word should. In the faulty capture, TLAST is missing on the sixteenth accepted word or is asserted on a different word. This happens because the fault drives TLAST from a count of elapsed clock cycles rather than accepted transfers. Once stalls stretch the packet, the marker drifts away from the true boundary. The first packet may even be correct, so it is worth inspecting the following packet as well. On the source side, TLAST can also be seen changing during a stall, which the protocol checker flags too.
The interesting thing about this fault is that a protocol checker has no idea what our intended packet length is. A stable but wrongly placed TLAST can be perfectly legal AXI-Stream and still be completely wrong. This is why the design includes an application-level packet checker.
Fault three: the write that never completes
Select mode 3 with an AXI skew of 16 cycles and click Clear + stop. Then arm the native timeout trigger and a System ILA capture on the address handshake.
lab_arm_fault 3
lab_arm_system axi_addressAfter clicking Apply + restart in the GUI, display the captures by running the commands below.
lab_show_capture
lab_show_capture system_ilaThis fault is my favourite because it needs two captures to tell the whole story. In the System ILA capture, which triggers on AWVALID with the trigger positioned early at sample 32 to retain the start of the transaction, we see the address accepted on AWVALID and AWREADY, with the data handshake following separately around 16 clocks later. Both transfers complete and BREADY is high; however, BVALID never asserts. The faulty target responds only when the address and data handshakes occur on the same clock edge, which AXI does not require. In fact, the healthy transactions alternate between address-first and data-first ordering.
In the native capture, the AXI probe shows both channels accepted at the master and remembered at the target, while the transaction age field counts up to the 1024-clock deadline. At this point, the timeout bit asserts and the GUI reports AXI response timeout, with the completed write count stuck at zero.
Neither capture alone contains both the initial handshakes and the timeout. The native trace has 768 pre-trigger samples and still cannot reach back to the start of the transaction. Trigger position is a decision we need to make, not a default we should simply accept.
Fault four: catching the rare packet
Select mode 4 with rare packet 8, remembering that the numbering starts at zero. Keep the standard packet and stall settings and click Clear + stop. The native checker is armed with the usual recipe. For the System ILA, we build a custom trigger that selects the corrupted word itself. Run these commands before restarting the experiment.
lab_arm_fault 4
set rare_ila [lab_recipe_setup system_ila 768]
set_property TRIGGER_COMPARE_VALUE eq32'h0000008E [lab_probe $rare_ila net_lot_0_axis_tdata]
set_property TRIGGER_COMPARE_VALUE eq1'b1 [lab_probe $rare_ila net_slot_0_axis_tvalid]
set_property TRIGGER_COMPARE_VALUE eq1'b1 [lab_probe $rare_ila net_slot_0_axis_tready]
set_property TRIGGER_COMPARE_VALUE eq1'b1 [lab_probe $rare_ila net_slot_0_axis_tlast]
run_hw_ila $rare_ilaAfter clicking Apply + restart in the GUI, display the captures by running the commands below.
lab_show_capture
lab_show_capture system_ilaThe capture signature we are looking for is as follows. Packet 8 carries the values 0x80 through 0x8F, and the fault flips bit 0 of the final word, but only when an earlier word in that packet stalled. As a result, the accepted sequence ends 0x8D, 0x8E, 0x8E, with TLAST on the second 0x8E, while the next packet resumes cleanly at 0x90. The custom trigger requires TDATA to equal 0x8E with TVALID, TREADY, and TLAST all high. This selects exactly the corrupted word and avoids the legitimate 0x8E that precedes it. In the native capture, the data error bit asserts on the sink mismatch and the expected word probe reads 0x0000008F, matching the GUI first-error fields: expected 0x0000008F, observed 0x0000008E, packet 8.
Every handshake here is legal and the packet length is correct; the payload is simply wrong. Only the application checker can know that.
The real fun starts when we set the rare packet to 10000, where the expected and corrupted final values become 0x0002710F and 0x0002710E. At 50 MHz, the standard capture depth records just 20.48 microseconds, nowhere near enough to record ten thousand packets and search through them afterwards. Instead, we use the trigger state machine in the native fast ILA. This uses three states: wait for the packet counter to equal the selected packet; wait for the flag indicating an earlier stall in that packet; then wait for the data error and trigger. The repository includes the trigger state-machine file, with probe names checked against the generated LTX, along with a small Python script to regenerate it for a different target packet. This is what advanced triggering is really about: using logic instead of memory to focus a small buffer on a rare event.
Fault five: the vanishing event
Select mode 5 and click Clear + stop, then arm the fast checker and the slow ILA on the CDC completion.
lab_arm_fault 5
lab_arm_cdc_slowAfter clicking Apply + restart in the GUI, display the captures by running the commands below.
lab_show_capture
lab_show_capture ila_slowThe capture signature we are looking for is as follows. In the slow ILA 64-bit probe, with most of the 61.44 microseconds of buffer held before the trigger on destination done, the fault mode bit reads 1 and the destination count finishes well below the 32 events sent, nominally around 8. The synchronised pulse bit shows only those pulses that the slow domain actually sampled, followed one slow clock later by its delayed copy. The receiver increments on a sample where the synchronised pulse is high and the delayed copy is still low. In the fast capture, the count comparison bit asserts when the completed totals disagree, and the GUI reports a CDC event count mismatch with 32 sent and fewer received.
The cause is the classic two-flop synchroniser being used on a pulse. The source generates one 20 ns pulse every 33 fast clocks, while the destination samples every 80 ns. As a result, short pulses can occur entirely between its sampling edges and are simply never seen. The healthy comparison, mode 0 with only the slow ILA armed, shows the request and acknowledge bits following one another as the handshake holds each event until the destination confirms it, finishing at 32/32.
It is worth being clear about what we are demonstrating here. This is deterministic event loss caused by the sampling window; the clocks are deliberately related to make it repeatable. Neither RTL simulation nor an ILA can show us the analogue resolution of a metastable flip-flop, but they can absolutely show us a pulse that was never sampled.
Fault six: is the FIFO broken?
Select mode 6, retaining the 48-in-64 stall pattern so that the FIFO fills, and click Clear + stop. Then arm the occupancy checker and a System ILA capture on the source stall.
lab_arm_fault 6
lab_arm_system source_stallAfter clicking Apply + restart in the GUI, display the captures.
lab_show_capture
lab_show_capture system_ilaThis final mode is the one that generates the most heated lab conversations, because the evidence initially points at the vendor IP. The AMD AXI4-Stream Data FIFO owns the storage and flow control, while a VHDL wrapper maintains its own occupancy count from the accepted transfers.
Simultaneous operations should leave the occupancy unchanged. Instead, the wrapper occupancy field decrements by one on the following sample and the mismatch bit asserts. Both modulo-16 push and pop counters increment, consistent with one word in and one word out. We can reconstruct the expected behaviour from the captured events: next occupancy equals occupancy plus push minus pop. This shows exactly where the wrapper diverges. The injection occurs once per restart and the incorrect count then remains offset. It can even wrap when the FIFO drains, so we inspect the first divergence rather than the later wrapped value.
Meanwhile, the data sequence and TLAST checks stay clean, and the System ILA shows the real FIFO handshakes delivering every word correctly. The AMD FIFO is doing its job; the defect is in our accounting. The independent checks are what let us say this with confidence, rather than filing a support case for a bug that is ours.
Wrapping up
Across the six experiments, each faulty capture is paired with a healthy run using identical settings. For every run, we save the waveform, the GUI-exported JSON snapshot that records the applied settings and first-error fields, and a short note identifying the decisive signals. This habit turns a debug session into evidence, which is the difference between fixing a bug and being able to demonstrate that it was fixed.
The nice thing about the register-controlled fault selection is that every one of these experiments is repeatable without rebuilding the design. This makes the laboratory ideal for practising trigger design, capture positioning, and, above all, the discipline of triggering on causes rather than symptoms. As always, the design, recipes, and guides are in the GitHub repository. It makes a great rainy afternoon of hands-on debugging practice.
FPGA Conference
FPGA Horizons London- October 6th and 7th 2026 - get Tickets here.
FPGA Journal
Read about cutting edge FPGA developments, in the FPGA Horizons Journal or contribute an article.
Workshops and Webinars:
If you enjoyed the blog why not take a look at the free webinars, workshops and training courses we have created over the years. Highlights include:
Upcoming Webinars Timing, RTL Creation, FPGA Math and Mixed Signal
Professional PYNQ Learn how to use PYNQ in your developments
Introduction to Vivado learn how to use AMD Vivado
Ultra96, MiniZed & ZU1 three day course looking at HW, SW and PetaLinux
Arty Z7-20 Class looking at HW, SW and PetaLinux
Mastering MicroBlaze learn how to create MicroBlaze solutions
HLS Hero Workshop learn how to create High Level Synthesis based solutions
Perfecting Petalinux learn how to create and work with PetaLinux OS
Boards
Get an Adiuvo development board:
Adiuvo Embedded System Development board - Embedded System Development Board
Adiuvo Embedded System Tile - Low Risk way to add a FPGA to your design.
SpaceWire CODEC - SpaceWire CODEC, digital download, AXIS Interfaces
SpaceWire RMAP Initiator - SpaceWire RMAP Initiator, digital download, AXIS & AXI4 Interfaces
SpaceWire RMAP Target - SpaceWire Target, digital download, AXI4 and AXIS Interfaces
Embedded System Book
Do you want to know more about designing embedded systems from scratch? Check out our book on creating embedded systems. This book will walk you through all the stages of requirements, architecture, component selection, schematics, layout, and FPGA / software design. We designed and manufactured the board at the heart of the book! The schematics and layout are available in Altium here. Learn more about the board (see previous blogs on Bring up, DDR validation, USB, Sensors) and view the schematics here.
All words in this blog were written by a human.
