In our previous blog post, we detailed the successful installation of the XDMA driver and its subsequent testing on the Alveo U55C CMS design. With these foundational steps behind us, we're now poised to start application development. However, before diving into the development, it's crucial to grasp a fundamental concept pivotal to working with PCIe: the Base Address Registers (BARs).
Why BARs Matter
Base Address Registers (BARs) enable communication and data transfer between the host and card PCIe devices. BARs play a critical role in defining how a peripheral within a PCIe system presents its memory and I/O space, a concept we must understand to effectively work with the Alveo platform or other Programmable Logic PCIe applications.
Decoding the BARs: A Simplified Overview
At the simplest level a BAR defines the contiguous memory allocation or I/O space required by a PCIe device. To communicate the location and size of the BARs required by a PCIe device. The PCIe component specification defines the PCIe register configuration and initialisation sequence which enables the address space requirements to be implemented by the Operating System.
During system initialisation the configuration of all PCIe cards is read and address spaces are assigned. Initially the PCIe device BAR registers define the size required for each BAR with the device. This is communicated to the system by the BAR registers being set by the system to all 1’s and the device clearing the LSB to 0’s to represent the address space needed.
With the address space known the system is able to allocate the address space, writing back the assigned base address to the PCIe device BARs. This means in the system address space we now have several regions allocated for PCIe BARs for both end points and the root. This allows not only the root to communicate with an end points but also, an end point to communicate with a end point via the root.
When discussing PCIe addressing the term Memory Space is used for high bandwidth, large volumes of data transactions. While the term I/O space is used to define slow speed control, communication and reporting transactions.
What this means for our Alveo application is that memory space is used for DMA transfers, while IO space is used to configures the DMA and IP modules within the programmable logic design ultimately resulting in AXI Lite transactions within the programmable logic.
Conceptually how transactions work is that an application running on the CPU, will make a read or write request to the BARs allocated address space which will then be routed to the end point provided the access requested resides within one of the BARs by the PCIe Root.
Practical Application: Configuring the XDMA
The configuration of the XDMA within our CMS example design configures the XDMA IP as a DMA subsystem. In this configuration one BAR (BAR1) is presented for configuration which is used to provide the AXI Lite interface. While the DMA AXI Master (BAR0) is connected to the BRAM, this BAR does not need to be configured by the user. If we want to bypass the DMA and allow AXI4 access this would be BAR2 and can be enabled by selecting the PCIe to DMA bypass option in the XDMA IP configuration.
While a DMA based approach is suitable for many applications. There are cases we might want more fine grained control. In this case can configure the XDMA IP not as DMA but as a PCIe bridge instead. This will provide several BARs to be configured by the user as desired for the application being developed.
For the rest of these blogs however, I will keep the design configured as DMA.
When it come to developing applications for Alveo in Vivado outside of the XDMA IP block we are working with AXI and AXI lite as normal.
This leads to the question how does the mapping from a BAR to AXI / AXI Lite address work? It is rather simple, in that any access to the BAR address range will be mapped into the address range defined by the AXI to PCIe Translation definition. For our design the translation is straight forward as the PCIe address mapping is set to map to 0x0.
In the above example I used one of the applications which came with the XDMA driver installation(reg_rw) to write IO information to the AXI lite network within the U55C. This transaction was write output data to a AXI GPIO connected to a ILA in the Alveo card. I added this into the CMS design and this gives us the ability to see the result of the PCIe/AXI transaction in the logic. Along with the simple reg read and write program there are also other applications which will help us read and write using DMA to and from the Alveo U55C.
Working with these programs and creating our own application is something we will look at in the next blog!
Workshops and Webinars
If you enjoyed the blog why not take a look at the free webinars, workshops and training courses we have created over the years. Highlights include
Professional PYNQ Learn how to use PYNQ in your developments
Introduction to Vivado learn how to use AMD Vivado
Ultra96, MiniZed & ZU1 three day course looking at HW, SW and Petalinux
Arty Z7-20 Class looking at HW, SW and Petalinux
Mastering MicroBlaze learn how to create MicroBlaze solutions
HLS Hero Workshop learn how to create High Level Synthesis based solutions
Perfecting Petalinux learn how to create and work with petalinux OS
Embedded System Book
Do you want to know more about designing embedded systems from scratch? Check out our book on creating embedded systems. This book will walk you through all the stages of requirements, architecture, component selection, schematics, layout, and FPGA / software design. We designed and manufactured the board at the heart of the book! The schematics and layout are available in Altium here Learn more about the board (see previous blogs on Bring up, DDR validation, USB, Sensors) and view the schematics here.
Sponsored by AMD