One of the big advantages of the Kria SoM is the ease with which new applications can be created, either around the existing applications or with new AI algorithm dropped in.
Knowing how to do both is critical in order to be able to get the most out of the Kria SoM.
In our previous blog, we looked at setting up the Kria and running the smart camera application from the command line.
There is, however, another way to run the smart camera application. By using Jupyter Notebook labs, we can connect to the Kria SoM over an ethernet network and create applications that use both existing and new AI networks.
If you are not familiar with Jupyter and IPython, Jupyter provides a client server application that uses a web server on the embedded device (Kria SoM in this case) to run a IPython kernel. The Jupyter Notebooks stored on the embedded device is in JSON format. This provides an integrated development environment (IDE) within a web browser.
To get started using Jupyter on the Kria SoM, we can log on using the UART/USB terminal and start the Jupyter Notebook server.
To do this, we need to know the IP address which has been assigned to the Kria SoM by our network. We can obtain this by issuing the command ifconfig.
Once the IP address of the Kria SoM is known, we can start a Jupyter session by entering the command Jupyter-lab –ip=<board ip addr>.
This will start the Jupyter server and provide an IP address and token that can be copied and pasted into a browser to connect to the Jupyter server.
Connecting to the Jupyter server will open the Jupyter labs environment and we can open and create notebooks through this environment. We can even open terminals and create advanced dashboards if we wish to install Voila.
Under the panel on the left, you will notice a SmartCam directory that contains a SmartCam IPython notebook. Double clicking on this will open the notebook and start a kernel on the Kria SoM. This notebook contains an implementation of the smart camera application.
We can run the application by executing each of the cells in the notebook. What is interesting in the notebook is the creation of the GStreamer pipeline which grabs the image from the USB or MIPI camera.
One interesting observation in the Jupyter Notebook is that the new Xilinx Intelligent Video Analysis SDK (IVAS) which provides developers a range of GStreamer plugins can be used in conjunction with official GStreamer plugins to create the image processing pipeline.
The IVAS framework enables a connection to the underlaying FPGA accelerators and the DPU from the application. Documentation on the IVAS is limited at the moment, but I expect more to be released in the coming weeks and months since it is critical to the Kria operation.
To get started creating our own application, we need to understand the GStreamer pipeline. A visualization is provided as part of the Jupyter Notebook. With a little change to the cell, it can be saved as a png and viewed in the notebook.
The ability to visualize the GStreamer pipeline can be critical when creating new applications and solutions. This ability to visualize GStreamer pipes is included for debugging and can be used to show the different states of the pipeline, for example playing or paused.
By examining the GStreamer graph for the smart camera application, you can see several instances from the IVAS including the DPU kernel and the output Sink.
By the time we step through the script, we have the smart camera application up and running as it was before, but we are now able to make modifications to the application if we want to leverage the Jupyter Notebook.
To get started developing applications for the Kria, we might want to add in additional Python packages like OpenCV for example. To enable this, we will need a C Compiler on the Kria SoM.
By default, there is not a C Compiler included in the rootfs but we can remedy this easily by the using the dnf package manager.
The first step is to install a C compiler ID to run the command dnf repoquery. This will query the repository and list the available packages.
The next step is to install the PetaLinux self-hosted package group. This will provide GCC along with other commonly used development tools such as make etc. The command to install this package group is dnf install packagegroup-petalinux-self-hosted.
Once this group is installed, we are able to install Python packages like OpenCV that need the ability to compile C applications during the installation process.
Next, we will look at how to update the AI application running in the overlay.