AlokA.ai

As explained in earlier blog edge AI is running your AI inference as close to sensor detecting the data. Conventional AI would require you to first transmit the data to a server where a GPU farm would process the information, run the your trained models, and extract the inference. The edge AI is a paradigm shift where the processing is done almost with the sensor.

How is Edge AI done

Edge AI requires additional compute resource along with the sensor. The additional compute unit is typically an additional piece of hardware which is capable of running inference. We would not be using this HW to train or create new models. The model training and evaluation will already be complete and we will use a trained model along with this piece of hardware to run inference.

Compute Resources

There is a myriad choices to enable AI compute at the edge. It can be a computer, a raspberry pi, a kit like Jetson X2, FPGA etc. There are a wide array of choices for the HW at the edge. You need to take a decision based on cost and application. I have come across Medtech companies that have laptop along with their hi-end cameras. These laptops are more than capable to run analyses on the images to detect issues in the retinal scan. There are grain scanners and sorters that use high speed cameras along with FPGAs to identify good grains from bad.

Edge AI has been one of the most interesting areas for development recently. Google, Envidia, Intel etc. have all released products targeting this space. We will discuss some of these offerings and give a highlight about each of them. In later blogs, I will explain how to get these devices working with examples

Intel Movidius Stick

Movidius, now owned by Intel, pioneered VPUs (Visual Processing Units). Their chips were made for processing images at a very high speed with low power consumption. One of their offerings is the Movidius stick, which can plug into the USB slot of a raspberry pi or Ubuntu PC/laptop. Using the Intel OpenVino platform, the stick accelerates image processing. On Raspberry Pis, improvements of upto 80 times have been reported. Some of the drawbacks of these sticks is the limited model support. The stick supports a few Tensorflow and Caffe models. The stick is known to support Inception and MobileNet models but may not work with custom models. In case you plan to use this device, you should verify if your models will work with this kit. Another drawback is that it supports on USB 2.0, which limits the speed with which the stick communicates with the host computer.

Jetson Nano Envidia

Envidia GPUs have been the standard AI hardware platform of choice. The Jetson Nano at $99 is the latest offering targeted for small, cheap and low power applications. They have other offerings like the Jetson TX2, TX1, Tegra etc. which are bigger, more expensive and are capable to do a lot more. The Jetson Nano is like a good device and supports more models than the Movidius and Google Coral. The flexibility of compatible models and applications seems to be the best for the Jetson Nano. It is also possible to train models using the Jetson Nano, however I can’t think of many use cases for this feature for now. On the downside, it has limited connectivity options because it does not support neither WiFi nor Bluetooth. Connectivity is primarily through Ethernet, so it is appropriate for applications where wired connections suffice.

Google Coral TPU

The TPU or Tensor Processing Unit is an ASIC designed by google for processing neural networks. It is highly optimised for matrix operations which is foundation of neural network processing. Google recently released the Google Coral dev board that brings the power of TPUs to the edge. The Google coral boards are designed to do quick inference at the edge. It does not support back-propagation and therefore cannot be used for training. It has WiFi, Bluetooth and Ethernet, so has much better connectivity than the Jetson Nano.

In this blog we barely scratch at the myriad of options with which we can implement AI at the edge. Later blogs we will look at how these tools and more.