Posted on

The Future of AI Servers

The Future of AI Servers

Following up on the recent announcement of our new Raspberry Pi 4 AI Servers, it seems that AI servers running on Arm processors are gaining more and more traction in the market due to their natural fit at the IoT and Edge layers of infrastructure.  Let’s take a quick look at some of the unique properties that make AI Servers running on Arm a great strategy for AI/ML and AIoT deployments, to help understand why this is so important for the future.

Power – Many IoT deployments do not have luxuries that “regular” servers enjoy such as reliable power and connectivity, or even ample power for that matter.  While Intel has spent decades making excellent, though power hungry processors, Arm has focused on efficiency and battery life, helping to explain they they dominate the market in tablets and smartphones.  This same efficiency is then leveraged by IoT devices running AI workloads, so Edge devices responsible for computer vision, image classification, object detection, deep learning, or other workloads can operate with a much lower thermal footprint than a comparable x86 device.

Size – Similar to the underlying reasons behind power efficiency, the physical size and dimensions of Arm AI Servers can be made smaller than the majority of x86 designs.  Attaching AI Accelerators such as the Gyrfalcon 2801 or 2803 via USB to boards as small as 2 inches square (such as the NanoPi Neo2) is possible, or the addition of a Google Coral TPU via the mini-PCIe slot on a NanoPi T4 bring an enormous amount of inferencing to AI Servers in tiny form factors. 

Cost – Here again, Arm SoC’s and Single Board Computers typically have a rather large cost advantage versus x86 embedded designs.  

Scalability – This is a critical factor in why Arm will play a massive role in the future of AI Servers, and why miniNodes has begun to offer our Raspberry Pi 4 AI Server.  As mentioned, low power, cheap devices make great endpoints, but, there is also a role for “medium” sized AI servers handling larger workloads, and Arm partners are just now starting to bring these products to market.  An example is the SolidRun Janux AI Server, which also makes use of the same Gyrfalcon AI Accelerators used by our nodes.  So, you can get started training your models, testing out your deployment pipeline, understanding the various AI frameworks and algorithms, and getting comfortable with the tools, and very easily scale up as your needs expand.  Of course, once you reach massive amounts of deep learning and AI/ML processing, enterprise Arm server options exist for that as well.

Flexibility – Taking Scalability one step further, the Arm AI servers also allow for a great amount of flexibility in the specific accelerator hardware (Gyrfalcon, Google Coral, Intel Movidius), the frameworks used (Caffe, PyTorch, TensorFlow, TinyML), and the models (ResNet, MobileNet, ShuffleNet) employed.  

Ubiquity – A final piece of the overall AI Server ecosystem is the ease of access to this type of hardware, and low barriers to entry.  The Raspberry Pi and similar types of boards are distributed globally, and readily available in nearly all markets.

As you can see, our view is that the future of AI servers is based on Arm SoC’s, and now is the time to start exploring what that means for your workload.

Posted on

Recap: Building an Arm-Powered IoT, Edge, and Cloud Infrastructure

Intro

At Arm’s annual TechCon event in San Jose, Arm CEO Simon Segars presented a vision of the future where a trillion connected devices interact seamlessly with each other and pass data between the Cloud, the Edge, and the Internet of Things, at a scale unimaginable even just a few years ago. Self driving cars will generate massive amounts of sensor information and data, 5G wireless will enable increased connection speeds and reduced latency, and artificial intelligence will provide scientific breakthroughs in materials, technologies, medicines, and energy. This vision of the future state of the connected world is something we have heard about for several years now, with countless written articles, interviews, social media posts, conference talks, and various other forms of media addressing the topic.

However, when seeking out real-world examples of this architecture in practice to help learn and understand how the bits and pieces work together, we came up empty. There were no purpose-built sample projects, pre-written code examples, or other working prototypes of these principles available. Surely there are some internal, private teams building out this type of infrastructure for specific use-cases and organizational needs, but there were no public / open projects to learn from.

Thus, it was time to take action, and build a prototype infrastructure ourselves! With the help of the Arm Innovator Program, we set out on a journey to develop a proof-of-concept that encapsulates as many of these concepts as possible, leveraging currently available technologies and showcasing Arm’s diverse portfolio of products and ecosystems. With help from the Works on Arm program via Packet.com, we began brainstorming.  Our goal was to deploy IoT endpoints to a handful of locations around the world, and capture environmental data via sensors on those devices. From there, we wanted to feed that data to a local Edge Server, which would be responsible for translating the data to a usable format and sending it further upstream, to a Cloud Server functioning as a data warehouse and visualization platform.

In this article we’ll take an in-depth look at the project, and detail the key technologies to give a better idea of what this kind of system entails. I’ll also provide a summary of our lessons learned, which hopefully help you to build and iterate faster, and avoid some potential pitfalls along the way.

Design

When thinking about the design of this project, we wanted to keep things simple, as the purpose of this exercise it to demonstrate capability and build a proof-of-concept, but not an actual product shipped to real, paying customers.  Thus, we made hardware and software selections based on cost and availability, as opposed to “most appropriate” for the intended use. We also knew we would have relatively small data-sets, and reliable power and internet connectivity for all of our devices.  Your real-world IoT deployments may not have these luxuries, so, your hardware and software selections may not be as straightforward as ours were.  Many IoT projects have to be tolerant of lost network connectivity, unreliable power delivery, or harsh environmental conditions.  But we were fortunate to have consistent power and internet.  Let’s go through our inventory of Arm-powered hardware and software, keeping in mind the rather ideal conditions we’ve got:

1. IoT Endpoints

Hardware

  • Raspberry Pi 3B+
  • Sparkfun Qwiic HAT
  • Sparkfun Lightning Detector
  • Sparkfun Environmental Combo Sensor
  • Sparkfun GPS Sensor

Software

  • Arm Mbed Linux OS
  • Arm Pelion Device Management

 

2. Edge Nodes

Hardware

  • Linaro / 96Boards HiKey, and HiKey 960

Software

  • Linaro Debian Linux

 

3. Cloud Server

Hardware

  • Ampere eMAG, hosted by Packet.com

Software

  • Debian Linux
  • InfluxDB
  • Grafana

 

As you can see, we have made some selections that fit our small project well, but as mentioned may not be suitable for all IoT use cases depending on your project’s environmental conditions.  However, let’s start detailing the items, beginning with the IoT Endpoint.  We’re using a Raspberry Pi 3B, a Sparkfun Qwiic HAT, and Sparkfun sensors to capture Temperature, Humidity, Barometric Pressure, CO2, and tVOC (volatile organic compounds).  We have lightning detection capability (currently not being used, but, available) as well, and GPS so that we can determine precisely where the Endpoint is located.  As for software, because these devices are out in the wild, scattered literally across the globe, we needed a framework to allow remote monitoring, updating, and application deployment.  Arm Mbed Linux OS is a lightweight, secure, container-based operating system that meets these requirements.  It is currently still in Technical Preview, but is far enough along in development that it meets our project needs and is working great.  A total of 10 Raspberry Pi Endpoints were built and sent around the globe, with several across the United States, as well as Cambridge, Budapest, Delhi, Southern India, and one spare unit left over for local testing.

Turning to our Edge Nodes, these are the simplest component in our project’s infrastructure. These are 96Boards devices, chosen for their support and ease-of-use.  Linaro and the 96Boards team do an excellent job of building ready-made Debian images with updated kernels, applications, and drivers for their hardware, making for a great out-of-the-box experience. Two of these devices are currently provisioned, one in India and one in the United States, each serving their geographic region. The devices aggregate the IoT Endpoint data stream, convert it to the format needed by the Cloud Server, and publish the data to the Cloud.

Finally, the Arm-powered Cloud Server is an Ampere eMAG server, hosted by Packet.com. It is an enterprise-grade machine, and functions as the data warehouse for all of the IoT data, as well as a visual platform for charting and viewing the data in a time-series fashion thanks to InfluxDB and Grafana. Packet.com has datacenters around the world, and their native tooling and user interface make deploying Arm Servers quick and easy.

Now that the system architecture has been described, let’s take a look at the application architecture, and start to dissect how data flows from the IoT Endpoints, to the Edge, to the Cloud. As mentioned, Mbed Linux OS is a container-based OS, which is to say that it is a minimal underlying operating system based on Yocto, providing a small, lightweight, secure foundation to which the Open Container Initiative (OCI) “RunC” container engine is added.  RunC can launch OCI compliant containers built locally on your laptop, then pushed to the Endpoint via the Mbed Linux tooling, no matter where the device is located.  In our particular case, we chose a small Alpine Linux container, added Python, added the Sparkfun libraries for the sensors, and created a small startup script to begin reading data from the sensors when the container starts.  The container also has an MQTT broker in it, which is responsible for taking that sensor data, turning it into a small JSON snippet, and publishing it to a specific known location (the Edge Server).

The Edge Servers are a more traditional Debian operating system, with Python installed as well.  There is a Python script running as a daemon that captures and parses the incoming MQTT from IoT Endpoints, converts it to an InfluxDB formatted query, and publishes it to the specified Influx database, which is running on the Ampere eMAG Cloud Server.

Finally, the Cloud Server is an enterprise-grade Ampere eMAG Arm Server.  It is graciously hosted by the Works on Arm project at Packet.com, in their New Jersey datacenter. This server is also running Debian, and has InfluxDB and Grafana installed for storage and visualization of the data being sent to it from the Edge Nodes.  Thus, our IoT, Edge, and Server are all Arm-powered!

Construction Challenges

Building a container to hold our application did prove more challenging then anticipated, as a result of some needed functionality not provided by the ready-made Mbed Linux downloads. Normally, this could be easily solved by adding the desired packages to the Yocto build scripts and rebuilding from source…however, there is one additional and very unique quirk to this project: We have decided to exclusively use Arm-powered Windows on Snapdragon laptops to build the project!  These laptops are highly efficient, with all-day battery life and far better performance than previous generations offered.  One limitation however, is they are currently unable to run Docker, which we would need to re-build Mbed Linux from source.  Thus, instead of adding the necessary packages to Yocto and recompiling, we instead had to manually port Device Tree functionality, gain access to the GPIO pins on the Pi, enable the I2C bus and tooling, and finally expose that functionality from the host OS to the container, all by way of manually lifting from Raspbian.  Obviously, we placed this limitation upon ourselves, but it does demonstrate that there are still a few shortcomings in the developer experience on Arm.

A second valuable lesson learned is with the native Mbed tooling for initially deploying devices.  Provisioning and updating devices with Pelion Device Management is a straightforward process, except for one small but critical hiccup we experienced.  It is worth noting here again that Mbed Linux OS is in a Technical Preview status, and the feedback we were able to give to the Mbed team as a result of this process has been incorporated and will make the final product even better!  However, when following the documentation to provision devices for the first time, a Developer Certificate is issued. That certificate is only valid for 90 days, and after that time you can no longer push containers to a device in the field. That Certificate can certainly be updated via the re-provisioning process, but, you must be on the same network as the device in order to perform that action. Our devices are already out in the field, so that is not possible at this point.  Thus, we have a fleet of devices that cannot receive their intended application.  On the plus side, this exercise proved it’s worth by highlighting this point of failure, and resulted in the valuable documentation update so that your project can be a success!

Conclusion

In the end, we were able to successfully provision just a few devices that we still had local access to, and prove that the theory was sound and demonstrate a functional prototype at Arm TechCon!

Using a pair of freshly provisioned Raspberry Pi’s, the containerized application was pushed Over The Air to them, via the Mbed CLI.  Pelion showed the devices as Online, and the device and application logs in the Dashboard reported the container was started successfully.  Sure enough, on the Edge Node, data began streaming in, and the MQTT Broker began taking those transmissions, translating them to Influx, and sending them upstream to the Cloud Server.  Logged into Grafana running on the Cloud Server, that data could then be inspected and visualized.

Thus, while it wasn’t quite as geographically diverse as hoped, we did actually accomplish what we set out to do, which was build an end-to-end IoT, Edge, and Cloud infrastructure running entirely on Arm!  The data that is flowing is certainly just a minimal example, but as a proof-of-concept we can truthfully say that the design is valid and the system works!  Now, we’re excited to see what you can build to bring Simon Segar’s vision of the connected future to life!

Posted on

64-bit Ubuntu Raspberry Pi 3 Arm Server Image Now Available

This morning there is some great news for fans of the popular Raspberry Pi 3 single board computer, looking to run 64-bit Ubuntu Arm Server on their board!

 

The Ubuntu team, with support from Arm, has released a ready-made image that can be written to an SD Card and directly booted on a Raspberry Pi 3B or 3B+, with no configuration necessary.  We were able to give this image a test, and although it is technically considered a beta, it seems most everything is working and all of the standard functionality one would expect from Ubuntu Server intact!

 

You can download the image here:  http://cdimage.ubuntu.com/releases/18.04/release/

How to Install Ubuntu on the Raspberry Pi 3

Once the image is downloaded, it needs to be extracted, and can then be written to an SD Card.  Of course, the higher the read and write speed of the SD Card, the better overall system performance will be.

 

After getting the image written and inserted in to the Pi, take note that the first boot may take a few minutes while the OS goes through a few setup routines.

 

A quick run through the system showed the basic console hardware requirements of HDMI, USB, and Ethernet all worked out of the box, as well as WiFi.  SSH is enabled and working, and normal software installation and updating via ‘apt’ package management is working great.  As an added bonus, the image comes with ‘cloud-init’ setup to automatically expand the partition on the SD Card to the maximum capacity of the card, generate SSH keys, configure networking for the LXD container runtime (which is also preinstalled), and finally force a password change upon first login to the system.

 

All said, this means the Ubuntu Arm Server image is ready to use immediately upon writing the SD Card and booting the Pi!

 

In the past, it was technically possible to bootstrap a system using a custom built kernel and an Ubuntu rootfs, then add Pi-specific firmware and drivers.  After that you had to add users, manually install networking, and add even basic system utilities.  That process required in-depth knowledge of system installation and configuration, and was not something most users could tackle on their own.  However, thanks to the efforts of the Ubuntu Arm team in creating this new ready-made image, no advanced knowledge of the Linux build process is required, and even casual Raspberry Pi users can be up and running easily!

 

One final thing to keep in mind, is that this image is fully intended to be a 64-bit Ubuntu Arm Server platform!  Use cases such as File or Print servers, DNS, MySQL or other database servers, web front-end caching, or other lightweight services all make sense for this platform.  It can also be used for installation and testing of Aarch64 software, developing and compiling Arm64 applications, exploring containers, or even production workloads where possible!  Small, distributed compute workloads, IoT services, Industrial Internet of Things, environmental monitoring, remote compute capacity in non-traditional settings, or many other uses cases are all possible.  While a desktop *can* be installed, due to the limited memory on the Raspberry Pi, only a lightweight desktop like LXDE or XFCE will truly work, with both Mate and Gnome quickly running out of memory, moving to Swap, and then slowing the system to a crawl.   Even so, desktop performance in this image is not optimized, so sticking with the intended use of this image as a Server OS makes the most sense.

 

In summary, thanks to a collaborative effort from Arm and the Ubuntu teams, the community now has a ready-made Raspberry Pi 3B(+) 64-bit Ubuntu Arm Server image!
Posted on

ArmTechCon Recap

As you may have seen here and here, miniNodes recently got invited to participate at ArmTechCon, inside Arm’s own “Innovation Pavilion” in the Expo Hall.  Because our core business of hosting tiny Arm Servers isn’t that exciting to show off, especially at the biggest Arm ecosystem event of the year, we partnered with Robert Wolff and the awesome team at 96Boards to come up with something a bit more intriguing.   🙂

After some back and forth, we landed on a solar powered, connected, mobile developer and edge computing platform. The idea was to build a self-contained and self-powered box that could be taken out and used in geographically isolated areas, that could still have connectivity back to a central cloud provider. The actual use cases could vary dramatically, but the common theme is that there is a lack of infrastructure, electricity, or wifi in the targeted region. The box would be powered by solar panels for this iteration, but could also accept other renewable sources such as wind, hydroelectric via a waterwheel or impeller, geothermal, or more.

So, as one potential use case, we envisioned using the box in remote villages or locales that don’t have the typical infrastructure needed to teach development, AI, machine learning, edge computing, remote code or container deployment, or other advanced computer science topics.

The end goal is to provide everything as open source, with a Bill of Materials and instructions for anyone to replicate the build, using readily available, off-the-shelf parts with no customization necessary. For the demo unit though, the project hasn’t made it quite that far yet.  For this prototype, the box consisted of a foldable solar panel array, that was hooked up to a charge controller, which then fed a battery pack. The battery pack was run over to an inverter, so that we could power multiple standard devices. The first device to be powered was a 96Boards Dragonboard, that had a small LCD attached for graphical output, and had a 4G LTE cellular mezzanine which provided data to the Dragonboard.  This, as long as there is cell service, the Dragonboard has connectivity to the internet!  At that point, we had effectively built a solar powered, self sustaining compute workstation that could connect to the internet nearly anywhere!

However, because we were just doing a proof of concept, we thought it would be fun to go even one step further!  Next, we setup sharing on the Dragonboard’s cellular connection, and ran an ethernet cable out from the Dragonboard over to a Raspberry Pi 3 Compute Module.  This Pi was running a service from Microsoft called Azure IoT Edge, which is a product that allows you to remotely push containers and code to an IoT device, or receive data and telemetry back from a device out in the wild.  Thus, as long as there is adequate sunlight (or another renewable source of power) and cell coverage, the box can be remotely monitored and even updated from anywhere.  Or, thanks to its LCD and USB keyboard, it can be used as a workstation in places where infrastructure is lacking.

Another potential use case for the platform could be as an environmental monitoring solution. When equipped with a gyroscope, the box could detect movements from events such as a rock slide, avalanche, mud slide, volcanic activity, etc. Any anomoly can be reported back to the central servers immediately for analysis.

When equipped with a camera, the box could also visually monitor the environment, and detect changes in imagery such as a smoke plume for early forest fire detection, wildlife movement, vehicles approaching locations where there should not be any, or more.

Finally, because of the device’s Raspberry Pi Compute Module carrier board, the box has the ability to run targeted workloads of its own, for extreme edge computing. The workloads can be updated, changed, and monitored remotely, again due to the Dragonboard’s cellular connectivity to the Microsoft Azure IoT Edge platform.

ArmTechCon was a big success, and it’s incredible what can be built using Arm technology.  Be sure to check back for status updates as the solar compute box undergoes future development and iterations!

Posted on

ARM Server Update, Summer 2018

Continuing our quarterly ARM Server update series, it is now Summer 2018 so it is time to review the ARM Server news and ecosystem updates from the past few months!  This blog series only covers the ARM Server highlights, but for more in-depth ARM Server news be sure to check out the Works on Arm Newsletter, delivered every Friday by Ed Vielmetti!

Looking at our recent blog posts, the most important headline seems to be the rumored exit from the business by Qualcomm.  Although, at the moment, this has not been confirmed, if true it would be a major setback for ARM Servers in the datacenter.  The Qualcomm Centriq had been shown to be very effective by CloudFlare for their distributed caching workload, and had been shown by Microsoft to be running a portion of the Azure workload as well.

However, just as Qualcomm is rumored to be exiting, Cavium has released the new ThunderX2 to general availability, and several new designs have now been shown and are listed for sale.  The ThunderX2 processor is a 32-core design that can directly compete with Xeons, and provides all of the platform features that a hyperscaler would expect.

Finally, in software news, Ubuntu has released it’s latest 18.04 Bionic Beaver release, which is an LTS version, thus offering 5 years of support.  As in the past, there is an ARM64 version of Ubuntu, which should technically work on any UEFI standard ARM Server.  Examples include Ampere X-Gene servers, Cavium ThunderX servers, Qualcomm, Huawei, HP Moonshot, and AMD Seattle servers.

As always, make sure to check back for more ARM Server and Datacenter industry news, or follow us on Twitter for daily updates on all things ARM, IoT, single board computers, edge computing, and more!

 

Posted on

miniNodes ARM Innovators Program Interview

The full Arm Innovators Program interview is now posted, and we are proud to be highlighted by Arm for our innovations in the Arm Server ecosystem!

As you can see, we are currently prototyping a Raspberry Pi Cluster PCB that will hold 5 Raspberry Pi Computer on Module (CoM) boards, with a power input and ethernet switch built in.

This Raspberry Pi Cluster Board will allow the Docker, Kubernetes, OpenFasS, Minio, and other cluster projects to easily develop, test, and build their software in a cheap and convenient way, with no cabling mess.  Home automation, IoT, and hardware hacking are other potential uses for the board.

We’re still a few weeks away from launching, but keep watching this space as we will be sure to make an announcement as soon as it is ready!

mininodes-arm-innovator

Posted on

Fedora IoT Edition Approved

The Fedora Council has authorized a new Fedora Edition (as opposed to a Spin), dedicated to IoT devices and functionality!  Fedora ARM developer Peter Robinson is heading up the effort, congratulations to him!  He has information available on his blog located here:  https://nullr0ute.com/2018/03/fedora-iot-edition-is-go/, and there is also an official Ticket capturing the Approval located here:  https://pagure.io/Fedora-Council/tickets/issue/193

The Wiki is just getting built out now, so there is not a whole of information on it quite yet, but keep checking back as it takes shape:  https://fedoraproject.org/wiki/Objectives/Fedora_IoT

 

Posted on

Hosted Raspberry Pi 3 Servers Now Available!

miniNodes.com is proud to be the first cloud hosting provider to offer the new Raspberry Pi 3 as a hosted server.  The Raspberry Pi 3 combines a powerful new Broadcom quad-core 64-bit ARM processor, 1gb of RAM, and the reliable Raspbian Stretch linux operating system.  This makes the Raspberry Pi 3 a great platform for a small ARM server that offers plenty of compute capacity for basic services such as hosting a website or email, API hosting and development, lightweight development frameworks such as NodeJS application hosting, Internet of Things gateways and communication servers, IoT endpoints, Azure Edge container hosts, and more.  The Raspberry Pi 3 server is also a great way to experiment with ARM servers in the cloud, and ensure code compatibility with other more powerful ARM servers that are forthcoming.  Each hosted Raspberry Pi 3 server comes with SSH access and a dedicated IP address, making deployments to the server easy and familiar to developers.

Check them out here:  https://www.mininodes.com/product/raspberry-pi-3-server/