By using OpenNebula’s new provisioning tools, cloud admins can now expand their private clouds in an incredibly flexible way using resources offered by third-party cloud providers like AWS and Equinix Metal, incorporating when necessary the distributed dedicated infrastructure they need to satisfy their users’ requirements for fault tolerance, capacity or low latency. 

OpenNebula users can automatically allocate resources when needed, deploying and controlling edge nodes based on the current demand at those specific geographical locations. This approach simplifies significantly the process of provisioning and managing edge resources, without the organization that’s using this solution having to provide or own those underlying resources at all.

OpenNebula also offers a simple, but powerful approach for running containerized applications and workflows—both on-premises and on cloud or edge locations—by directly using Docker official images from the Docker Hub and running them as lightweight Firecracker microVMs. For those cases where Kubernetes is required or is the best fit, OpenNebula also provides a Certified Kubernetes Virtual Appliance available from the OpenNebula Public Marketplace, although for Kubernetes deployments at the edge we normally recommend a lighter solution based on K3s clusters 😉

The New AWS Wavelength Service

Recently, Amazon Web Services (AWS), in collaboration with Verizon, Vodafone and other 5G telecommunication providers, has presented its new AWS Wavelength service (read here the full announcement by AWS Chief Evangelist Jeff Barr back in August). Wavelength Zones bring AWS compute and storage capabilities and services to the edge of existing 5G networks, embedding AWS hardware and software within their data centers. This enables developers to innovate and build a new class of edge applications that can exploit high bandwidth and ultra-low latencies as offered by the new 5G networks.

Thanks to AWS Wavelength, application traffic from 5G devices can reach the servers running in Wavelength Zones without leaving the telecommunications network, thus avoiding having to traverse multiple hops across the Internet to reach their final destination, as it happens with a traditional approach based on a centralized cloud solution. This new service enables both developers and end-users to finally take full advantage of the latency and bandwidth benefits offered by 5G networks 📱

At OpenNebula, we have already started testing the new AWS Wavelength resources as part of our ONEedge initiative. Eventually, this new service will be incorporated into our catalogue of cloud and edge providers available for OpenNebula users. We expect this integration to really simplify the process of provisioning and managing resources close to 5G devices, helping organizations using OpenNebula to build and quickly deploy edge applications that can benefit from 5G high bandwidth and ultra-low latency, including machine learning, video streaming, multiplayer gaming, Internet of Things, augmented reality, and real-time analytics. 

OpenNebula’s First 5G Edge Architecture based on AWS Wavelength

As part of our first tests of this new AWS service, we’ve adapted the scenario described by AWS Senior Developer Mike Coleman in a post on AWS Wavelength published in early August. In our case, a company with an OpenNebula private cloud wants to deploy a multi-container application at the edge (i.e a Machine Learning solution), closer to the 5G devices of their end-users. The following diagram describes how this would be implemented based on the features provided by OpenNebula and on the new resources made available by AWS Wavelength:

AWS Wavelength is designed to provide access to services and applications that require low latency, but it’s important to remember that you don’t need to deploy your entire application in a Wavelength Zone. You only need to deploy those latency-sensitive parts of your application that are really going to benefit from being deployed at the 5G edge. In our demo scenario, the API server and inference engine are located on the Wavelength Zone because one of the design goals of the application is low-latency processing of the inference requests. On the other hand, given that the web server doesn’t have those latency requirements, it doesn’t really need to be hosted on the Wavelength Zone.

Each Wavelength Zone is associated with a specific AWS Region, known as the “parent region”. For our experiment we have picked the Boston area, which is one of the first regions—along with San Francisco—in which the new Wavelength service was made available. Also, Wavelength instances are only accessible from 5G devices on a specific telecom provider network, in this case from those of Verizon customers in the Boston area 🧱

For the deployment of a multi-container application that benefits from this approach, we have used a number of well-known open source technologies. One of them has been K3s, a certified Kubernetes distribution originally developed by Rancher Labs and now hosted by the Cloud Native Computing Foundation (CNCF). K3s is a lightweight, production-grade distribution designed for organizations looking to run Kubernetes in resource-constrained environments, which makes it ideal for deployments at the edge. We have used a customized K3s image for this demo, but in the near future users will be able to deploy a K3s cluster by simply using its public Docker image 🤩

When bare-metal resources are available, OpenNebula users can also benefit from our latest, super-cool integration with Firecracker. Firecracker is a new open source virtualization technology—widely used by AWS as part of its Fargate and Lambda services—especially designed for serverless deployments. By running application containers (e.g. the K3s Docker image) as a Firecracker microVM, we immediately obtain the enhanced security and workload isolation of a traditional VM, but without undermining the speed and resource efficiency of a container.

Unfortunately, right now, bare-metal instances are not available in the current Wavelength zones, so we cannot use Firecracker for our 5G edge deployment, only at the associated AWS parent region (i.e. us-east-1). Thus, for Wavelength instances, and thanks to another great feature of OpenNebula, we can use Ubuntu’s LXD system containers to deploy K3s agents on the Wavelength resources.

As show in the figure, in order to deploy a containerized application composed of different components, OpenNebula allows to instantiate a K3s cluster across multiple hosts with mixed hypervisors and then let the customer deploy the application (e.g. using an helm chart or kubectl) by scheduling the components on the right resources, typically deploying the latency-sensitive components (i.e. the Inference Engine and the API server) on the Wavelength Zone, and the rest of components (i.e. Web Server) on the AWS parent region.

Integration of AWS Wavelength Resources within OpenNebula

OK, let’s get into some more detail… 🤓 The first step required to set up AWS Wavelength resources is the deployment of an AWS Virtual Private Cloud (VPC) with two zones: one is related to the associated AWS parent region, and one is related to the Wavelength Zone. We have then to associate to the VPC an Internet Gateway that is used to assign public IPs to resources that are deployed within the parent region, plus a Carrier Gateway that is used to assign carrier public IPs to the resources deployed on the Wavelength Zone.

In the VPC we have to define two subnets: one for the resources at the parent region and one for Wavelength Zone resources. The parent region subnet will be associated with the Internet Gateway to get public IPs, whereas the Wavelength subnet will be associated with the Carrier Gateway to get public IPs from the 5G carrier network.

The Carrier Gateway in a Wavelength Zone only allows access from the carrier’s 5G network. So, since the Wavelength zone resources cannot be accessed by using the internet, it is not possible to provision, configure and set up those resources by directly accessing them. In order to integrate Wavelength Zone resources with OpenNebula, we have to use the parent zone’s servers as “bastion hosts” to access Wavelength Zone resources via SSH, since they are only reachable through the private VPC subnet. Resources in the parent region can also be used to deploy those parts of our application that are not latency-sensitive or require high-bandwidth. 

Provisioning resources on regular AWS zones for deploying application parts that are not latency-sensitive is already possible for OpenNebula and can be performed by using its standard OneProvision tool. By using a bastion host and customized SSH configuration files, it is then possible to provision and configure instances on the Wavelength Zone and to add them as hosts to the OpenNebula front-end. Since OpenNebula uses SSH to perform any operation on the hosts, once bastion and Wavelength resources are set up, it is possible to deploy containerized applications (i.e. a K3s cluster) both on the parent region and on the Wavelength Zone instances.

To Boldly Go… 🚀

As we have confirmed first-hand, the new AWS Wavelength services opens up a whole new world of possibilities for companies willing to get the most out of edge computing. At the moment it is not possible to completely automate the provisioning of Wavelength Zone resources due to some limitations in our OneProvision tool, but we are working on it in order to incorporate features that will allow organizations using OpenNebula to easily provision AWS Wavelength resources and quickly deploy Kubernetes clusters and containerized applications on the 5G edge.

We will keep you updated about our ONEedge initiative and about the progress on integrating AWS Wavelength resources within your OpenNebula enterprise cloud. We will back soon to show some interesting use cases and some examples on how to provision containerized applications on the 5G edge using OpenNebula’s new container orchestration and edge features. Stay tuned! 📡