Work done by China Mobile in the Big Cloud Elastic Computing System

The aim of this post is to describe a new OpenNebula LXC driver (OneLXC) developed by China Mobile to allow the management of hosts and the deployment of lxc domains in OpenNebula using the LXC hypervisor.

Features

OneLXC mainly consists of two components:

• IM_MAD: a series of remote scripts that are able to monitor the remote hosts
• VMM_MAD: a series of remote scripts to manage lxc domains.

OneLXC is very similar with kvm driver because libvirt can support kvm and lxc. They both use virsh command to monitoring hosts and operating virtual machines.

Currently OneLXC supports some simple functions as follows:

• monitoring host information, for example, cpu and memory
• deploy/delete/monitoring(poll) lxc domains and their info

Developing Enviroment

• three host machines with ubuntu 12.04 operating system（amd64
• opennebula-3.2.1
• libvirt-0.9.8
• lxc-0.7.5

How to install and use OneLXC Driver?

To install the OneLXC driver run the “install.sh” script provided. This script will copy the necessary files into the OpenNebula installation tree alongside OpenNebula itself.

Driver Configuration

In order to enable the OneLXC driver, it is necessary to modify “oned.conf” accordingly. This is achieved by setting the IM_MAD and VM_MAD options as follows:

#-------------------------------------------------------------------------------
# LXC Information Driver Manager Configuration
# -r number of retries when monitoring a host
# -t number of threads, i.e. number of hosts monitored at the same time
#-------------------------------------------------------------------------------
IM_MAD = [
name = "im_lxc",
executable = "one_im_ssh",
arguments = "-r 0 -t 15 lxc" ]

#-------------------------------------------------------------------------------
VM_MAD = [
name = "vmm_lxc",
executable = "one_vmm_exec",
arguments = "-t 15 -r 0 lxc",
default = "vmm_exec/vmm_exec_lxc.conf",
type = "lxc" ]
#-------------------------------------------------------------------------------

The name of the driver needs to be provided at the time of adding a new host to OpenNebula. For example, we can use command “onehost create” and “onehost list” to create and show the hosts.

After adding hosts, we can use onevm create and onevm show <vm_id> to deploy a lxc domain and show its informations, for  example as follows:

Driver files

The OneLXC driver package contains the following files. Note that they are referenced using $ONE_LOCATION as the base directory, therefore meaning a self-contained installation of OpenNebula.

• $ONE_LOCATION/etc/vmm_exec/vmm_exec_lxc.conf: Configuration file to define the default values for the LXC domain definitions.
• $ONE_LOCATION/var/remotes/vmm/lxc/: Scripts used to perform the operations on the lxc domains. These files are called “remotes”, meaning they are copied to the remote hosts and executed there.
• $ONE_LOCATION/var/remotes/im/lxc.d/: Scripts used to fetch information from the remote hosts (memory, cpu use…). These scripts are copied to the remote hosts and executed there.
• oned.conf: Example OpenNebula configuration file with the LXC drivers enabled.

Source files

• src/vmm/LibVirtDriverLXC.cc: The libvirt driver to generate the lxc domain’s deployment configuration file.

Image files

Different from kvm and xen image file, the lxc domain’s image actually is a directory called “rootfs”. For the sake of transfering easily, we compress it, copy it to the target host and decompress it again in the destination. Note： the file permission of  /usr/bin/sudo of lxc domain must be 4755.Otherwise, it can not use root privilege to execute command in lxc domain.

Virtual Machine’s Configuration file

NAME = lxc_2
CPU = 1
MEMORY = 1024
VCPU = 2

DISK = [
source = "/opt/nebula/images/lxc.tar.gz"
]

Bugs and problems

• There is a synchronous problem between the “cancel” operation and “tm_delete.sh” for OneLXC driver.
• OneLXC driver cann’t implement “reboot”, “shutdown” and “restart” operations because maybe libvirt or LXC doesn’t support.
• How to generate the lxc “config” file in “rootfs” directory dynamically? Because each lxc domain has different image file path.
• perhaps there are a lot of bugs but could not find them

Source Code Download

Because lxc driver function is not perfect，I will submit the code later. If someone want it now, you can download the draft version from here:

https://github.com/cmri/opennebula-3.2.1-lxc.git