Managing resource usage in private or community clouds where the provisioning model is not pay-per-use oriented is often complicated. Resources are available to end users seemingly for free which means that one of the main motivators for responsible resource usage — a monthly bill — is missing. This can eventually lead to a situation in which the pool of available resources is depleted and a large portion of existing allocations is underused, unused, or entirely forgotten. For a resource provider interested in offering fixed amount of resources to the largest possible number of users, this issue needs to be addressed.

Depending on a number of factors such as size of the infrastructure, type of the provided service, and technical expertise available to the provider, this issue can be addressed in a variety of ways. Commonly considered approaches are:

• Advanced scheduling with so-called “fair-share” capabilities
• Restricted resource allocation time for faster allocation turnover
• Restrictive per-user or per-group quotas
• Non-technical solutions, e.g. sending monthly usage reports to department heads or community representatives

This post explores a more benevolent iteration on the second approach meant to improve resource allocation turnover as well as limit the number of forgotten long-running virtual machines posing a considerable security risk to the infrastructure.

Design Concepts
The goal was to create a simple mechanism capable of:

• Setting expirations on running virtual machines
• Notifying users about approaching expiration dates
• Letting users move expiration dates to the future (extending VM lifetime)

while reusing as many existing features of OpenNebula as possible to make long-term maintainability and compatibility feasible. The resulting implementation — a tool called Berta — developed by CESNET’s Department of Distributed Computing is described below.

Initially, Berta will contact OpenNebula and request a list of running virtual machines, users, and groups. An OpenNebula-supported scheduled action is set for all running virtual machines with a configurable future execution time. A real-world example would be “terminate-hard” in three months’ time. This ends step (1).

In step (2), Berta sends a notification e-mail to all users with previously scheduled actions nearing execution time. This email contains basic information about what’s going to happen and when, along with brief instructions on how to prevent it. Depending on configuration, users have generally a few weeks to take action and/or ask user support for help.

In step (3a), users who decided to extend the lifetime of their virtual machines log in via Sunstone GUI and remove scheduled actions. Using OCA (XML RPC) or OpenNebula command line tools is also possible. New actions will be automatically added by Berta during the next cycle, coming back to step (1).

Step (3b) does not require any user intervention and represents automated execution of scheduled actions by OpenNebula. Forgotten or intentionally ignored resources are freed.

Installation Guide
There are multiple ways of installing Berta. For convenience, we offer a natively packaged version containing the entire runtime in an embedded form (no external dependencies). The native packages are available from The EGI AppDB.


BASE_URL=http://repository.egi.eu/community/software/berta
RLS=2.x/releases
DSTR=centos-7-x86_64

wget $BASE_URL/$RLS/repofiles/$DSTR.repo -O /etc/yum.repos.d/berta.repo
yum install -y berta sendmail

Since the tool is written in Ruby, installation using RubyGems(.org) is also possible.

Configuration Guide
Adjust the following options in ‘/etc/berta/berta.yml’, provided you installed a native rpm package referenced above:

opennebula:
# OpenNebula secret (privileged account)
secret: ‘oneadmin:opennebula’
# OpenNebula XML RPC endpoint
endpoint: ‘http://localhost:2633/RPC2’

expiration:
# How far in the future should a new action be scheduled
offset: 3 months
# Type of action to schedule
action: terminate-hard

notification:
# How long before action execution should the user be notified
deadline: 14 days

After configuration, change the content of `/etc/berta/email.erb` to your liking (see ERB documentation if you are not familiar with its syntax). Variables you can use in your template are documented in the provided example.

Berta uses commonly available sendmail command (system-wide executable binary) to send emails to users. This requires a properly configured and running local sendmail MTA service, at least as a smarthost passing messages to a relay server. Please, consult sendmail documentation for configuration details.

First Run
It is recommended to start with a dry run, to see what Berta decides to do and send to users. In this mode, it will not make changes in OpenNebula or send actual emails. If everything goes smoothly, continue by enabling Berta only for a small subset of users/groups using the ‘include’ filter and expand the affected fraction of users over time.

In order to run Berta periodically, enable and start the bundled-in ‘berta-cron’ service which runs every six hours by default.