Blog Article:

How Long Does It Take to Deploy 100 Virtual Machines?

Big Cloud

Oct 11, 2012

Work done by China Mobile in the Big Cloud Elastic Computing System 

The answer is less than 5 minutes!

We recently made an interesting test to see how fast can OpenNebula deploy a huge amount of VMs in short time. The testbed includes 200 hosts and runs VMs in local disk. For some reasons the scheduler policy must be setup as Packing Policy. To accelerate the deployment speed we use the cache patch (available at dev.opennebula.org). In this test we call “onevm create” command 100 times in a script to create 100 VMs as fast as possible.

In the first test we used OpenNebula mm_sched scheduler with the following configuration:

SCHED_INTERVAL = 60
MAX_VM = 300
MAX_DISPATCH = 30
MAX_HOST = 1

The result is that about 30VMs booted in 1 minute, the next 30 VMs  in 20 minutes, and the last 40 VMs booted finally in 60 minutes.  The next figure shows the overall evolution of the VM set (X-axis is the number of VMs, and Y-axis is the deployment time in seconds).

We thought that image caching can help in this scenario, but also a couple of optimizations where needed. So we made the second test with the following configurations:

SCHED_INTERVAL = 180
MAX_VM = 300
MAX_DISPATCH = 200
MAX_HOST = 1

Additionally, we increased the concurrency for the TM ssh driver (up to 100 threads) to not to limit the number of concurrent image transfers. Thanks Ruben (OpenNebula Chief Architect) for this hint.

As the figure shows, in this round, all VMs can be finished in one sched interval, being roughly 5 minutes the slowest boot time.

The conclusion is :

  1. Image Caching can greatly accelerate the deployment speed when OpenNebula uses local storage.
  2. To make image cache efficient some additional optimizations to the scheduler configuration may be needed.
  3. OpenNebula is good at scheduling a big amount of VM requests in short time.

3 Comments

  1. Cyril Rohr

    Interesting. Have a look at http://vimeo.com/39257324 for another experiment of how you can submit and have 100VMs deployed and running in ~3 minutes (start at 3:12 for the relevant part).

    Reply
  2. Fred Smit

    How big are the VM images ? What are their characteristics ? What class of servers? Networking speeds ? What is the underlying storage ?

    Reply
  3. Big Cloud

    Image size is about 20 GB. Servers are general 2 CPU nodes with 16G ram and 4 sata disks with raid5. We accelerate VM deployment with VM image cache in local disk. Without cache, one vm may take more than half an hour to deploy.

    Reply

Submit a Comment

Your email address will not be published. Required fields are marked *