Work done by China Mobile in the Big Cloud Elastic Computing System
The answer is less than 5 minutes!
We recently made an interesting test to see how fast can OpenNebula deploy a huge amount of VMs in short time. The testbed includes 200 hosts and runs VMs in local disk. For some reasons the scheduler policy must be setup as Packing Policy. To accelerate the deployment speed we use the cache patch (available at dev.opennebula.org). In this test we call “onevm create” command 100 times in a script to create 100 VMs as fast as possible.
In the first test we used OpenNebula mm_sched scheduler with the following configuration:
SCHED_INTERVAL = 60 MAX_VM = 300 MAX_DISPATCH = 30 MAX_HOST = 1
The result is that about 30VMs booted in 1 minute, the next 30 VMs in 20 minutes, and the last 40 VMs booted finally in 60 minutes. The next figure shows the overall evolution of the VM set (X-axis is the number of VMs, and Y-axis is the deployment time in seconds).
We thought that image caching can help in this scenario, but also a couple of optimizations where needed. So we made the second test with the following configurations:
SCHED_INTERVAL = 180 MAX_VM = 300 MAX_DISPATCH = 200 MAX_HOST = 1
Additionally, we increased the concurrency for the TM ssh driver (up to 100 threads) to not to limit the number of concurrent image transfers. Thanks Ruben (OpenNebula Chief Architect) for this hint.
As the figure shows, in this round, all VMs can be finished in one sched interval, being roughly 5 minutes the slowest boot time.
The conclusion is :
- Image Caching can greatly accelerate the deployment speed when OpenNebula uses local storage.
- To make image cache efficient some additional optimizations to the scheduler configuration may be needed.
- OpenNebula is good at scheduling a big amount of VM requests in short time.
Interesting. Have a look at http://vimeo.com/39257324 for another experiment of how you can submit and have 100VMs deployed and running in ~3 minutes (start at 3:12 for the relevant part).
How big are the VM images ? What are their characteristics ? What class of servers? Networking speeds ? What is the underlying storage ?
Image size is about 20 GB. Servers are general 2 CPU nodes with 16G ram and 4 sata disks with raid5. We accelerate VM deployment with VM image cache in local disk. Without cache, one vm may take more than half an hour to deploy.