OpenNebula for C/I – Running over 4 million VMs per year

“WE NEEDED SOMETHING THAT WAS DESIGNED TO CREATE VMs ON DEMAND”

The Qt Company develops and delivers the Qt development framework under commercial and open source licenses. They enable the reuse of software code across all operating systems, platforms and screen types, from desktops and embedded systems to wearables and mobile devices. Qt is used by more than one million developers worldwide and is the leading independent technology behind millions of devices and applications.  

Before starting to use OpenNebula, Qt had a performance problem with the CI system. The autotests took over 30 times longer to run compared to a normal situation. The company identified different bottlenecks: 

• The bandwidth with which the virtual machines stored local data to their virtual hard drives was an issue. The servers, on which our virtual machines ran, store all their data on a centralized storage. That storage had to be cached somehow.
• Several generations of hardware installed, with different speeds sharing the same bandwidth.
• The centralized storage repeatedly went down when trying to start more than 200 VMs in the CI. And consequently, all builds and autotest runs were affected.

In addition, they were using Jenkins, the open source CI/CD automation software. “Jenkins was not intended to run VMs that pop in and out of existence. We had to create nodes via Python scripts and the state machines became painful,” said Tony Sarajärvi, CI Tech Lead at The Qt Company. “We needed something that was designed to create VMs on demand.” This fact turned out to be the key factor for choosing OpenNebula.

“With open source non-proprietary code we can go deep into the root causes of problems and fix drivers if that’s needed to make our VMs run smoothly without hiccups.”

Tony Sarajärvi, CI Tech Lead

“OUR NEW SPINAL CORD IS BASED ON OPENNEBULA”

Qt was using VMware vSphere to create and run VMs, but they were looking for a change. The VMWare’s ESXi licenses were quite expensive for the size of the company and the amount of number crunching hardware they wanted to use. They wanted something that was broadly used and open source so that the community could fix bugs and not everything would fall into their own hands. So they looked for open-source hypervisors, KVM. OpenStack was their initial option. However, they quickly learned about OpenNebula and decided to forge ahead with it.

“OpenNebula was more lightweight than OpenStack. We don’t need all the fancy modules it has. After more than three years using OpenNebula in production, we can confirm that it is a great tool for doing the simple task of spawning millions of virtual machines per year that live only a matter of minutes, before being disposed of. That’s what our CI is all about. It verifies developers commits in multiple different environments, after which we either have a positive or negative result” said Sarajärvi.

The swap to OpenNebula came along with the new hardware changes. With open source non-proprietary code they went deeper into the root causes of problems and fixed drivers to make their VMs run smoothly without hiccups.

“OpenNebula was more lightweight than OpenStack. After more than three years using OpenNebula in production, we can confirm that it is a great tool.”

Tony Sarajärvi, CI Tech Lead

Qt is currently running over 4 million virtual machines per year on OpenNebula. OpenNebula is hosted in their stack of servers and meant to take care of the infrastructure, with approximately 30 Dell PowerEdge R430 servers with dual 20 core CPUs and 400 GB of RAM doing the heavy lifting of running the builds themselves.

WITH OPENNEBULA, Qt WILL CONTINUE EXPANDING TO NEW AREAS

“We have already stumbled upon the limit that it doesn’t support Parallels for example. But we’re working on that ourselves so that we can use macOS hardware as a deployment platform as well,” said Sarajärvi. “So in that sense, OpenNebula with its open source enables us to continue expanding to areas where it wasn’t originally even intended”.