Live scaling for VMs with fixed service offerings on KVM

[bernardodemarco]

Live scaling for VMs with fixed service offerings on KVM

This specification introduces a new feature enabling users to live scale VMs created with fixed service offerings on the KVM hypervisor.

Table of contents

• Live scaling for VMs with fixed service offerings on KVM
  • Table of contents
  • 1. Problem description
  • 2. Proposed changes
    • 2.1. Apache CloudStack definition of domain XMLs with KVM
    • 2.2. New cluster-wide settings to control the memory and vCPUs maximum capacity for live scaling
    • 2.3. scaleVirtualMachine API
  • 3. Conclusion and Limitations

1. Problem description

Currently, Apache CloudStack supports three types of compute offerings: fixed, custom constrained and custom unconstrained offerings. Fixed offerings have a fixed number of vCPUs, vCPU speed, and RAM memory. Custom constrained offerings accept a fixed vCPU speed, and a range of vCPUs and RAM memory in which users can select a value from. Lastly, custom unconstrained offerings accept an arbitrary amount of vCPUs, vCPU speed and RAM memory.

When using KVM as hypervisor, Apache CloudStack supports scaling Stopped instances by executing the scaleVirtualMachine API. During this process, since the VMs are stopped, their metadata is updated in the database and, thus, when the VMs are later started again, their respective domain XMLs are created with the updated attributes.

For running VMs, Apache CloudStack only supports live scaling VMs with custom constrained and custom unconstrained compute offerings¹. For VMs with fixed compute offerings, users necessarily need to stop the VMs, change their service offerings through the scaleVirtualMachine API and start them again. However, depending on the criticality of the applications running on the VMs, the downtime caused by the scaling process is highly undesirable.

2. Proposed changes

To address the described problem, this specification introduces the feature of live scaling VMs with fixed service offerings when using the KVM hypervisor on Apache CloudStack cloud environments. A high-level design of the feature is presented, briefly describing the proposed changes to the generation of guest VM domain XMLs, the scaleVirtualMachine API workflows and the global settings controlling the scaling process.

2.1. Apache CloudStack definition of domain XMLs with KVM

During the process of deploying virtual machines with KVM, a VM transfer object (VirtualMachineTO) is implemented given a VirtualMachineProfile, which is an object that stores the attributes of the VM. The implementation of VMs transfer objects is illustrated in the following activity diagram:

When configuring the memory and vCPU attributes for the VMs, Apache CloudStack implements the following workflow:

Regarding the definition of domain XMLs for guest VMs, the following boilerplate is used by the Apache CloudStack Agent:

<domain type='kvm' id='4'>
    <!--(...)-->
    <maxMemory slots='16' unit='KiB'>1930240</maxMemory>
    <memory unit='KiB'>1048576</memory>
    <currentMemory unit='KiB'>1048576</currentMemory>
    <vcpu placement='static' current='1'>2</vcpu>
    <!--(...)-->
    <cpu mode='custom' match='exact' check='full'>
        <!--(...)-->
        <numa>
            <cell id='0' cpus='0-1' memory='1048576' unit='KiB'/>
        </numa>
        <!--(...)-->
    </cpu>
</domain>

Regarding vCPU configuration, the content of the vcpu element defines the maximum number of vCPUs that can be allocated for the guest VM. Its current property defines the number of vCPUs that are effectively active². As can be analyzed from the above activity diagrams, for custom constrained compute offerings, the content of the vcpu element is defined as the offering's maximum number of vCPUs. For custom unconstrained offerings, the element's content is defined as the vm.serviceoffering.cpu.cores.max global setting value, defaulting to the host maximum CPU capacity in scenarios that the global setting is equal to zero.

As for guest VMs memory configuration, the maxMemory element defines the maximum memory allocation for the VM. The memory element represents the maximum amount of memory available for the VM at boot time. The currentMemory represents the actual allocation of memory for the VM³. Lastly, Libvirt currently requires the specification of NUMA nodes to enable memory hotplug. Each cell element represents a NUMA node. The cpus attribute define the range of CPUs that are part of the node. The memory attribute represents the amount of memory in use by the VM⁴.

As can be noticed from the VM configuration workflow, for custom constrained compute offerings, the maxMemory element content is defined as the offering's maximum number of memory. For custom unconstrained offerings, the element's content is defined as the vm.serviceoffering.ram.size.max global setting value, defaulting to the host maximum memory capacity in scenarios in which the global setting is equal to zero.

Hence, to address the live scaling of VMs with fixed offerings, the first validation on the configuration of memory and vCPU for a guest VM, that checks whether the VM is dynamically scalable, will be modified. Currently, it considers a VM to be dynamically scalable if its offering is dynamic (that is, either the amount of vCPUs, vCPU speed or RAM memory is not specified); the VM's dynamically_scalable property is true; and the global setting enable.dynamic.scale.vm is true.

Therefore, the check for dynamic offerings will be removed from the above mentioned validation. As a consequence of that, when the dynamically_scalable property of VMs is true and the enable.dynamic.scale.vm global setting is true, then the domain XMLs of guest VMs will always have a range of memory and vCPUs to be scaled up to, even when the VMs are created from fixed compute offerings.

To define the upper limit of the memory and vCPU live scaling range, the kvm.cpu.dynamic.scaling.capacity and kvm.memory.dynamic.scaling.capacity global settings will be considered (see new cluster-wide settings). Therefore, the maximum number of vCPUs will be retrieved from the kvm.cpu.dynamic.scaling.capacity setting value and the maximum number of memory will be retrieved from the kvm.memory.dynamic.scaling.capacity value. When these settings are equal to zero, the host maximum capacity of CPU and memory will be considered.

This procedure will also be applied for custom constrained offerings to enable seamless live scaling between all types of compute offerings existing in Apache CloudStack. Thus, at the KVM level, all types of VMs hosted in a given host will be homogeneous regarding the CPU and memory upper limits. At the Apache CloudStack level, on the other hand, the Management Server will be responsible for validating the computing resources ranges defined in the constrained offerings.

2.2. New cluster-wide settings to control the memory and vCPUs maximum capacity for live scaling

As mentioned on the domain XMLs definition section, the vm.serviceoffering.cpu.cores.max and vm.serviceoffering.ram.size.max global settings are used to control de maximum amount of vCPUs and memory to which VMs with custom unconstrained offerings can be live scaled to. However, both settings are also currently used to limit the maximum amount of CPU and RAM that can be defined for compute offerings and allocated to VMs.

To segregate goals and responsibilities, two new cluster-wide settings will be introduced:

Name	Type	Scope	Description
kvm.memory.dynamic.scaling.capacity	Integer	Cluster	Defines the maximum memory capacity in MiB for which VMs can be dynamically scaled to with KVM. The kvm.memory.dynamic.scaling.capacity setting's value will be used to define the value of the <maxMemory /> element of domain XMLs. If it is set to a value less than or equal to 0, then the host's memory capacity will be considered.
kvm.cpu.dynamic.scaling.capacity	Integer	Cluster	Defines the maximum vCPU capacity for which VMs can be dynamically scaled to with KVM. The kvm.cpu.dynamic.scaling.capacity setting's value will be used to define the value of the <vcpu /> element of domain XMLs. If it is set to a value less than or equal to 0, then the host's CPU cores capacity will be considered.

Therefore, both settings will be used to exclusively control the maximum live scaling capacity for memory and vCPUs. To maintain compatibility with the current behavior, the values of the vm.serviceoffering.cpu.cores.max and vm.serviceoffering.ram.size.max global settings will be used to populate the initial values of the new cluster-wide settings.

2.3. scaleVirtualMachine API

The following activity diagram illustrates the current implementation of the VM scaling process:

As can be noticed, when upgrading running VMs, after performing some general validations, the Management Server checks if the VM is running on KVM, and if its offering is not dynamic (that is, vCPU, vCPU speed and RAM memory are specified for the offering). If the VM meets these conditions, then an exception is thrown, informing the end user that KVM does not support live scaling VMs with fixed compute offerings.

Therefore, the above-mentioned check for dynamic offerings will be removed from the scaling workflow. This is possible because the domain XMLs of guest VMs will be prepared to support live scaling when the global setting enable.dynamic.scale.vm and the VM's dynamically_scalable property are set to true.

Additionally, it is relevant to note that the current workflow does not update the VM's CPU quota percentage based on its new CPU frequency. To address this, the current implementation will be extended so that, before building the ScaleVmCommand, the new CPU quota percentage is calculated and included in the command payload, along with a flag indicating whether the CPU cap has changed between the old and new service offerings.

At the Agent side, the following workflow will be executed:

Thus, if both the old and the new service offerings have CPU limitation enabled, the domain's CPU quota will be updated on the fly. If the old service offering does not have CPU limitation enabled but the new one does, the domain's CPU quota and period parameters will be set. Conversely, it the old offering has CPU limitation enabled and the new one does not, these parameters will be removed from the domain.

To update the CPU scheduling parameters of a domain on the fly, the virDomainSetSchedulerParameters Libvirt API will be used. To remove CPU limitation from a domain, the API will be called with the vcpu_quota parameter set to 17,592,186,044,415. Due to a Libvirt regression (see Libvirt regression description), it is not possible to set this value to -1. However, setting it to 17,592,186,044,415 has the same effect, ensuring that CPU limitations are effectively removed from running domains despite the Libvirt's constraint.

3. Conclusion and Limitations

This proposal introduces support for live scaling of VMs with fixed service offerings when using the KVM hypervisor in Apache CloudStack environments. As a limitation, it is important to highlight that live scaling will not be supported for existing domains created prior to upgrading to the Apache CloudStack release in which this patch is included. This is because their domain XMLs are not prepared to support live CPU and memory scaling.

To enable live scaling for such VMs, the following steps are required:

1. Ensure that the enable.dynamic.scale.vm global setting is enabled.
2. Ensure that the VM's template is marked as dynamically scalable.
3. Ensure that the VM itself is marked as dynamically scalable.
4. Stop and start the VM so that its domain is recreated with the required structure for live scaling.

Footnotes

1. The global setting enable.dynamic.scale.vm controls whether the live scaling feature is enabled or not. ↩

2. More information about the configuration of guest VMs CPU attributes can be found at the Libvirt documentation. ↩

3. More information about the configuration of guest VMs memory attributes can be found at the Libvirt documentation. ↩

4. More information about the configuration of guest NUMA topology can be found at the Libvirt documentation. ↩

[weizhouapache]

@bernardodemarco
proposal looks good overall.

If I understand correctly, this means setting maxMemory for VMs with dynamic scaling enabled, right?

I am wondering whether using global configurations is the best approach. In most cases, users tend to scale CPU/memory relative to the current VM size rather than relying on a fixed global limit.

For example:

• Relative scaling (e.g., doubling): 2GB → 4GB, 4GB → 8GB, 8GB → 16GB
• Fixed increment scaling: 2GB → 4GB, 4GB → 6GB, 8GB → 10GB

Would it make more sense to support scaling policies based on the current allocation instead of a global maximum?
The global maximum can remain