12 Comments

1. 

   Poil on Jul 27, 2012 at 8:39 am

   Hi,

   It doesn’t work on Ubuntu 12.04, I’m looking why.

   root@vmhost02l:/opt/MegaRAID/MegaCli# vm-balancer.py
   Found cset at /usr/bin/cset
   Found numactl at /usr/bin/numactl
   Found virsh at /usr/bin/virsh
   cset: –> modified cpuset “VMBS0”
   cset: –> modified cpuset “VMBS1”
   cset: moving following pidspec: 31702,23797,31990,31842,31980,31762,31729,31786,23800,23801,31733,31784,31785,32022,32020,32021,32019,31846,32000
   cset: moving 19 userspace tasks to /VMBS0
   [==================================================]%
   cset: **> 19 tasks are not movable, impossible to move
   cset: done
   cset: moving following pidspec: 30560,30581,23881,31328,23869,30585,31324,30595,30589,30597,23880,31322,31323,31329,31325,31327,31326,30399
   cset: moving 18 userspace tasks to /VMBS1
   [==================================================]%
   cset: **> 18 tasks are not movable, impossible to move
   cset: done

   For your information, you can replace
   ps -e | grep kvm | grep -v –
   [pythoncode]
   with
   pgrep kvm

   Best regards

   Reply
2. 

   Poil on Jul 27, 2012 at 9:27 am

   Ok, I’ve patch cpuset, it’s now working.
   Sorry to have troubled you.

   Reply
3. 

   Poil on Jul 27, 2012 at 9:36 am

   Another question is there a difference between using vm-balancer (which use cset) and virsh cpupin ?

   Reply
4. 

   Shankhadeep on Jul 27, 2012 at 9:59 pm

   Virsh cpu pin does basically the same thing for cpus however you need cpuset support if you want to control memory location as well. Libvirt cpuset mechanism is clunky and requires the definition file to be changed everytime you want to pin cpu and memory. CSET just works in most cases for this function.

   Reply
5. 

   Poil on Jul 31, 2012 at 10:03 am

   On ubuntu 12.04, VM are already in cgroups, so I’ve written a small stupid script, I share it here :

   #!/bin/bash
   # Auteur : Benjamin DUPUIS
   # Role : Test pinning vCPU on pSOCKET
   # Date : 20/07/2012
   # Version : 1.00
   #———————————————-
   # Historique :
   # 20/07/2012 – v1.00 BDU: Initial version
   ################################################

   VM_DEF_DIR=’/etc/libvirt/qemu/’
   SYS_DEV_CPU=’/sys/devices/system/cpu/’

   [[ ! -d ${VM_DEF_DIR} ]] && exit 2
   [[ ! -d ${SYS_DEV_CPU} ]] && exit 3

   #———————————————-
   # Host
   cd ${SYS_DEV_CPU}

   # Number of core
   cfgCORE=$(for cpuCore in $(ls -d cpu[0-9]*); do
   echo -n “${cpuCore##*cpu} ”
   cat ${cpuCore}/topology/core_siblings_list
   done)

   # Number of socket
   CoreList=$(echo “${cfgCORE}” | cut -d” ” -f2 | sort)
   SocketList=$(echo “${cfgCORE}” | cut -d” ” -f2 | sort | uniq)
   nSocket=$(echo “${SocketList}” | wc -l)

   i=1
   firstr=0
   for socket in ${SocketList}; do
   [[ ${firstr} == 0 ]] && oldsocket=${socket} && firstr=1
   if [[ ${socket} != ${oldsocket} ]]; then
   i=$((${i} + 1))
   echo “${socket} != ${oldsocket}”
   fi
   tabsocket[${i}]=$(echo “${cfgCORE}” | awk -v Sock=${socket} ‘BEGIN { first=0 } {
   if ($2 ~ Sock) {
   if (first!=1) {
   str=$1
   first=1
   } else {
   str=str”,”$1
   }
   }
   } END { print str }’)
   oldsocket=${socket}
   done

   #———————————————-
   # Virtuals Machines

   cd ${VM_DEF_DIR}
   # nombre de vCPU alloués
   TOTAL_vCPU=$(cat * | awk ‘BEGIN { CPUn=0; FS = “[]” } /cpu/ { CPUn=CPUn+$3 } END { print CPUn }’)

   # number of vCPU/socket max
   MID_vCPU=$((${TOTAL_vCPU} / ${nSocket}))

   curCore=1
   for myVM in $(ls); do
   vmCPU=”${vmCPU} ${myVM%%.*}”
   nvmCPU=$((${nvmCPU} + $(cat ${myVM} | awk ‘BEGIN { FS = “[]” } /cpu/ { print $3 }’)))
   if [[ ${nvmCPU} -ge ${MID_vCPU} ]]; then
   cfgVMCPU[${curCore}]=${vmCPU}
   debugVMCPU[${curCore}]=${nvmCPU}
   curCore=$((${curCore} + 1))
   nvmCPU=0
   vmCPU=””
   fi
   done
   cfgVMCPU[${curCore}]=${vmCPU}
   debugVMCPU[${curCore}]=${nvmCPU}

   #———————————————-
   # Configuration

   for j in $(seq 1 ${nSocket}); do
   echo “On Core n° ${j} : ${cfgVMCPU[${j}]} For a total of ${debugVMCPU[${j}]} vCPUs”
   for vm in ${cfgVMCPU[${j}]}; do
   # num vCPU sur la VM
   vmncpu=$(awk ‘BEGIN { CPUn=0; FS = “[]” } /cpu/ { CPUn=CPUn+$3 } END { print CPUn }’ ${VM_DEF_DIR}/${vm}.xml)
   # bind of each vCPUs on a same core
   for vmcpu in $(seq 1 ${vmncpu}); do
   # Bind vcpu on a socket
   virsh vcpupin ${vm} $((${vmcpu} – 1 )) ${tabsocket[$j]}
   # Bind KVM process on the same processor/memory node
   cset set -s /libvirt/qemu/${vm} -c ${tabsocket[$j]} -m $(( $j – 1 ))
   done
   done
   done

   Reply
6. 

   Poil on Jul 31, 2012 at 2:39 pm

   Hum,

   You can remove my comment
   Here is the script in the good version

   http://own-you.com/tmp/cpu_pinning.sh

   Reply
7. 

   Shank on Jul 31, 2012 at 3:09 pm

   Hi, thanks for sharing, can you share the cpu set patch you did with a small explanation? I had issues making cset work with libvirt also, it wouldn’t recognize the directory structure when libvirt created subsets first. I had to turn off cpuset cgroup support in libvirt and use it only with cpuset.

   Reply
8. 

   Poil on Jul 31, 2012 at 4:11 pm

   I was using the cpuset binary in v1.0 but I have to fix it by adding a prefix to all cset path.

   In v2.0 (http://own-you.com/tmp/cpu_pinning.sh) I don’t use cpuset binary anymore

   My cpuset is mounted in /cpuset
   My libvirt subset are in /cpusets/libvirt/qemu/${vm}/cpuset.cpus

   Reply
9. 

   Shankhadeep Shome on Jul 31, 2012 at 4:50 pm

   Ah ok I just saw the script, thanks for the contribution 🙂

   Reply
10. 

    Poil on Aug 1, 2012 at 2:44 pm

    I’ve upload v2.10
    Changelog: Use only started VM and exec if diff since last exec

    I run it all 15 mins on my KVM hosts

    Reply
11. 

    Mark on Nov 8, 2012 at 6:43 pm

    @Poli – Thanks for this script!

    Very useful, I modified it to use virsh dumpxml instead of reading the configs from /etc/libvirt/qemu/*.xml as this seemed the best way to integrate it with OpenNebula.

    I also changed the line to mount /cpusets to be slightly more modern. Also just ‘mount -t cpuset’ always returned nothing for me – apparently this worked for you?

    [[ -z $(mount|grep ‘/cpusets’) ]] && mount -t cgroup none /cpusets/ -o cpuset

    Reply
12. 

    Poil on Nov 26, 2012 at 10:10 am

    Hi,

    Yes on ubuntu “mount -t cpuset” is working, but I think we don’t need it, cgroup already exist “cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)”
    So, I think we can remove cpuset dependancy and use this path. How is it on Suse ?

    I’ve also added a check at the beggining
    ## Sub Check if cpuset directory exist for this VM
    for vm in ${CUR_VM}; do
    if [[ ! -d “/cpusets/libvirt/qemu/${vm}” ]]; then
    echo “Error cpuset for VM ${vm} not found”
    rm -f ${LIB_DIR}/cpupinning.cfg
    exit 1
    fi
    done

    Reply