Charles N Wyble Journal

Earlier today I was going to copy some virtual machines onto a USB drive for the upcoming Defcon conference. I ran dmesg to see what the device name of the drive was, and my logs were filled with lost interrupt messages:


[ 8546.803754] rtc: lost some interrupts at 512Hz.
[ 8546.823739] rtc: lost some interrupts at 512Hz.
[ 8546.843730] rtc: lost some interrupts at 512Hz.

I did some research and came across:

this post  and it linked to this vmware documentation.

I have done the following things to improve performance:

   1. Set tools.timeSync = true in /etc/vmware/config.
   2. Turned off NTP and w32time in the guests and configured NTP on the host.
   3. Switched to the server kernel on the host and guest kernels. I am running Ubuntu Feisty (7.04) on the host and it was using the generic kernel for some reason. My kernel version is: Linux vmware-server 2.6.20-15-server #2 SMP Sun Apr 15 06:22:36 UTC 2007 x86_64 GNU/Linux
   4. Set /proc/sys/vm/swappiness to 0. I have 4 gigabytes of ram on this server so there shouldn't be any need to swap ever! :)
   5. Set vmware host memory settings (from the vmware console under the host->settings->memory tab) to allow some virtual memory to be swapped (the second radio button option). I set host ram for virtual machines as high as it could go. (Screenshot)
   6. Set host.useFastClock = FALSE and monitor_control.virtual_rdtsc = FALSE in /etc/vmware/config. This eliminates the RTC messages on the host, and greatly reduces the number of context switches.
   7. Installed vmware tools in all guests.
   8. Set all of my guests to use only 1 CPU.

For more information on linux performance tuning  please see these excellent texts. I am still learning about the topic myself.


So after making all of these changes I restarted vmware. Now for performance statistics. I gathered some rough ones before making the changes under a fairly typical workload (a few linux and windows guests running doing moderate levels of activity). The load was hovering around 5 and the context switches were 20k+.

While running all of the below tests I am running another instance of vmstat sampling every second to catch any major spikes that happen between sampling intervals.

Here are the stats after all machines have been booted and all services verified as running:

First the running virtual machines:

charles@vmware-server:~/vmware-machines/File Server$ ps -ef | grep vmx
charles  10979     1  6 20:02 ?        00:00:38 /usr/lib/vmware/bin/vmware-vmx -C /vmware-prod/home/charles/vmware-machines/Internal Server/Internal Server.vmx -@ ""
charles  10999     1  3 20:02 ?        00:00:18 /usr/lib/vmware/bin/vmware-vmx -C /vmware-prod/home/charles/vmware-machines/VPN Server/VPN Server.vmx -@ ""
charles  11022     1  3 20:02 ?        00:00:19 /usr/lib/vmware/bin/vmware-vmx -C /vmware-prod/home/charles/vmware-machines/PrepRep/PrepRep.vmx -@ ""
charles  11079     1  5 20:04 ?        00:00:22 /usr/lib/vmware/bin/vmware-vmx -C /vmware-prod/home/charles/vmware-machines/File Server/File Server.vmx -@ ""
charles  11103     1 17 20:05 ?        00:01:00 /usr/lib/vmware/bin/vmware-vmx -C /vmware-prod/home/charles/vmware-machines/W2k3 App Server/W2k3 App Server.vmx -@ ""
charles  11129     1 11 20:07 ?        00:00:26 /usr/lib/vmware/bin/vmware-vmx -C /vmware-prod/home/charles/vmware-machines/Public Server/Public Server.vmx -@ ""
charles  11153     1 16 20:09 ?        00:00:22 /usr/lib/vmware/bin/vmware-vmx -C /vmware-prod/home/charles/vmware-machines/W2K3-Production/W2K3-Production.vmx -@ ""
charles  11184 10451  0 20:11 pts/0    00:00:00 grep vmx
charles@vmware-server:~/vmware-machines/File Server$


And now the performance:

charles@vmware-server:~$ vmstat 10 30
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0    140  26500  37580 3338504    0    0    92   197   75  290  4 24 71  1
 5  0    140  26120  37620 3338956    0    0    28   489  195 3400  0  4 93  3
 6  0    140  25004  37644 3338980    0    0     1   826  166 2902  0  1 99  0
12  0    140  23952  37664 3339052    0    0     0    31  118 2839  0  0 99  0
 5  0    140  23828  37672 3339052    0    0     0    18  116 2828  0  0 99  0
 1  0    140  25832  37652 3335664    0    0     2   658  145 2907  0  1 99  0
 5  0    140  26080  37676 3335668    0    0     1    72  122 2843  0  1 99  0
 7  0    140  25708  37684 3335696    0    0     2    21  119 2864  0  1 99  1
 3  0    140  25336  37712 3335712    0    0     2    52  121 2848  0  1 98  1
 3  0    140  23972  37740 3335748    0    0     2   800  162 2921  0  1 97  2
 6  0    140  24104  37756 3336148    0    0     1    72  120 2877  0  0 98  1
 6  0    140  23856  37768 3336180    0    0     2    52  119 2875  0  1 99  0
 8  0    140  25740  37368 3333484    0    0     1  1128  160 2937  0  1 98  1
 5  0    140  25864  37388 3333660    0    0    14   706  189 3405  0  2 95  3
 5  0    140  25244  37412 3333680    0    0     2    54  133 2898  0  1 99  0
 5  0    140  24748  37432 3333724    0    0     1    81  124 2867  0  1 99  0
 6  0    140  27376  37456 3333744    0    0     1  1086  162 2889  0  1 98  1
 3  0    140  27000  37480 3335288    0    0     6   148  174 3031  0  5 94  2
 3  0    140  24776  37512 3337892    0    0    44   152  234 3086  0  3 96  1
 2  0    140  28008  37436 3334380    0    0   166  1179  192 3013  0  2 96  1
 3  0    140  27264  37476 3334432    0    0     7    80  141 2903  0  1 99  1
 5  0    140  26272  37500 3334472    0    0     2   498  119 2899  0  2 98  0
 4  0    140  25776  37516 3334488    0    0     2   641  136 2840  0  1 99  0
 6  0    140  25532  37536 3334636    0    0    13   448  140 3779  1  2 95  2
 6  0    140  25036  37560 3334656    0    0     1   670  129 2848  0  1 99  0
 6  0    140  25160  37580 3334684    0    0     2   198  130 2879  0  0 99  0
 5  0    140  24416  37600 3334712    0    0     1   824  149 2877  0  1 99  0
 6  0    140  23548  37632 3334764    0    0     5   410  156 2957  0  2 98  0
 5  0    140  23548  37640 3334796    0    0     1    41  109 2863  0  1 99  0
 1  1    140  24816  37552 3333048    0    0    27  1106  181 3069  0  3 94  3
charles@vmware-server:~$

I highly recommend doing the above items on Vmware Server and possibly on ESX server as well. Performance of my server farm is vastly improved.