qemu swap usage on vm.swappiness = 0

[Guest seccentral]

Hello

This is my first post :) *hi*

 

Now to the matter at hand:

 

I have a few servers that run centos 6 64bit and have 64GB ram each. I use those boxes as hypervisors. I have about a few hundred vps's virtualized with KVM.

 

To my astonishment i found at rather random times that some of my VPS's are getting their pids killed for apparently no reason untill i checked the memory usage.

Now, on each hypervisor i have about 40 up to 50 GB of ram(on average) allocated and the rest are free (as provided by free -m and top). The swap space however is a different issue (not knowing that i needed it anyway i had it set to 1GB). 1GB was fine up to the point where i deployed the kvm vps's and let them run for about a day or two.

Some nodes seem to fill up swap. Some nodes never do. all machines are the same hardware wise.

So i set vm.swappiness to zero on all nodes in hope to solve this issue but it doesn't do it.

 

I was left with the option of increasing the swap space by making a swapfile of 5GB somewhere and swapon it.

 

This is however a 'hack' that i'd rather avoid.

I've posted this question on kvm, qemu forums and mailing lists with no resolution (not even a reply).

 

I'm baffled by the fact that swap is never used on some nodes that run the same number of vps's that have the same configuration (i have some standard plans 1gb ram 10gb hdd etc running ubuntu 12.04 lts inside all vps nodes)

so if for instance

node 1 - 20 vps instances - zero swap usage

node 2 - 20 vps instances - swap fills about 20mb/day (average number, actual size varies randomly from my perspective)

 

top -M with Op enter says this on the swap using node:

Code:

top - 14:33:46 up 18 days, 5:11, 1 user, load average: 0.21, 0.19, 0.13

Tasks: 324 total, 1 running, 323 sleeping, 0 stopped, 0 zombie

Cpu(s): 0.3%us, 0.9%sy, 0.0%ni, 98.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 62.886G total, 39.360G used, 23.525G free, 8749.754M buffers

Swap: 5117.977M total, 1115.953M used, 4002.023M free, 10.496G cached

 

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND

6833 qemu 20 0 1441m 761m 4968 S 0.0 1.2 39:37.41 55m qemu-kvm

5173 qemu 20 0 1439m 742m 4968 S 0.3 1.2 34:32.42 51m qemu-kvm

11653 qemu 20 0 1437m 729m 4968 S 0.3 1.1 34:24.61 50m qemu-kvm

13449 qemu 20 0 1445m 738m 5008 S 0.3 1.1 36:27.34 49m qemu-kvm

7783 qemu 20 0 1451m 753m 4996 S 0.3 1.2 33:49.16 49m qemu-kvm

4873 qemu 20 0 1451m 745m 4968 S 0.7 1.2 31:48.03 48m qemu-kvm

5928 qemu 20 0 1448m 755m 4968 S 0.3 1.2 34:39.14 47m qemu-kvm

11294 qemu 20 0 1434m 730m 5004 S 0.3 1.1 33:11.42 47m qemu-kvm

10016 qemu 20 0 1450m 757m 4968 S 0.0 1.2 35:15.59 47m qemu-kvm

14265 qemu 20 0 1441m 745m 4968 S 0.3 1.2 38:34.28 46m qemu-kvm

9065 qemu 20 0 1450m 748m 4968 S 0.3 1.2 31:58.57 44m qemu-kvm

12901 qemu 20 0 1448m 756m 4968 S 0.0 1.2 36:51.08 44m qemu-kvm

8028 qemu 20 0 1446m 738m 4968 S 0.3 1.1 32:18.04 43m qemu-kvm

 

etcetera

and this on a node that doesnt use swap:

Code:

top - 14:32:00 up 20 days, 19:17, 1 user, load average: 0.03, 0.07, 0.08

Tasks: 320 total, 1 running, 319 sleeping, 0 stopped, 0 zombie

Cpu(s): 0.3%us, 0.3%sy, 0.0%ni, 99.2%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st

Mem: 62.886G total, 56.751G used, 6281.445M free, 14.480G buffers

Swap: 5021.977M total, 0.000k used, 5021.977M free, 20.504G cached

 

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND

1 root 20 0 19232 1520 1232 S 0.0 0.0 0:03.10 0 init

2 root 20 0 0 0 0 S 0.0 0.0 0:04.15 0 kthreadd

3 root RT 0 0 0 0 S 0.0 0.0 0:03.81 0 migration/0

4 root 20 0 0 0 0 S 0.0 0.0 0:05.79 0 ksoftirqd/0

5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 0 migration/0

6 root RT 0 0 0 0 S 0.0 0.0 0:01.92 0 watchdog/0

7 root RT 0 0 0 0 S 0.0 0.0 0:04.81 0 migration/1

8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 0 migration/1

9 root 20 0 0 0 0 S 0.0 0.0 0:03.97 0 ksoftirqd/1

10 root RT 0 0 0 0 S 0.0 0.0 0:01.63 0 watchdog/1

etc

 

Continue reading...