S
seccentral
Hello
This is my first post
*hi*
Now to the matter at hand:
I have a few servers that run centos 6 64bit and have 64GB ram each. I use those boxes as hypervisors. I have about a few hundred vps's virtualized with KVM.
To my astonishment i found at rather random times that some of my VPS's are getting their pids killed for apparently no reason untill i checked the memory usage.
Now, on each hypervisor i have about 40 up to 50 GB of ram(on average) allocated and the rest are free (as provided by free -m and top). The swap space however is a different issue (not knowing that i needed it anyway i had it set to 1GB). 1GB was fine up to the point where i deployed the kvm vps's and let them run for about a day or two.
Some nodes seem to fill up swap. Some nodes never do. all machines are the same hardware wise.
So i set vm.swappiness to zero on all nodes in hope to solve this issue but it doesn't do it.
I was left with the option of increasing the swap space by making a swapfile of 5GB somewhere and swapon it.
This is however a 'hack' that i'd rather avoid.
I've posted this question on kvm, qemu forums and mailing lists with no resolution (not even a reply).
I'm baffled by the fact that swap is never used on some nodes that run the same number of vps's that have the same configuration (i have some standard plans 1gb ram 10gb hdd etc running ubuntu 12.04 lts inside all vps nodes)
so if for instance
node 1 - 20 vps instances - zero swap usage
node 2 - 20 vps instances - swap fills about 20mb/day (average number, actual size varies randomly from my perspective)
top -M with Op enter says this on the swap using node:
Code:
top - 14:33:46 up 18 days, 5:11, 1 user, load average: 0.21, 0.19, 0.13
Tasks: 324 total, 1 running, 323 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.9%sy, 0.0%ni, 98.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 62.886G total, 39.360G used, 23.525G free, 8749.754M buffers
Swap: 5117.977M total, 1115.953M used, 4002.023M free, 10.496G cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
6833 qemu 20 0 1441m 761m 4968 S 0.0 1.2 39:37.41 55m qemu-kvm
5173 qemu 20 0 1439m 742m 4968 S 0.3 1.2 34:32.42 51m qemu-kvm
11653 qemu 20 0 1437m 729m 4968 S 0.3 1.1 34:24.61 50m qemu-kvm
13449 qemu 20 0 1445m 738m 5008 S 0.3 1.1 36:27.34 49m qemu-kvm
7783 qemu 20 0 1451m 753m 4996 S 0.3 1.2 33:49.16 49m qemu-kvm
4873 qemu 20 0 1451m 745m 4968 S 0.7 1.2 31:48.03 48m qemu-kvm
5928 qemu 20 0 1448m 755m 4968 S 0.3 1.2 34:39.14 47m qemu-kvm
11294 qemu 20 0 1434m 730m 5004 S 0.3 1.1 33:11.42 47m qemu-kvm
10016 qemu 20 0 1450m 757m 4968 S 0.0 1.2 35:15.59 47m qemu-kvm
14265 qemu 20 0 1441m 745m 4968 S 0.3 1.2 38:34.28 46m qemu-kvm
9065 qemu 20 0 1450m 748m 4968 S 0.3 1.2 31:58.57 44m qemu-kvm
12901 qemu 20 0 1448m 756m 4968 S 0.0 1.2 36:51.08 44m qemu-kvm
8028 qemu 20 0 1446m 738m 4968 S 0.3 1.1 32:18.04 43m qemu-kvm
etcetera
and this on a node that doesnt use swap:
Code:
top - 14:32:00 up 20 days, 19:17, 1 user, load average: 0.03, 0.07, 0.08
Tasks: 320 total, 1 running, 319 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.3%sy, 0.0%ni, 99.2%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 62.886G total, 56.751G used, 6281.445M free, 14.480G buffers
Swap: 5021.977M total, 0.000k used, 5021.977M free, 20.504G cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
1 root 20 0 19232 1520 1232 S 0.0 0.0 0:03.10 0 init
2 root 20 0 0 0 0 S 0.0 0.0 0:04.15 0 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:03.81 0 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:05.79 0 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 0 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:01.92 0 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:04.81 0 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 0 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 0:03.97 0 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:01.63 0 watchdog/1
etc
Continue reading...
This is my first post
*hi*Now to the matter at hand:
I have a few servers that run centos 6 64bit and have 64GB ram each. I use those boxes as hypervisors. I have about a few hundred vps's virtualized with KVM.
To my astonishment i found at rather random times that some of my VPS's are getting their pids killed for apparently no reason untill i checked the memory usage.
Now, on each hypervisor i have about 40 up to 50 GB of ram(on average) allocated and the rest are free (as provided by free -m and top). The swap space however is a different issue (not knowing that i needed it anyway i had it set to 1GB). 1GB was fine up to the point where i deployed the kvm vps's and let them run for about a day or two.
Some nodes seem to fill up swap. Some nodes never do. all machines are the same hardware wise.
So i set vm.swappiness to zero on all nodes in hope to solve this issue but it doesn't do it.
I was left with the option of increasing the swap space by making a swapfile of 5GB somewhere and swapon it.
This is however a 'hack' that i'd rather avoid.
I've posted this question on kvm, qemu forums and mailing lists with no resolution (not even a reply).
I'm baffled by the fact that swap is never used on some nodes that run the same number of vps's that have the same configuration (i have some standard plans 1gb ram 10gb hdd etc running ubuntu 12.04 lts inside all vps nodes)
so if for instance
node 1 - 20 vps instances - zero swap usage
node 2 - 20 vps instances - swap fills about 20mb/day (average number, actual size varies randomly from my perspective)
top -M with Op enter says this on the swap using node:
Code:
top - 14:33:46 up 18 days, 5:11, 1 user, load average: 0.21, 0.19, 0.13
Tasks: 324 total, 1 running, 323 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.9%sy, 0.0%ni, 98.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 62.886G total, 39.360G used, 23.525G free, 8749.754M buffers
Swap: 5117.977M total, 1115.953M used, 4002.023M free, 10.496G cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
6833 qemu 20 0 1441m 761m 4968 S 0.0 1.2 39:37.41 55m qemu-kvm
5173 qemu 20 0 1439m 742m 4968 S 0.3 1.2 34:32.42 51m qemu-kvm
11653 qemu 20 0 1437m 729m 4968 S 0.3 1.1 34:24.61 50m qemu-kvm
13449 qemu 20 0 1445m 738m 5008 S 0.3 1.1 36:27.34 49m qemu-kvm
7783 qemu 20 0 1451m 753m 4996 S 0.3 1.2 33:49.16 49m qemu-kvm
4873 qemu 20 0 1451m 745m 4968 S 0.7 1.2 31:48.03 48m qemu-kvm
5928 qemu 20 0 1448m 755m 4968 S 0.3 1.2 34:39.14 47m qemu-kvm
11294 qemu 20 0 1434m 730m 5004 S 0.3 1.1 33:11.42 47m qemu-kvm
10016 qemu 20 0 1450m 757m 4968 S 0.0 1.2 35:15.59 47m qemu-kvm
14265 qemu 20 0 1441m 745m 4968 S 0.3 1.2 38:34.28 46m qemu-kvm
9065 qemu 20 0 1450m 748m 4968 S 0.3 1.2 31:58.57 44m qemu-kvm
12901 qemu 20 0 1448m 756m 4968 S 0.0 1.2 36:51.08 44m qemu-kvm
8028 qemu 20 0 1446m 738m 4968 S 0.3 1.1 32:18.04 43m qemu-kvm
etcetera
and this on a node that doesnt use swap:
Code:
top - 14:32:00 up 20 days, 19:17, 1 user, load average: 0.03, 0.07, 0.08
Tasks: 320 total, 1 running, 319 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.3%sy, 0.0%ni, 99.2%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 62.886G total, 56.751G used, 6281.445M free, 14.480G buffers
Swap: 5021.977M total, 0.000k used, 5021.977M free, 20.504G cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
1 root 20 0 19232 1520 1232 S 0.0 0.0 0:03.10 0 init
2 root 20 0 0 0 0 S 0.0 0.0 0:04.15 0 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:03.81 0 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:05.79 0 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 0 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:01.92 0 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:04.81 0 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 0 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 0:03.97 0 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:01.63 0 watchdog/1
etc
Continue reading...