• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

VMWare Performance Issues

X

xtox

Limp Gawd
Joined
Jul 8, 2004
Messages
242
Hopefully someone may be able to help us a little bit more with an issue we are having. We have 3x esxi servers running 4.0, pretty beefy with 2x 6core cpu & 32GB RAM each. They each run a terminal server (windows 2008 r2) and a couple of other machines.

We are having big big problems with terminal server performance, the cpu keeps spiking to 100% which is causing users to lag badly & freeze up. We have given the machines 4vcpus to try and take advantage of more cores and even reserved shares. Lots of other little tweaks here and there but nothing fixes it.

What's funny is that when logging into v-centre we can see the hosts are taking it pretty easy with cpu usage being between 9%-30% at the times when the hosts spike. Looking at individual vm performance it does not take full advantage of the 8800MHz it has assigned, only around half of it. Other VMs on the boxes are fine, very low cpu usage so it's just the TS boxes. Each TS box is on a seperate esxi server so it's weird that it spikes on every one of them.

Could this point to an I/O problem as the only common factor is the iscsi storage but why would this cause the spikes we are seeing?

I've tried to keep this short so let me know if you need more info! Help much appreciated guys!
 
Have you used any windows server tools to find out what process is spiking the CPU?
 
It can be an array of things so I can't pinpoint it. However my big concern is why vmware is not allocating it more MHz? Is there a way to force it to have the full 4 cores and not scale up when it needs to? It doesn't seem to be doing a good job at catching up.
 
What's the CPU load on the host box when this VM does that? How many vCPUs are at 100%? Just one or all four?
 
All good questions so far, but I doubt giving it more processors is going to help. I've only got 7 hosts and about 100 VMs, but I've never had to assign anything more than 2 processors and that was on a heavy usage SQL 2008 box.

I'd be curious to know how many users are using this TS box and what they're doing with it.

Also, what amount of RAM do you have allocated and what is the swap file on the TS box set to? I've seen CPU spikes caused by under-allocated RAM and swap file issues.
 
Let me rephrase, did you attempt to pinpoint it using Diganostic tools? Just because it's a VM doesn't mean that Windows Diagnostics don't apply.
 
Last edited:
Verify that your guest OS is not swapping heavily. Also check the CPU Ready time under the Performance Tab for the VM.

Check to see if the VM has any memory limit placed on it under Settings - Resources.
 
we had a similar issue, which turned out to be related to two things on the terminal servers themselves, unrelated to VMWare.
- users had a lot of printers assigend to them through a policy (which they really didn't need), which caused the print spooler to spike for a long time when a user logged in: solved by rethinking the printer assignments, reducing the amount of printers per user.
- a bad printer driver, causing the print spooler to spike and sometimes crash during normal operation.
 
I'm with Vader, normal rules still apply. Have you looked at running something like process explorer to track what activity is going on, or have you looked at your SAN and esx to check things like queue lengths and IO?

I'm not an expert on vSphere but I was always lead to believe that you don't necessarily want to throw cores at VM's because you can end up making things slower because the vSphere scheduler has to be able to allocate the resources to the VM which has an overhead, or something along those lines..
 
Why I'm asking my CPU questions. ;)
Click to expand...

Don't you have some VNX Blog updates you need to do? I'm patiently waiting...lol. BTW..did you have to require quicktime in those vids on our VMworld Sessions?..:D Apple guys..all the same.....:p
 
Vader said:
Don't you have some VNX Blog updates you need to do? I'm patiently waiting...lol. BTW..did you have to require quicktime in those vids on our VMworld Sessions?..:D Apple guys..all the same.....:p
Click to expand...

Yeah yeah...I have a lot of things to do...
 
Vader said:
Don't you have some VNX Blog updates you need to do? I'm patiently waiting...lol. BTW..did you have to require quicktime in those vids on our VMworld Sessions?..:D Apple guys..all the same.....:p
Click to expand...

ROFL. Yeah NetJunkie, go do your community service!

Its interesting that the OP reserved resources and is still seeing the issues like this. Assuming it boils down to CPU issues. If you reserve enough of your total allocated CPU resources you should be experiencing very little, if any CPU ready. Scheduling still comes into play by some amount, but you are not competing for the resources. I've seen similar results though with TS. Some of it I could not explain. Maybe I found a barrier in my ability... but everyone was stumped. I got some talented people on the line with VMware support just to verify I wasn't insane, and verified that the results did not line up with the numbers.

The first thing I would do is drop to 1 vCPU on one of them, and slowly ramp up user load. It would be interesting to compare the results you see between the two configurations.
 
Hey guys,

Thank you so much for all your posts. Things are looking better at the moment since I moved a few servers onto another iscsi target. Could IO have been the issue all along?

Previously all cores within the VM would hit 100% but now it looks as if only one or two will peak when the others remain low usage. This should mean the TS users aren't lagging as much.

I still have each server set to 4vcpu, do you think this should be downgraded? I am not entirely clued up on how this works but v-centre shows the MHz will be reduced to 2000 compared to 8000 if I reduce to 1vcpu. Would this not hinder performance or does it do something fancy and still use al the cores?

I don't have any performance stats to hand as the issue seems to be partly resolved but let me know what stats you want captured if you still need any.

Thanks again!
 
xtox said:
Could IO have been the issue all along?
Click to expand...
Yes. I think its plausable that if storage was starving an application you could see other symptoms, sure.

I still have each server set to 4vcpu, do you think this should be downgraded? I am not entirely clued up on how this works but v-centre shows the MHz will be reduced to 2000 compared to 8000 if I reduce to 1vcpu. Would this not hinder performance or does it do something fancy and still use al the cores?
Click to expand...

It means your physical host has 2ghz physical cores, so 4 vCPU x 2000mhz each = 8000mhz total assigned. That isn't a linear 8000, no one thread is going to push all 4 cores, but if you have a multithreaded application it could use them in a way -- even then those various threads can each only go as fast as one core. Another possibility is each user runs a particular app that is CPU intensive it could spread those processes out among the vCPUs.

TS is weird to me in how it behaves when virtual, but one rule always remains true whatever it is your virtualizing when you're not 'sure' what resources you'll need. Start small and grow vs. the other way around. You want to make it the 'right size' neither to big or too small.
 
Last edited:
Start small and grow vs. the other way around. You want to make it the 'right size' neither to big or too small.
Click to expand...

Excellent advice!
 
You must log in or register to reply here.
Back
Top