Few days ago, I was doing regular upgrades within one of the vSphere environments I’m responsible for. During these upgrades, I’ve noticed there are some Windows 2008 R2 servers still on vHardware version 10. Obviously, I decided to go ahead and upgrade them too. Little did I know how’s that’s going to end up…

The upgrade itself went pretty straight forward. The VM was powered off correctly from within the guest OS and the upgrade apparently was supposed to run on the next boot up. I tasked vCenter with powering on the VM and moved on.

After two or three hours, I was preparing to finish up and write a report for the changes I committed when I’ve noticed one of the W2008R2 VMs was still off.

I must have missed it – I thought and launched the power on operation again… only to see it fail with the following error:

The destination host is not compatible with the hardware version to which the virtual machine is scheduled to be upgraded. In order to proceed with the operation, virtual machine’s scheduled compatibility upgrade should be disabled.

Hmmm… the scheduled upgrade? I thought the VM was already upgraded… I opened up the VM page and double checked the current version of the vHardware.

vmhardware13

Ok, that was getting annoying. As you can see, the VM hardware version was already 13, so there was no way there was yet another “scheduled upgrade” on this VM. All of the 6 hosts in the environment should be able to handle this VM. Something was wrong.

I decided to open up the VMX file of that VM to check it for errors. I’ve logged into the ESXi hosting it and open up the VMX with vi editor:

root@esxi-4 # vi ./win2008.vmx

Surprisingly, I’ve found out that this VM is in fact waiting for an upgrade (your guess is as good as mine) as the following records were present in the VMX file:

(...)
virtualHW.scheduledUpgrade.when = "always"</strong>```
  
virtualHW.version="13"  
<strong>virtualHW.scheduledUpgrade.state = "done"</strong>  
guestOS="win2008r2"  
<strong>tools.upgrade.policy = "upgradeAtPowerCycle"<br></br></strong>(...)

What this meant was that the VM was still scheduled to upgrade on the next boot up, even thought it was already on the highest version possible. Because of it, none of the hosts was willing run it (as the hosts “figured out” the VM must be right and there must be a newer VM version they are not aware of yet).

I’ve decided to remove those records and see if that fixes the problem. Firstly, I took a backup of the VMX file as it is just in case:

root@esxi-4 # cp ./win2008.vmx win2008.vmx.bak

Having that done, I open up the VMX yet again and deleted all three of the records mentioned above:

root@esxi-4 # vi ./win2008.vmx  
(...)virtualHW.scheduledUpgrade.when = "always"  
virtualHW.version="13"  
virtualHW.scheduledUpgrade.state = "done"  
guestOS="win2008r2"  
tools.upgrade.policy = "upgradeAtPowerCycle"
(...)

Just a reminded for all of you reading unaware of vi commands: to save the file type in :w and then press enter; to exit the editor type in :q and then press enter. On regular Li/Unix version of vi editor, you should be able to save&quit by just running :wq command, but it is unfortunately unsupported on ESXi version.

After saving the file and exiting the ESXi shell, I tried to power on the VM again… and it worked!

I hope this blog will help other people who might stumble across such problem in the future, because I thought for a second (before I found the faulty records) I might have to restore that VM and waste much more time on it than I finally did.