Had a problem this evening where an ESXi host would choke on the 6.5 U2 update. vSphere reported “The host returns esxupdate error code:15. The package manager transaction is not successful. Check the Update Manager log files and esxupdate log files for more details.” Which lead me to https://kb.vmware.com/kb/2030665
esxupdate.log had the Python errors listed in that KB. Quick fix right? No, that would be too easy. After following the easy instructions in that KB where you just delete the /locker/packages/6.5.0 directory. I kicked off VUM and was presented with the same error. Tried the long fix. Recreated the 6.5.0 structure from a good host. Again same error.
Some digging lead me to https://blog.definebroken.com/2017/07/28/patching-vsphere-esxi-to-6-5u1-failing-with-error-15-cause-ran-out-of-vfat-space-due-to-vsantrace-and-vsanobserver-files/
He had a problem with vsantrace files. I checked and that wasn’t my problem. But I did cause me to watch the vmkernel.log to see if there were any clues there.
Sure enough I was getting a bunch of out of space errors:
2018-08-11T04:28:18.037Z cpu23:124089)WARNING: VFAT: 313: VFAT volume mpx.vmhba32:C0:T0:L0:8 (UUID 597f645d-2327f2da-1218-246e963e79d0) is full. (585696 sectors, 0 free sectors)
Did some du work and found that /store/locker/core was taking up ~80MB
[root@HOSTA1:/vmfs/volumes/597f645d-2327f2da-1218-246e963e79d0] du -h . 256.0K ./packages/var/db/locker/vibs 256.0K ./packages/var/db/locker/profiles 768.0K ./packages/var/db/locker 1.0M ./packages/var/db 1.3M ./packages/var 1.5M ./packages 80.2M ./var/core 80.4M ./var 256.0K ./epd-new 82.4M .
Sure enough there was a zdump file in var/core
A simple rm and then everything worked.
I didn’t catch this in my initial troubleshooting because the VFAT partition reported as 28% in use.
Since I was patching ESX itself the first step is to delete and recreate the vmtools and floppy images vCenter uses for client OS installation. These are stored in the /store/packages/<ESXversion> directory. This freed a bit of space, but all the space on the partition was consumed when it copied the updated versions and packages for installation.
Sometimes it is just a different simple fix.
vChris 