Author: Chris

just a guy 'ya know... I lift weights, ride my bicycle, & talk virtualization.

vTPM & Native Key Provider for vSphere – Updated May 2025

~ Chris ~ Leave a comment

I wrote this up a while back and then updated it recently when a couple of people I work with needed information on Windows 11 Guests in vSphere.

Version/Changelog

  • v1.0 – Dec 2022
  • v1.1 – May 2025
    • Added Concepts section to help with vocabulary.
    • Added note about Horizon information.
    • Updated links for Broadcom URLs, Removed deprecated information.

vTPM overview:

  • vTPM information is stored in the .nvram file for the VM.  This file, and a few others, are encrypted with VM Encryption. Encrypting the entire VM (VMDKs) is not required. If you are protecting VMs with a vTPM be certain your backup methods also capture the .nvram file otherwise the recovered VM will not boot.
    • Note this means you are required to have a Key Provider to handle the Encryption keys.
  • vSphere 8 added a vTPM provision policy. This allows vTPM devices to be replaced during clone/deployment. In vSphere 7 the vTPM is cloned along with the VM resulting in identical secrets. To replace the vTPM in vSphere 7 the vTPM must be removed and re-added to replace the secrets. Changing the vTPM will have an impact on any guest functions that rely on the secrets.

Performance

  • There is no observed performance difference between the Native Key Provider (NKP) and an external provider. There is the chance that a remote KIMP would be slower due to latency/external calls. The NKP is sufficient to handle anything that is within the vCenter maximums.

Key Backups

  • vSphere Native Key Provider is backed up as part of the vCenter Server file-based backup. One manual backup needs to be performed on the vCenter or you will see a warning in the console every 24 hours.
  • You can also backup just the NKP data separately as a PKCS #12 file.
  • Keeping a separate copy of the NKP is advised just in case the vCenter cannot be restored.
  • It is recommended to name your NKP with a descriptive name to help identify which key provider is in use.

Concepts

  • Key types
    • Key Derivation Key (KDK) – the seed that allows the Native Key Provider to generate keys. This can be stored in the physical TPM 2.0 or encrypted ESXi configuration to allow for decryption when the NKP is offline, or inaccessible.
    • Key Encryption Key (KEK) – a key that is used to encrypt other keys
    • Data Encryption Key (DEK) – the key that is used to encrypt protected data. The DEK is protected with the KEK.
  • Shallow Rekey is when the Key Encrypting Key (KEK) is replaced. The DEK is decrypted and re-encrypted with a new KEK. This is generally very quick, consumes very little resources, and non-disruptive. Shallow rekey of a VM can be done with the VM powered on.
  • Deep Rekey is when the Data Encryption Key (DEK) or the key that encrypts the data itself is replaced. This involves decrypting and then re-encrypting all the protected data. This will use a fair amount of resources, and the length of time will be determined by the amount of data that needs to be processed. A deep rekey of a VM requires it to be powered off.

Host integration

  • If you have a TPM 2.0 chip on the ESX hosts the Key Derivation Key (KDK) from the Key Provider is stored on the hosts and sealed with the physical TPM chip.  This will allow use of encrypted datastores if the Key Provider is offline. (in 7.0 U2 and later). This is HIGHLY recommended to prevent circular dependencies.

ELM & Cross-vCenter vMotion

  • The Native Key Provider data is NOT shared when vCenters are in Enhanced Link Mode. If you want to use the same key provider for the linked vCenters it must be imported (see resources section)
  • Guest VMs that are migrated across will still need access to the original Key Provider to unlock the vTPM files at guest OS boot.
  • VMs with vTPM will still be configured with their original key provider after migration.
  • VMs can have a shallow rekey operation performed to change which key provider is used for encryption.

vSAN & Encryption

  • vSAN has two levels of encryption. Data-in-transit and Data-at-rest.
    • Data-in-transit encrypts the transmission of data between the hosts.
    • Data-at-rest encrypts the data when it is written to disk.
  • If you encrypt the vSAN datastore you do not need to use VM encryption. Likewise, if you encrypt the VM you do not need to encrypt the datastore.  Encrypting the VMs will likely have a negative impact any DeDupe and Compression ratios.
  • vSAN datastore encryption requires a disk format change. Like other disk format changes this is done in a rolling fashion through each disk group in the cluster.
    • To speed up this process you can allow reduced redundancy during the reformat. This is similar to maintenance mode with the ensure accessibility option.
  • With vSAN 8.0 ESA. Encryption can only be enabled when the cluster is created. Once it is enabled it cannot be disabled.
  • Data at rest is encrypted after all other processing, such as deduplication, is performed.
  • VMs storage policies on stretched clusters should be evaluated before encryption is enabled with the ‘Allow reduced redundancy’ option. If Secondary (site) failures to tolerate is 0 read operations will happen on the secondary object, this may increase latency. This is not a concern if SFTT is greater than 0 or the allow reduced redundancy option is not selected.

Swapping Key Providers

  • Changing Key providers is straightforward process and involves a ‘shallow’ re-key. This changes the Key Encryption Key and can be performed without guest downtime.
  • Swapping can happen between any supported key provider

Horizon with Windows 11 and vTPM

NOTE: This section was written and researched while Horizon was a VMware product. Now that Horizon is with Omnissa this information should be validated against their documentation. This was current as of Dec 2022.

  • Golden image must not have a vTPM. Horizon should add a vTPM in the provisioning settings.
    • Provisioning a Win11 machine without a vTPM can be done with WinPE or MS Deployment Toolkit
  • The golden image must have Microsoft Virtualization Based Security (VBS) enabled.
  • Instant Clone Mode B where no parent VMs exist is not supported with vTPM enabled machines. Planned for future Horizon release. Will require vSphere 7.0 U3f+

Resources:

vCenter, vSAN & ESXi End of Support – Oct 2, 2025 – FOR REAL this time.

~ Chris ~ Leave a comment

VMware traditionally offered a time period of Technical Guidance after a version went end of support. During Technical Guidance there would be no patches, no vulnerability scans, and no product enhancements. One could call in to support and get best-effort support without access to engineering or deep log reviews. If you did call in you would often get in touch with a support engineer who would help you with a Sev-1, with the intention that your next planned step would be to upgrade to a supported version. In my experience the depth of the interaction depended on the TSE that was assigned the ticket. It turned out positive if they had the knowledge and availability. As time passed beyond the end of support the odds of a positive experience quickly downhill.

Broadcom is not offering post general support tech guidance any longer (with a minor exception). On October 2nd 2025 version 7 of vCenter, vSAN, and ESXi will be self/community supported. See https://blogs.vmware.com/cloud-foundation/2025/03/31/reminder-vsphere-7-to-reach-end-of-service-october-2-2025/ (Exceptions exist for those who have current support purchased prior to May 6, 2024)

A line has to be drawn in the sand somewhere. This is the line. Good, Bad, or Ugly. Each team and organization should make the upgrade decision based on their risk tolerance. I’m encouraging everyone to be upgraded well before the Oct date because in the high-wire balancing act that is IT I want to be working with a net.

Aria Operations Management Packs End of Support

~ Chris ~ 2 Comments

Just came across this one today – https://knowledge.broadcom.com/external/article/373307

Broadcom is marking a large number of Aria Operations Management Packs as End of General Support as of Oct 1, 2024. There are a lot of good ones there that I know are in use today. It seems like most of them are going self-service and will require building of your own management pack with the Management Pack Builder. According to a blog I found on the vmware blogs site this seems straightforward. And there is a community page so you can ask your peers for help.

I know I’ll be talking about this and seeing what functionality is needed with the teams I interact with.

vSphere 8.0 Update 2 Upgrade guidance

~ Chris ~ Leave a comment

I spent some time today helping to refresh the great work of Ariel Sanchez Mora (arielsanchezmora.com) and Christina Griffus on vSphere 8.0 upgrade guidance. I added some specific info for vSphere/ESXi 8.0 Update 2.

Check out the new repo at https://github.com/vSphere8upgrade/7u3-to-8u2
Original source material at https://github.com/vSphere8upgrade/7u3-to-8u1

You can see the U2 content is still a Work in Progress. Follow along as it gets better.

Unassociated vSAN objects – this time with PowerCLI

~ Chris ~ Leave a comment

I came across https://kb.vmware.com/s/article/70726 – Procedures for identifying Unassociated vSAN objects and it reminded me of my old post https://vchrisblog.com/2018/07/13/unassociated-vsan-objects

The VMware KB is updated and has a better filter for finding the UUIDS
cat unassoc.txt | awk '{print $2}' | grep '^[0-9a-f]' >> /tmp/uuids.txt
I like the addition of the grep ^[0-9a-f] filter, it only grabs valid hex chars and will likely filter out things that I did not see in my environment. The rest of the code is largely the same manipulation I laid out in 2018.

There is also some code in the KB to perform the search for unassociated objects in PowerCLI. It does this by generating a vSANView and querying it for anything that is not associated with a VM object. Then taking the JSON object returned by the query. It seems you still need to do the removal in RVC. It’s worth reviewing the VMware KB to get some ideas.

The KB falls short of describing how to use the resulting unassociated object list (from rvc or PowerCLI) to perform any deletions. You can refer to my blog linked above for the delete method. As always double, and triple check your work before you delete anything. I cannot stress this enough, If you delete objects from vSAN they are gone forever. Because the lists we are generating are not associated with VMs there is a good chance they are invisible to your backup systems and you have no safety net here.

Play safe.

MS NETLOGON & vSphere Authentication

~ Chris ~ Leave a comment

In Nov 2022 MS published KB 5021130 and released an update to address CVE-2022-38023 (Privilege Escalation). The update to how NETLOGON sealing is likely to impact vSphere Authentication providers.

I have heard that Integrated Windows authentication will likely be impacted. The jury is still out if AD over LDAP(s) or AD FS are impacted.

Right now I recommend that you reach out your VMware account team, and your Microsoft account team.

I’ll update this blog entry as I hear more and the impact is understood.

Additional resources:
https://kb.vmware.com/s/article/90227
https://communities.vmware.com/t5/VMware-vCenter-Discussions/vCenter-Computer-Accounts-log-5840-after-November-2022-Windows/td-p/2939852

February 2023 Windows 2022 update prevents Secure Boot enabled VMs from booting on ESXi 7 and below

~ Chris ~ Leave a comment

UPDATE: VMware has issued a patch for ESX 7.0. This will address the issue and the MS patch can be safely deployed to VMs on updated hosts.

I’ve kept the information below to help you assess impact to your environment and provide mitigation information if upgrading hosts is not immediately possible.

Solution:

Install ESXi 7.0U3k on hosts – https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3k-release-notes.html

If you already have VMs that will not boot you can migrate them to a host running 7.0 U3k and then boot them up.

Conditions where this will impact you (all must be true):

  1. VM OS is Windows 2022
  2. Secure Boot is enabled
  3. ESXi host is running a version prior to 7.0U3k
    1. Note ESX v6.x and prior versions are no longer supported. v6.7 is now listed in the VMware KB as being impacted by this. I do not expect a patch for 6.x

Current workarounds (choose one):

  1. Disable SecureBoot on impacted VMs
  2. Upgrade ESXi host to v8 or v7.0U3k
  3. Prevent the installation of February 14 2023 Patch (MS KB5022842).
    1. This is not recommended long term. Only a temporary delay is prudent.

NOTE: Uninstalling KB50252842 will NOT resolve the boot issue. You must disable secure boot, or upgrade the host to return the VM to an operational state.

Links:

MS KB (KB5022842): https://support.microsoft.com/en-gb/topic/february-14-2023-kb5022842-os-build-20348-1547-be155955-29f7-47c4-855c-34bd43895940

VMware KB (90947): https://kb.vmware.com/s/article/90947

VMware 7.0 U3k release notes: https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3k-release-notes.html

Code Samples:

William Lam – Get & Set SecureBoot functions: https://github.com/vmware/PowerCLI-Example-Scripts/blob/master/Scripts/SecureBoot.ps1

As shown in William Lam’s code secure boot is exposed on a PowerCLI VM object within the ExtensionData property

$testVM = Get-VM <VMNAME>

$testVM.extensiondata.config.bootoptions.EfiSecureBootEnabled

This will return true if SecureBoot is enabled. Use this information as a starting point if you would like to scan your environment with powershell. You can further limit your search using the Guest.OSFullName property of the VM for 2022, or if you have hardware version <18 the OS value for Win2022 might be “…2016 or later…” refine your search terms as necessary

Changelog:

2/21/23

  • Added Solution section.
  • Updated header synopsis.
  • Updated conditions and workarounds to include 7.0U3k and 6.7 information.
  • Added 7.0 U3k release notes to links section

vTPM & Native Key Provider

~ Chris ~ Leave a comment

Had a question recently about vTPMs for VMS. This use case also included questions about vSAN disk encryption. Here are the bulletpoints from my documentation reading and discussions with others. I’m reasonably certain that all of this information is correct, if I got anything wrong put it in the comments and I’ll address it.

vTPM overview:

  • vTPM information is stored in the .nvram file for the VM.  This file, and a few others, are encrypted with VM Encryption. Encrypting the entire VM (VMDKs) is not required. If you are protecting VMs with a vTPM be certain your backup methods also capture the .nvram file otherwise the recovered VM will not boot.
  • vSphere 8 added a vTPM provision policy. This allows vTPM devices to be replaced during clone/deployment. In vSphere 7 the vTPM is cloned along with the VM resulting in identical secrets. To replace the vTPM in vSphere 7 the vTPM must be removed and re-added to replace the secrets. Changing the vTPM will have an impact on any guest functions that rely on the secrets.

Performance:

  • There is no observed performance difference between the Native Key Provider (NKP) and an external provider. There is the chance that a remote KIMP would be slower due to latency/external calls. The NKP is sufficient to handle anything that is within the vCenter maximums.

Key Backups:

  • vSphere Native Key Provider is backed up as part of the vCenter Server file-based backup. One manual backup needs to be performed on the vCenter or you will see a warning in the console every 24 hours.
  • You can also backup just the NKP data separately as a PKCS #12 file.
  • Keeping a separate copy of the NKP is advised just in case the vCenter cannot be restored.

Host integration:

  • If you have a TPM 2.0 chip on the ESX hosts the Key Derivation Key (KDK) from the Key Provider is stored on the hosts and sealed with the physical TPM chip.  This will allow use of encrypted datastores if the Key Provider is offline. (in 7.0 U2 and later). This is HIGHLY recommended to prevent circular dependencies.

ELM & Cross-vCenter vMotion.

  • The Native Key Provider data is NOT shared when vCenters are in Enhanced Link Mode. If you want to use the same key provider for the linked vCenters it must be imported (see resources section)
  • Guest VMs that are migrated across will still need access to the original Key Provider to unlock the vTPM files at guest OS boot.
  • VMs with vTPM will still be configured with their original key provider after migration.

vSAN:

  • vSAN has two levels of encryption. Data-in-transit and Data-at-rest.
    • Data-in-transit encrypts the transmission of data between the hosts.
    • Data-at-rest encrypts the data when it is written to disk.
  • If you encrypt the vSAN datastore you do not need to use VM encryption. Likewise if you encrypt the VM you do not need to encrypt the datastore. Encrypting the VMs will likely have a negative impact any DeDupe and Compression ratios.
  • vSAN datastore encryption requires a disk format change. Like other disk format changes this is done in a rolling fashion through each disk group in the cluster.
    • To speed up this process you can allow reduced redundancy during the reformat. This is similar to maintenance mode with the ensure accessibility option.
  • With vSAN 8.0 ESA. Encryption must be enabled when the cluster is created, and cannot be turned off.
  • Data at rest is encrypted after all other processing, such as deduplication, is performed.
  • VMs storage policies on stretched clusters should be evaluated before encryption is enabled with the ‘Allow reduced redundancy’ option. If Secondary (site) failures to tolerate is 0 read operations will happen on the secondary object, this may increase latency. This is not a concern if SFTT is greater than 0 or the allow reduced redundancy option is not selected

Swapping Key Providers

  • Changing Key providers is straightforward process and involves a ‘shallow’ re-key. This changes the Key Encryption Key and can be performed without guest downtime.
  • Swapping can happen between any supported key provider

Horizon with Windows 11 and vTPM:

  • Golden image must not have a vTPM. Horizon should add a vTPM in the provisioning settings.
    • Provisioning a Win11 machine without a vTPM can be done with WinPE or MS Deployment Toolkit
  • The golden image must have Microsoft Virtualization Based Security (VBS) enabled.
  • Instant Clone Mode B where no parent VMs exist is not supported with vTPM enabled machines. Planned for future Horizon release. Will require vSphere 7.0 U3f+

Resources:

Storage Sense and OneDrive in VDI and FSLOGIX

~ Chris ~ Leave a comment

I ran into a weird issue when I was migrating 100s of GB of data to OneDrive. I was getting an error that the disk was out of space but when I checked the local disk there was plenty of space. I eventually traced this to the OneDrive cache that was housed in my FSLOGIX. I needed to run storage sense and have it dehydrate the local files so that I would have space. I set a GPO on the OU where the computers reside to enable Storage sense, run it daily, and to dehydrate any file that was not touched in 1 day. I learned a few other things along the way.

My VDI desktop is a non-persistent instant clone that has my profile data redirected to an FSLOGIX container on a NAS. Our profiles have a 25GB quota. We redirect Documents, Desktop, AppData and some other folders. We specifically exclude Downloads, and browser cache/temp folders. OneDrive’s Cache by default lives in the user’s profile

During my testing I attempted to change the location of the OneDrive folder to a folder outside of the profile (remember it’s redirected with FSLOGIX, and this is a non-persistent desktop so everything not redirected is destroyed on logout). I used the DefaultRootDir setting in the OneDrive GPO to send this to c:\tmp. The initial login and configure went just fine. I logged out, and when I logged back in with a new session OneDrive was very angry. It choked on the missing OneDrive – [Organization] root folder. I created this and OneDrive then complained that there was nothing there. I modified the image so that the root folder was preset and OneDrive would not launch automatically and complain. On another computer I saw a large number of files being deleted on my OneDrive. I assume this was related to the missing files on the test computer.

Ultimately I decided even if I was able to get OneDrive to stop complaining, and to automatically launch for the user I couldn’t subject my users to the delay as OneDrive built all the stub-files on the local system. Not to mention the unaddressed deletes. This means I need to keep the OneDrive folder within the profile data that we are persisting for the user.

We have always set the FilesOnDemandEnabled in the OneDrive GPO to only download the necessary files to the local system. As you interact with files they are cached locally for you to use. This only handles the hydration of files, not the dehydration. I found that this is handled by Storage Sense, which should kick off when you are low on disk space. (No I have not found what the threshold for low is). This requires the storage service to be running on your computer. Because this is a VDI optimized windows configuration we have disabled unnecessary services like the storage service.

Enter the Storage Sense GPO to save me from myself. Through the GPO I enabled Storage sense, and set it to run daily, I also set the threshold to dehydrate files to 1 day. These are both the most aggressive that they can be set. This means that storage sense will run at some point every day (I did not investigate deeply to see when this happens, I know it is not at logon) and will dehydrate files that haven’t been accessed in the last 24 hours. I think this means that if my desktop session stays logged in the longest a file will remain cached is just under 48 hours.

I did not configure any of the other storage sense options like downloads, recycle bin or temp file cleanup. This is a non-persistent VDI desktop and we don’t capture these in the FSLOGIX profile so they are destroyed on logout. I also set storage sense to run daily so that it is likely to run at some point while a user is logged in. If a user logs in a 8a and out at 5 there is a greater chance that the cleanup will run if I have it execute daily. The 1 day dehydration was an aggressive setting for testing, I will likely change this to a more realistic setting for my every-day users.

Some interesting things to note:

  • Automatic dehydration will not impact files that a user has specified to ‘Always keep on this device’ they will only dehydrate when the user unchecks this. It should auto-dehydrate based on the storage sense policy or immediately if the user selects Free up space
  • The keep on this device setting is per device. In a VDI scenario is translates to keep with this profile. This does not impact the locality of data on physical endpoints, or other VDI desktops where the user gets a different profile. (On-prem vs Cloud if the profile is not shared between them)

Resources: