Category: VMware

vTPM & Native Key Provider for vSphere – Updated May 2025

~ Chris ~ Leave a comment

I wrote this up a while back and then updated it recently when a couple of people I work with needed information on Windows 11 Guests in vSphere.

Version/Changelog

  • v1.0 – Dec 2022
  • v1.1 – May 2025
    • Added Concepts section to help with vocabulary.
    • Added note about Horizon information.
    • Updated links for Broadcom URLs, Removed deprecated information.

vTPM overview:

  • vTPM information is stored in the .nvram file for the VM.  This file, and a few others, are encrypted with VM Encryption. Encrypting the entire VM (VMDKs) is not required. If you are protecting VMs with a vTPM be certain your backup methods also capture the .nvram file otherwise the recovered VM will not boot.
    • Note this means you are required to have a Key Provider to handle the Encryption keys.
  • vSphere 8 added a vTPM provision policy. This allows vTPM devices to be replaced during clone/deployment. In vSphere 7 the vTPM is cloned along with the VM resulting in identical secrets. To replace the vTPM in vSphere 7 the vTPM must be removed and re-added to replace the secrets. Changing the vTPM will have an impact on any guest functions that rely on the secrets.

Performance

  • There is no observed performance difference between the Native Key Provider (NKP) and an external provider. There is the chance that a remote KIMP would be slower due to latency/external calls. The NKP is sufficient to handle anything that is within the vCenter maximums.

Key Backups

  • vSphere Native Key Provider is backed up as part of the vCenter Server file-based backup. One manual backup needs to be performed on the vCenter or you will see a warning in the console every 24 hours.
  • You can also backup just the NKP data separately as a PKCS #12 file.
  • Keeping a separate copy of the NKP is advised just in case the vCenter cannot be restored.
  • It is recommended to name your NKP with a descriptive name to help identify which key provider is in use.

Concepts

  • Key types
    • Key Derivation Key (KDK) – the seed that allows the Native Key Provider to generate keys. This can be stored in the physical TPM 2.0 or encrypted ESXi configuration to allow for decryption when the NKP is offline, or inaccessible.
    • Key Encryption Key (KEK) – a key that is used to encrypt other keys
    • Data Encryption Key (DEK) – the key that is used to encrypt protected data. The DEK is protected with the KEK.
  • Shallow Rekey is when the Key Encrypting Key (KEK) is replaced. The DEK is decrypted and re-encrypted with a new KEK. This is generally very quick, consumes very little resources, and non-disruptive. Shallow rekey of a VM can be done with the VM powered on.
  • Deep Rekey is when the Data Encryption Key (DEK) or the key that encrypts the data itself is replaced. This involves decrypting and then re-encrypting all the protected data. This will use a fair amount of resources, and the length of time will be determined by the amount of data that needs to be processed. A deep rekey of a VM requires it to be powered off.

Host integration

  • If you have a TPM 2.0 chip on the ESX hosts the Key Derivation Key (KDK) from the Key Provider is stored on the hosts and sealed with the physical TPM chip.  This will allow use of encrypted datastores if the Key Provider is offline. (in 7.0 U2 and later). This is HIGHLY recommended to prevent circular dependencies.

ELM & Cross-vCenter vMotion

  • The Native Key Provider data is NOT shared when vCenters are in Enhanced Link Mode. If you want to use the same key provider for the linked vCenters it must be imported (see resources section)
  • Guest VMs that are migrated across will still need access to the original Key Provider to unlock the vTPM files at guest OS boot.
  • VMs with vTPM will still be configured with their original key provider after migration.
  • VMs can have a shallow rekey operation performed to change which key provider is used for encryption.

vSAN & Encryption

  • vSAN has two levels of encryption. Data-in-transit and Data-at-rest.
    • Data-in-transit encrypts the transmission of data between the hosts.
    • Data-at-rest encrypts the data when it is written to disk.
  • If you encrypt the vSAN datastore you do not need to use VM encryption. Likewise, if you encrypt the VM you do not need to encrypt the datastore.  Encrypting the VMs will likely have a negative impact any DeDupe and Compression ratios.
  • vSAN datastore encryption requires a disk format change. Like other disk format changes this is done in a rolling fashion through each disk group in the cluster.
    • To speed up this process you can allow reduced redundancy during the reformat. This is similar to maintenance mode with the ensure accessibility option.
  • With vSAN 8.0 ESA. Encryption can only be enabled when the cluster is created. Once it is enabled it cannot be disabled.
  • Data at rest is encrypted after all other processing, such as deduplication, is performed.
  • VMs storage policies on stretched clusters should be evaluated before encryption is enabled with the ‘Allow reduced redundancy’ option. If Secondary (site) failures to tolerate is 0 read operations will happen on the secondary object, this may increase latency. This is not a concern if SFTT is greater than 0 or the allow reduced redundancy option is not selected.

Swapping Key Providers

  • Changing Key providers is straightforward process and involves a ‘shallow’ re-key. This changes the Key Encryption Key and can be performed without guest downtime.
  • Swapping can happen between any supported key provider

Horizon with Windows 11 and vTPM

NOTE: This section was written and researched while Horizon was a VMware product. Now that Horizon is with Omnissa this information should be validated against their documentation. This was current as of Dec 2022.

  • Golden image must not have a vTPM. Horizon should add a vTPM in the provisioning settings.
    • Provisioning a Win11 machine without a vTPM can be done with WinPE or MS Deployment Toolkit
  • The golden image must have Microsoft Virtualization Based Security (VBS) enabled.
  • Instant Clone Mode B where no parent VMs exist is not supported with vTPM enabled machines. Planned for future Horizon release. Will require vSphere 7.0 U3f+

Resources:

vCenter, vSAN & ESXi End of Support – Oct 2, 2025 – FOR REAL this time.

~ Chris ~ Leave a comment

VMware traditionally offered a time period of Technical Guidance after a version went end of support. During Technical Guidance there would be no patches, no vulnerability scans, and no product enhancements. One could call in to support and get best-effort support without access to engineering or deep log reviews. If you did call in you would often get in touch with a support engineer who would help you with a Sev-1, with the intention that your next planned step would be to upgrade to a supported version. In my experience the depth of the interaction depended on the TSE that was assigned the ticket. It turned out positive if they had the knowledge and availability. As time passed beyond the end of support the odds of a positive experience quickly downhill.

Broadcom is not offering post general support tech guidance any longer (with a minor exception). On October 2nd 2025 version 7 of vCenter, vSAN, and ESXi will be self/community supported. See https://blogs.vmware.com/cloud-foundation/2025/03/31/reminder-vsphere-7-to-reach-end-of-service-october-2-2025/ (Exceptions exist for those who have current support purchased prior to May 6, 2024)

A line has to be drawn in the sand somewhere. This is the line. Good, Bad, or Ugly. Each team and organization should make the upgrade decision based on their risk tolerance. I’m encouraging everyone to be upgraded well before the Oct date because in the high-wire balancing act that is IT I want to be working with a net.

vSphere 8.0 Update 2 Upgrade guidance

~ Chris ~ Leave a comment

I spent some time today helping to refresh the great work of Ariel Sanchez Mora (arielsanchezmora.com) and Christina Griffus on vSphere 8.0 upgrade guidance. I added some specific info for vSphere/ESXi 8.0 Update 2.

Check out the new repo at https://github.com/vSphere8upgrade/7u3-to-8u2
Original source material at https://github.com/vSphere8upgrade/7u3-to-8u1

You can see the U2 content is still a Work in Progress. Follow along as it gets better.

Unassociated vSAN objects – this time with PowerCLI

~ Chris ~ Leave a comment

I came across https://kb.vmware.com/s/article/70726 – Procedures for identifying Unassociated vSAN objects and it reminded me of my old post https://vchrisblog.com/2018/07/13/unassociated-vsan-objects

The VMware KB is updated and has a better filter for finding the UUIDS
cat unassoc.txt | awk '{print $2}' | grep '^[0-9a-f]' >> /tmp/uuids.txt
I like the addition of the grep ^[0-9a-f] filter, it only grabs valid hex chars and will likely filter out things that I did not see in my environment. The rest of the code is largely the same manipulation I laid out in 2018.

There is also some code in the KB to perform the search for unassociated objects in PowerCLI. It does this by generating a vSANView and querying it for anything that is not associated with a VM object. Then taking the JSON object returned by the query. It seems you still need to do the removal in RVC. It’s worth reviewing the VMware KB to get some ideas.

The KB falls short of describing how to use the resulting unassociated object list (from rvc or PowerCLI) to perform any deletions. You can refer to my blog linked above for the delete method. As always double, and triple check your work before you delete anything. I cannot stress this enough, If you delete objects from vSAN they are gone forever. Because the lists we are generating are not associated with VMs there is a good chance they are invisible to your backup systems and you have no safety net here.

Play safe.

MS NETLOGON & vSphere Authentication

~ Chris ~ Leave a comment

In Nov 2022 MS published KB 5021130 and released an update to address CVE-2022-38023 (Privilege Escalation). The update to how NETLOGON sealing is likely to impact vSphere Authentication providers.

I have heard that Integrated Windows authentication will likely be impacted. The jury is still out if AD over LDAP(s) or AD FS are impacted.

Right now I recommend that you reach out your VMware account team, and your Microsoft account team.

I’ll update this blog entry as I hear more and the impact is understood.

Additional resources:
https://kb.vmware.com/s/article/90227
https://communities.vmware.com/t5/VMware-vCenter-Discussions/vCenter-Computer-Accounts-log-5840-after-November-2022-Windows/td-p/2939852

February 2023 Windows 2022 update prevents Secure Boot enabled VMs from booting on ESXi 7 and below

~ Chris ~ Leave a comment

UPDATE: VMware has issued a patch for ESX 7.0. This will address the issue and the MS patch can be safely deployed to VMs on updated hosts.

I’ve kept the information below to help you assess impact to your environment and provide mitigation information if upgrading hosts is not immediately possible.

Solution:

Install ESXi 7.0U3k on hosts – https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3k-release-notes.html

If you already have VMs that will not boot you can migrate them to a host running 7.0 U3k and then boot them up.

Conditions where this will impact you (all must be true):

  1. VM OS is Windows 2022
  2. Secure Boot is enabled
  3. ESXi host is running a version prior to 7.0U3k
    1. Note ESX v6.x and prior versions are no longer supported. v6.7 is now listed in the VMware KB as being impacted by this. I do not expect a patch for 6.x

Current workarounds (choose one):

  1. Disable SecureBoot on impacted VMs
  2. Upgrade ESXi host to v8 or v7.0U3k
  3. Prevent the installation of February 14 2023 Patch (MS KB5022842).
    1. This is not recommended long term. Only a temporary delay is prudent.

NOTE: Uninstalling KB50252842 will NOT resolve the boot issue. You must disable secure boot, or upgrade the host to return the VM to an operational state.

Links:

MS KB (KB5022842): https://support.microsoft.com/en-gb/topic/february-14-2023-kb5022842-os-build-20348-1547-be155955-29f7-47c4-855c-34bd43895940

VMware KB (90947): https://kb.vmware.com/s/article/90947

VMware 7.0 U3k release notes: https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3k-release-notes.html

Code Samples:

William Lam – Get & Set SecureBoot functions: https://github.com/vmware/PowerCLI-Example-Scripts/blob/master/Scripts/SecureBoot.ps1

As shown in William Lam’s code secure boot is exposed on a PowerCLI VM object within the ExtensionData property

$testVM = Get-VM <VMNAME>

$testVM.extensiondata.config.bootoptions.EfiSecureBootEnabled

This will return true if SecureBoot is enabled. Use this information as a starting point if you would like to scan your environment with powershell. You can further limit your search using the Guest.OSFullName property of the VM for 2022, or if you have hardware version <18 the OS value for Win2022 might be “…2016 or later…” refine your search terms as necessary

Changelog:

2/21/23

  • Added Solution section.
  • Updated header synopsis.
  • Updated conditions and workarounds to include 7.0U3k and 6.7 information.
  • Added 7.0 U3k release notes to links section

vTPM & Native Key Provider

~ Chris ~ Leave a comment

Had a question recently about vTPMs for VMS. This use case also included questions about vSAN disk encryption. Here are the bulletpoints from my documentation reading and discussions with others. I’m reasonably certain that all of this information is correct, if I got anything wrong put it in the comments and I’ll address it.

vTPM overview:

  • vTPM information is stored in the .nvram file for the VM.  This file, and a few others, are encrypted with VM Encryption. Encrypting the entire VM (VMDKs) is not required. If you are protecting VMs with a vTPM be certain your backup methods also capture the .nvram file otherwise the recovered VM will not boot.
  • vSphere 8 added a vTPM provision policy. This allows vTPM devices to be replaced during clone/deployment. In vSphere 7 the vTPM is cloned along with the VM resulting in identical secrets. To replace the vTPM in vSphere 7 the vTPM must be removed and re-added to replace the secrets. Changing the vTPM will have an impact on any guest functions that rely on the secrets.

Performance:

  • There is no observed performance difference between the Native Key Provider (NKP) and an external provider. There is the chance that a remote KIMP would be slower due to latency/external calls. The NKP is sufficient to handle anything that is within the vCenter maximums.

Key Backups:

  • vSphere Native Key Provider is backed up as part of the vCenter Server file-based backup. One manual backup needs to be performed on the vCenter or you will see a warning in the console every 24 hours.
  • You can also backup just the NKP data separately as a PKCS #12 file.
  • Keeping a separate copy of the NKP is advised just in case the vCenter cannot be restored.

Host integration:

  • If you have a TPM 2.0 chip on the ESX hosts the Key Derivation Key (KDK) from the Key Provider is stored on the hosts and sealed with the physical TPM chip.  This will allow use of encrypted datastores if the Key Provider is offline. (in 7.0 U2 and later). This is HIGHLY recommended to prevent circular dependencies.

ELM & Cross-vCenter vMotion.

  • The Native Key Provider data is NOT shared when vCenters are in Enhanced Link Mode. If you want to use the same key provider for the linked vCenters it must be imported (see resources section)
  • Guest VMs that are migrated across will still need access to the original Key Provider to unlock the vTPM files at guest OS boot.
  • VMs with vTPM will still be configured with their original key provider after migration.

vSAN:

  • vSAN has two levels of encryption. Data-in-transit and Data-at-rest.
    • Data-in-transit encrypts the transmission of data between the hosts.
    • Data-at-rest encrypts the data when it is written to disk.
  • If you encrypt the vSAN datastore you do not need to use VM encryption. Likewise if you encrypt the VM you do not need to encrypt the datastore. Encrypting the VMs will likely have a negative impact any DeDupe and Compression ratios.
  • vSAN datastore encryption requires a disk format change. Like other disk format changes this is done in a rolling fashion through each disk group in the cluster.
    • To speed up this process you can allow reduced redundancy during the reformat. This is similar to maintenance mode with the ensure accessibility option.
  • With vSAN 8.0 ESA. Encryption must be enabled when the cluster is created, and cannot be turned off.
  • Data at rest is encrypted after all other processing, such as deduplication, is performed.
  • VMs storage policies on stretched clusters should be evaluated before encryption is enabled with the ‘Allow reduced redundancy’ option. If Secondary (site) failures to tolerate is 0 read operations will happen on the secondary object, this may increase latency. This is not a concern if SFTT is greater than 0 or the allow reduced redundancy option is not selected

Swapping Key Providers

  • Changing Key providers is straightforward process and involves a ‘shallow’ re-key. This changes the Key Encryption Key and can be performed without guest downtime.
  • Swapping can happen between any supported key provider

Horizon with Windows 11 and vTPM:

  • Golden image must not have a vTPM. Horizon should add a vTPM in the provisioning settings.
    • Provisioning a Win11 machine without a vTPM can be done with WinPE or MS Deployment Toolkit
  • The golden image must have Microsoft Virtualization Based Security (VBS) enabled.
  • Instant Clone Mode B where no parent VMs exist is not supported with vTPM enabled machines. Planned for future Horizon release. Will require vSphere 7.0 U3f+

Resources: