Category: Virtualization

February 2023 Windows 2022 update prevents Secure Boot enabled VMs from booting on ESXi 7 and below

~ Chris ~ Leave a comment

UPDATE: VMware has issued a patch for ESX 7.0. This will address the issue and the MS patch can be safely deployed to VMs on updated hosts.

I’ve kept the information below to help you assess impact to your environment and provide mitigation information if upgrading hosts is not immediately possible.

Solution:

Install ESXi 7.0U3k on hosts – https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3k-release-notes.html

If you already have VMs that will not boot you can migrate them to a host running 7.0 U3k and then boot them up.

Conditions where this will impact you (all must be true):

  1. VM OS is Windows 2022
  2. Secure Boot is enabled
  3. ESXi host is running a version prior to 7.0U3k
    1. Note ESX v6.x and prior versions are no longer supported. v6.7 is now listed in the VMware KB as being impacted by this. I do not expect a patch for 6.x

Current workarounds (choose one):

  1. Disable SecureBoot on impacted VMs
  2. Upgrade ESXi host to v8 or v7.0U3k
  3. Prevent the installation of February 14 2023 Patch (MS KB5022842).
    1. This is not recommended long term. Only a temporary delay is prudent.

NOTE: Uninstalling KB50252842 will NOT resolve the boot issue. You must disable secure boot, or upgrade the host to return the VM to an operational state.

Links:

MS KB (KB5022842): https://support.microsoft.com/en-gb/topic/february-14-2023-kb5022842-os-build-20348-1547-be155955-29f7-47c4-855c-34bd43895940

VMware KB (90947): https://kb.vmware.com/s/article/90947

VMware 7.0 U3k release notes: https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3k-release-notes.html

Code Samples:

William Lam – Get & Set SecureBoot functions: https://github.com/vmware/PowerCLI-Example-Scripts/blob/master/Scripts/SecureBoot.ps1

As shown in William Lam’s code secure boot is exposed on a PowerCLI VM object within the ExtensionData property

$testVM = Get-VM <VMNAME>

$testVM.extensiondata.config.bootoptions.EfiSecureBootEnabled

This will return true if SecureBoot is enabled. Use this information as a starting point if you would like to scan your environment with powershell. You can further limit your search using the Guest.OSFullName property of the VM for 2022, or if you have hardware version <18 the OS value for Win2022 might be “…2016 or later…” refine your search terms as necessary

Changelog:

2/21/23

  • Added Solution section.
  • Updated header synopsis.
  • Updated conditions and workarounds to include 7.0U3k and 6.7 information.
  • Added 7.0 U3k release notes to links section

vTPM & Native Key Provider

~ Chris ~ Leave a comment

Had a question recently about vTPMs for VMS. This use case also included questions about vSAN disk encryption. Here are the bulletpoints from my documentation reading and discussions with others. I’m reasonably certain that all of this information is correct, if I got anything wrong put it in the comments and I’ll address it.

vTPM overview:

  • vTPM information is stored in the .nvram file for the VM.  This file, and a few others, are encrypted with VM Encryption. Encrypting the entire VM (VMDKs) is not required. If you are protecting VMs with a vTPM be certain your backup methods also capture the .nvram file otherwise the recovered VM will not boot.
  • vSphere 8 added a vTPM provision policy. This allows vTPM devices to be replaced during clone/deployment. In vSphere 7 the vTPM is cloned along with the VM resulting in identical secrets. To replace the vTPM in vSphere 7 the vTPM must be removed and re-added to replace the secrets. Changing the vTPM will have an impact on any guest functions that rely on the secrets.

Performance:

  • There is no observed performance difference between the Native Key Provider (NKP) and an external provider. There is the chance that a remote KIMP would be slower due to latency/external calls. The NKP is sufficient to handle anything that is within the vCenter maximums.

Key Backups:

  • vSphere Native Key Provider is backed up as part of the vCenter Server file-based backup. One manual backup needs to be performed on the vCenter or you will see a warning in the console every 24 hours.
  • You can also backup just the NKP data separately as a PKCS #12 file.
  • Keeping a separate copy of the NKP is advised just in case the vCenter cannot be restored.

Host integration:

  • If you have a TPM 2.0 chip on the ESX hosts the Key Derivation Key (KDK) from the Key Provider is stored on the hosts and sealed with the physical TPM chip.  This will allow use of encrypted datastores if the Key Provider is offline. (in 7.0 U2 and later). This is HIGHLY recommended to prevent circular dependencies.

ELM & Cross-vCenter vMotion.

  • The Native Key Provider data is NOT shared when vCenters are in Enhanced Link Mode. If you want to use the same key provider for the linked vCenters it must be imported (see resources section)
  • Guest VMs that are migrated across will still need access to the original Key Provider to unlock the vTPM files at guest OS boot.
  • VMs with vTPM will still be configured with their original key provider after migration.

vSAN:

  • vSAN has two levels of encryption. Data-in-transit and Data-at-rest.
    • Data-in-transit encrypts the transmission of data between the hosts.
    • Data-at-rest encrypts the data when it is written to disk.
  • If you encrypt the vSAN datastore you do not need to use VM encryption. Likewise if you encrypt the VM you do not need to encrypt the datastore. Encrypting the VMs will likely have a negative impact any DeDupe and Compression ratios.
  • vSAN datastore encryption requires a disk format change. Like other disk format changes this is done in a rolling fashion through each disk group in the cluster.
    • To speed up this process you can allow reduced redundancy during the reformat. This is similar to maintenance mode with the ensure accessibility option.
  • With vSAN 8.0 ESA. Encryption must be enabled when the cluster is created, and cannot be turned off.
  • Data at rest is encrypted after all other processing, such as deduplication, is performed.
  • VMs storage policies on stretched clusters should be evaluated before encryption is enabled with the ‘Allow reduced redundancy’ option. If Secondary (site) failures to tolerate is 0 read operations will happen on the secondary object, this may increase latency. This is not a concern if SFTT is greater than 0 or the allow reduced redundancy option is not selected

Swapping Key Providers

  • Changing Key providers is straightforward process and involves a ‘shallow’ re-key. This changes the Key Encryption Key and can be performed without guest downtime.
  • Swapping can happen between any supported key provider

Horizon with Windows 11 and vTPM:

  • Golden image must not have a vTPM. Horizon should add a vTPM in the provisioning settings.
    • Provisioning a Win11 machine without a vTPM can be done with WinPE or MS Deployment Toolkit
  • The golden image must have Microsoft Virtualization Based Security (VBS) enabled.
  • Instant Clone Mode B where no parent VMs exist is not supported with vTPM enabled machines. Planned for future Horizon release. Will require vSphere 7.0 U3f+

Resources: