It’s Always DNS: Debugging a VCF Management Services Bootstrap Failure during 9.1.1 bringup

In an effort to adopt signed certificates in my lab, I decided to do a fresh bringup on a domain I own rather than some random one I made up for my lab, for example:
Old: wilson.lab
New: mydomain.com

For mydomain.com, to save having to create manual DNS entry for new services I bring up outside of VCF, I had a DNS wildcard entry pointing to my Container host. This is quite relevant because it was the root cause of an issue during the VCF MS bootstrap deployment. For context, I run an extensive OrbStack deployment at home for various services such as HomeAssistant, Grafana, Adguard etc.

Well, what was the issue?

During a new 9.1.1 deployment, I noticed Monitor VCF Management Services Deployment Task taking a long time. Looking in vCenter, the bootstrap VM was up and running, and whilst I was SSH’d on it to investigate, my connection dropped and I saw the VM was deleted and a new one deployed. Odd, I’ve used the same bringup configuration since 9.0 was released so the only change was the domain name. I’d also changed some IP addresses but I was sure all was good.

Annoyingly, the error message in VCF Installer (as well as SDDC Manager) precisely matched with the following Broadcom KB: https://knowledge.broadcom.com/external/article/450277 however after some digging it appeared to be nothing more than a coincidence.

2026/09/11 14:58:56 Waiting for VCF services runtime cluster nodes to become ready
>>> INFRA0002 - Waiting for VCF services runtime cluster nodes to become ready

I spent a good deal of time digging through the logs and saw that there was a 404 error when the bootstrap VM was trying to talk to vCenter /sdk endpoint, kubectl -n vmsp-platform logs deployment/vmsp-operator revealing:

"error":"failed to create vCenter client: failed to create vsphere client: POST \"/sdk\": 404 Not Found"

I did some pings, nslookups, and curls – all resolved fine and I could reach the Mgmt vCenter Server without issue, including from the Bootstrap VM.

If over two decades in IT has taught me something, the first question I ask when something that was working stops working is: what has changed? I knew the domain had changed, I knew IP addresses had changed. I checked, checked, and checked again and all was in order. The only remaining difference was the presence of the wildcard DNS entry of *. mydomain.com pointing to my Container host. I went through, created static host entries for all my services and then disabled the wildcard entry. As soon as I did this, everything started moving along.

I can only assume that the bootstrap VM creates an API call to the Management vCenter, and part of the payload is something like $vCenter.mydomain.com (ie, a regex or similar and not the actual FQDN) which then gets translated by the DNS server and points it to the host defined in the wildcard DNS entry. To confirm, I looked at my Traefik logs, which manages all my Container traffic ingress:

172.19.0.1 - - [11/Sep/2026:14:57:15 +0000] "POST /sdk HTTP/1.1" 404 19 "-" "-" 1738 "-" "-" 0ms

BIngo!

To sum up – if you are facing VCF 9.1.1 (or possibly earlier versions) bringup/VCF deployment errors particularly during Management Services Deployment, ensure the domain VCF is using does not have a DNS wildcard entry.

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.