UPDATE 16/11/2020 - Turns out my discovery was only a tip of the iceberg... William Lam followed up on my post with an article on his blog, where he actually finds out the root cause of this behavior. Very interesting read, make sure to take a look!
I was playing around with nested VMware Tanzu deployments when an error hit me at the very last stage of the deployment:
HTTP communication could not be completed with status 404
...which was quite annoying as it caused my newly setup Tanzu cluster to crash and burn.
I was picking my hair about this for some time and was looking through all different Tanzu articles on the web looking for an answer. All I knew was that the error was related to the Content Library URL: https://wp-content.vmware.com/v2/latest/lib.json
Determined to find an answer, I started my troubleshooting on the ESXi hosts in the cluster I planned to deploy Workload Management in:
[root@esxi011:~] wget https://wp-content.vmware.com/v2/latest/lib.json Connecting to wp-content.vmware.com (18.104.22.168:443) wget: error getting response: Cannot assign requested address [root@esxi011:~] ping wp-content.vmware.com PING wp-content.vmware.com (22.214.171.124): 56 data bytes 64 bytes from 126.96.36.199: icmp_seq=0 ttl=57 time=14.547 ms 64 bytes from 188.8.131.52: icmp_seq=1 ttl=57 time=14.368 ms --- wp-content.vmware.com ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 14.368/14.457/14.547 ms
Hmmm... interesting. So I can ping, but I can't wget. Let's try putting a test file on my server and publish it via HTTP:
[root@esxi011:~] wget ssh.guru/pub/test Connecting to ssh.guru (184.108.40.206:80) Connecting to ssh.guru (220.127.116.11:443) wget: error getting response: Cannot assign requested address
Still nothing! Now I knew there must be something wrong with the host itself as I could download the file just fine on any other system:
~ ❯ wget ssh.guru/pub/test 46m 18s URL transformed to HTTPS due to an HSTS policy --2020-11-06 00:45:29-- https://ssh.guru/pub/test Resolving ssh.guru (ssh.guru)... 18.104.22.168 Connecting to ssh.guru (ssh.guru)|22.214.171.124|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 0 [application/octet-stream] Saving to: ‘test.1’ test.1 [ <=> ] 0 --.-KB/s in 0s 2020-11-06 00:45:29 (0.00 B/s) - ‘test.1’ saved [0/0]
I started googling around and, by pure luck, I stumbled across this: https://communities.vmware.com/thread/563764
...which gave me an idea: how about I disable ESXi firewall? It's my homelab. What's the worst that could happen, eh?
[root@esxi011:~] esxcli network firewall set --enabled false
[root@esxi011:~] wget https://wp-content.vmware.com/v2/latest/lib.json Connecting to wp-content.vmware.com (126.96.36.199:443) lib.json 100% |****************************************************************| 332 0:00:00 ETA
WOOP WOOP! :D
This turned out to be enough to get the 404 error out of the way.
I am aware this solution is not ideal and only suitable for a homelab, but I'm pretty sure the problem lies somewhere in my setup (nested) and, to be totally honest, I can't be arsed to troubleshoot it. It's not like I'll be ever deploying Tanzu to nested ESXi in production... at least I hope not.