UPDATE - Turns out my discovery was only a tip of the iceberg… William Lam followed up on my post with an article on his blog, where he actually finds out the root cause of this behavior. Very interesting read, make sure to take a look!

I was playing around with nested VMware Tanzu deployments when an error hit me at the very last stage of the deployment:

HTTP communication could not be completed with status 404

…which was quite annoying as it caused my newly setup Tanzu cluster to crash and burn.

I was picking my hair about this for some time and was looking through all different Tanzu articles on the web looking for an answer. All I knew was that the error was related to the Content Library URL: https://wp-content.vmware.com/v2/latest/lib.json

Determined to find an answer, I started my troubleshooting on the ESXi hosts in the cluster I planned to deploy Workload Management in:

[root@esxi011:~] wget https://wp-content.vmware.com/v2/latest/lib.json
Connecting to wp-content.vmware.com (23.212.228.56:443)
wget: error getting response: Cannot assign requested address

[root@esxi011:~] ping wp-content.vmware.com
PING wp-content.vmware.com (23.212.228.56): 56 data bytes
64 bytes from 23.212.228.56: icmp_seq=0 ttl=57 time=14.547 ms
64 bytes from 23.212.228.56: icmp_seq=1 ttl=57 time=14.368 ms

--- wp-content.vmware.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 14.368/14.457/14.547 ms

Hmmm… interesting. So I can ping, but I can’t wget. Let’s try putting a test file on my server and publish it via HTTP:

[root@esxi011:~] wget ssh.guru/pub/test
Connecting to ssh.guru (54.37.17.173:80)
Connecting to ssh.guru (54.37.17.173:443)
wget: error getting response: Cannot assign requested address

Still nothing! Now I knew there must be something wrong with the host itself as I could download the file just fine on any other system:

~ ❯ wget ssh.guru/pub/test                                                                              46m 18s
URL transformed to HTTPS due to an HSTS policy
--2020-11-06 00:45:29--  https://ssh.guru/pub/test
Resolving ssh.guru (ssh.guru)... 54.37.17.173
Connecting to ssh.guru (ssh.guru)|54.37.17.173|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [application/octet-stream]
Saving to: ‘test.1’

test.1         [ <=>              ]        0  --.-KB/s    in 0s      

2020-11-06 00:45:29 (0.00 B/s) - ‘test.1’ saved [0/0]

I started googling around and, by pure luck, I stumbled across this: https://communities.vmware.com/thread/563764

…which gave me an idea: how about I disable ESXi firewall? It’s my homelab. What’s the worst that could happen, eh?

[root@esxi011:~] esxcli network firewall set --enabled false

Aaaaaaand then:

[root@esxi011:~] wget https://wp-content.vmware.com/v2/latest/lib.json
Connecting to wp-content.vmware.com (23.212.228.56:443)
lib.json             100% |****************************************************************|   332  0:00:00 ETA

WOOP WOOP! :D

This turned out to be enough to get the 404 error out of the way.

'Download remote files' tasks processing fine - no more 404s!

I am aware this solution is not ideal and only suitable for a homelab, but I’m pretty sure the problem lies somewhere in my setup (nested) and, to be totally honest, I can’t be arsed to troubleshoot it. It’s not like I’ll be ever deploying Tanzu to nested ESXi in production… at least I hope not.