What a curious fact, - I have not known about this. Great post, @Marco Thiel, thanks for the writing and insight! BTW our readers who'll get inspired by your post to build a web crawler, should explore NestGraph function, it might come handy in some cases (it takes some time run):
NestGraph[Import[#, "Hyperlinks"] &, "https://www.bing.com", 2, VertexLabels -> Placed["Name", Tooltip]]
