That's because of this I think:
EuclideanDistance[1, 2]
HammingDistance[1, 2]
the first one gives a number, while the second doesn't... So in order to fix it you have to put data in a certain format:
ClusteringTree[{1, 2, 5}, DistanceFunction -> EuclideanDistance]
ClusteringTree[{{1}, {2}, {5}}, DistanceFunction -> HammingDistance]