You set the significance level (rather than a P-value). The default (as stated in the documentation) is 0.05 but if you want it to be something else such as 0.10, then you can use the following:
data = RandomVariate[NormalDistribution[], 10];
ks = KolmogorovSmirnovTest[data, Automatic, "ShortTestConclusion", SignificanceLevel ->0.10]
