There is no right answer about what p-value should count. The 0.05 threshold represents a somewhat arbitrary balancing of the cost of false positives (finding that the data under consideration was not generated by the null hypothesis even though in fact it was) against the cost of false negatives (finding insufficient evidence to reject the null hypothesis even though that hypothesis is in fact false).

I just wrote a draft notebook on P-values that might possibly be of use to you in conceptualizing this issue and the difference between one-sided and two-sided P-values in particular. You can find it here:

https://community.wolfram.com/groups/-/m/t/1824481

Hi, Check in Library (go to Help and write for instance Significance level) how to run tests and conclude if the hypothesis is rejected or not. All that follows is from that:

BlockRandom[SeedRandom[1];
  data = RandomVariate[StudentTDistribution[5], 100]];

DistributionFitTest[data, 
 NormalDistribution[\[Mu], \[Sigma]], "ShortTestConclusion", 
 SignificanceLevel -> 0.1]

Out[2]= "Reject"

In[3]:= DistributionFitTest[data, 
 NormalDistribution[\[Mu], \[Sigma]], "ShortTestConclusion", 
 SignificanceLevel -> 0.05]

Out[3]= "Do not reject"

By default, 0.05 is used:

In[4]:= DistributionFitTest[data, NormalDistribution[\[Mu], \[Sigma]], 
  "TestConclusion", SignificanceLevel -> Automatic] // TraditionalForm

Out[4]//TraditionalForm= The null hypothesis that the data is distributed according to the NormalDistribution[\[Mu],\[Sigma]] is not rejected at the 5. percent level based on the Cramér-von Mises test.

Hi Seth, Great post! Will the contents of this notebook be part of your forthcoming book?

Thanks for the answer. I think that would be a great idea. By the way, would it be possible to have access to this notebook? Unfortunately, the attached version is not displaying properly.

Dear Seth, due to outstanding quality of your comment we moved it to Staff Pick column as a stand-alone contribution. Comments were slightly edited accordingly. You can find it here:

https://community.wolfram.com/groups/-/m/t/1824481