Message Boards

GROUPS:

Staff Picks Data Science Education Mathematics Graphics and Visualization Wolfram Language Statistics and Probability Computer-Based Maths

Seth Chandler

[Notebook] A precise explanation of the P - Value

Seth Chandler, University of Houston

Posted 7 months ago

6777 Views

5 Replies

25 Total Likes

Follow this post

We are now ready for a crucial concept : the p - value. You must memorize its definition and understand it. The p-value is a notoriously slippery concept. People misstate its meaning all the time. And they do so in ways that can have catastrophic consequences for research and for the humans it affects. Do not be one of those people.

The first thing to remember is that there are no p-values in the air. The p-value is attached to a statistic and to a test. The test may be legitimate only under certain circumstances and the p-value is only as good as the statistic with which it is associated. Let’s say that again. The p-value is attached to a statistic and to a test. The test may be legitimate only under certain circumstances and the p-value is only as good as the statistic with which it is associated. One more time. The p-value is attached to a statistic and to a test. The test may be legitimate only under certain circumstances and the p-value is only as good as the statistic with which it is associated.

To see what a p - value is, consider the following example. Let’s generate 100 draws from a particular logistic distribution. I’ll call it the null distribution. And let’s do the experiment 10,000 times.

SeedRandom[11102019]

In[]:=

nullDistribution=LogisticDistribution[3,2];

In[]:=

Dimensions[rv=RandomVariate[nullDistribution,{10000,100}]]

{10000,100}

Out[]=

Now, let's compute the same statistic on each of the 10, 000 samples of 100. My statistic will be the median. This gives us a list of 10,000 values.

(medians=Map[Median,rv])//Short

In[]:=

{3.19964,3.38066,9996,3.54773,3.06237}

Out[]//Short=

This list can be used to generate its own statistical distribution. This is NOT the same as the underlying null distribution. It is a derived distribution based on computing a statistic on numerous samples of the underlying null distribution. Read this paragraph until you completely understand it.

medianDistribution=EmpiricalDistribution[medians]

In[]:=

DataDistribution

Type: Empirical

Data points: 10000



Out[]=

This list can also be used to generate various Histograms that we can then plot. Here I show the “PDF”, “CDF” and “Survival Function” (“SF”) of the median distribution.

Column@KeyValueMap[Curry[Labeled]/*Framed][AssociationMap[hspecHistogram[medians,50,hspec,ImageSize300,BaseStyle14],{"PDF","CDF","SF"}]]

In[]:=

PDF

CDF

Out[]=

Now, suppose we have some data. Here it is:

sample=

387

100

373

100

207

,3,

367

100

527

100

299

100

381

100

163

211

100

139

179

261

,1,

407

100

353

100

273

102

187

139

303

100

513

100

253

100

339

100

291

100

537

100

243

100

219

189

113

461

100

357

100

371

100

191

141

291

100

477

100

443

100

433

100

257

100

249

100

263

109

459

100

481

100

137

383

100

529

100

169

100

517

100

253

211

100

147

371

100

237

279

100

237

100

359

100

167

132

257

100

403

100

417

100

291

100

453

100

163

323

100

119

;

In[]:=

Now let' s compute the same statistic on this data as we did on the data from the null distribution.

medianSample=Median[sample]

In[]:=

Out[]=

We now ask the big question: how likely is it we would see a statistic resembling the one we do (in a certain way) if the sample data came from the null distribution. I will call this statement “if the sample data came from the null distribution” the null hypothesis.

To figure this out, let's plot out median of the sample on our various histograms. Look at these graphics carefully. They are the most important picture in this essay.

CDF[medianDistribution,medianSample]

In[]:=

0.9166

Out[]=

sampleLineGraphic=GraphicsDashed,Text"statistic on our sample",medianSample,

,{-1,0},{1,1},InfiniteLine[{{medianSample,0},{medianSample,1}}];

Composition[

]

@KeyValueMap[Curry[Labeled]/*Framed]AssociationMaphspecShowHistogrammedians,50,hspec,

,sampleLineGraphic,{"PDF","CDF","SF"}

PDF

CDF

Out[]=

One - sided P - values

What you can see is that our sample median is on the high side if the data came from the null distribution. But is it implausibly high? I’m now going to annoy you by first showing you the wrong way to proceed because it seems so obviously right that you and others may be enticed by the idea. And in the process I hope to motivate why it is that P-Values are computed in the non-intuitive way that they are.

You might think you could look at the PDF of the median distribution evaluated at our sample median. But this is a problem: there was never an occasion on which the median of our sample was exactly equal to the median of any the list of 100 values in our 10,000 draw experiment.

MemberQ[medians,medianSample]

In[]:=

False

Out[]=

But the problem is deeper that having a discrete distribution for the statistic in question. Maybe we could use some fancy math to derive an expression for the PDF (or other identifying function) of a continuous distribution of the statistic in question. Or, failing that, suppose we ran zillions of experiments so that we ended up for all practical purposes with a continuous distribution for our medians. The code below simulates that process and creates something called a SmoothKernelDistribution.

mediansSKD=SmoothKernelDistribution[medians]

In[]:=

DataDistribution

Type: SmoothKernel

Data points: 10000



Out[]=

Here' s a plot of the result.

Show[Plot[PDF[mediansSKD,x],{x,1,5}],sampleLineGraphic]

In[]:=

Out[]=

We could now evaluate the PDF of this continuous distribution for the median statistic at the median of our sample.

PDF[mediansSKD,medianSample]

In[]:=

0.374981

Out[]=

But what on earth does that number mean? There isn’t any obvious meaning. To see this, suppose our distribution of our underlying statistic was narrowed and our sample statistic had the same CDF value relative to that distribution as our existing sample statistic has to our original distribution.

mediansSKD2=SmoothKernelDistribution[Rescale[medians,{2,5},{3.0,3.4}]]

In[]:=

DataDistribution

Type: SmoothKernel

Data points: 10000



Out[]=

newSampleMedian=Quantile[mediansSKD2,CDF[mediansSKD,medianSample]]

In[]:=

3.20667

Out[]=

newSampleLineGraphic=GraphicsDashed,Text"new statistic on our sample",newSampleMedian,

,{-1,0},{1,1},InfiniteLine[{{newSampleMedian,0},{newSampleMedian,1}}];

In[]:=

Here' s the new PDF with the new statistic marked.

Show[Plot[PDF[mediansSKD2,x],{x,1,5}],newSampleLineGraphic]

In[]:=

Out[]=

Our sample statistic is no more or less probable than it was before. But if we now evaluate the PDF value, here’s what we get:

PDF[mediansSKD2,newSampleMedian]

In[]:=

2.81236

Out[]=

Nothing fundamental has changed. And yet, we get a completely different number. So, we can’t readily use the PDF in a simple way to figure out anything about how likely it is that we would see the median in our sample if the null hypothesis were true. (There is a very fancy way, but we’re not going there today). Instead, we need to use a different property of the distribution of our median: the CDF (or SurvivalFunction).

Suppose that instead of evaluating the PDF of our median distribution at the value of our sample median, we instead computed the survival function of our median distribution at the value of our sample median. In short, we might compute the fraction of occasions on which the sample median would be greater than our sample median if the data came from the null distribution. The operator we want is > or ≥ and not =. We see that happens 8.34% of the time.

Counts@Thread[Greater[medians,medianSample]]

In[]:=

False9166,True834

Out[]=

And this value is robust against smushing or spreading of the underlying median distribution. I illustrate this below by doing the same computation but using the smushed median distribution.

Counts@Thread[Greater[Rescale[medians,{2,5},{3.0,3.4}],newSampleMedian]]

In[]:=

False9166,True834

Out[]=

That number of 834/10000 has a special meaning : it' s a one - sided P - Value. It' s a P - Value because we are computing a probability of an event given a distribution. And it' s one sided because we are looking only at the probability of values on one side of a particular value.

And so, now you know what a p - value is. Or at least you know that it has something to do with cumulative distributions of a statistic computed if the null hypothesis were true relative to the same statistic computed on the sample data.

A slight digression into statistical significance

I take a slight digression here into the subject of statistical significance. Thus far I have asked about the probability of getting a value as large as something if the statistic in question was derived from the null distribution. But I could ask the inverse question. How large would my sample statistic have to be before the probability of obtaining a value greater than that was less than some “significance level,” say 0.05. That’s the same as asking, in essence, what is the 0.95 quantile of the median distribution.

Quantile[medianDistribution,0.95]

In[]:=

3.6596

Out[]=

So, before I could say the sample was different in a “statistically significant” way from that expected by computing the median on the null distribution, I would need to see a sample median of at least 3.6596. And that is true (and only true) if the significance level was set at 0.05.

I could also set my significance level higher. I could set it to 0.01.

Quantile[medianDistribution,0.99]

In[]:=

3.95736

Out[]=

If so, I would need to see a sample median of at least 3.95736 before I could say that the difference between my sample median and the sample median drawn from the null distribution would be “statistically significant” at the 0.01 level.

Back to one - sided P - Values

Notice that we computed the probability that the computing the fraction of occasions on which the sample median would be higher than our sample median if the data came from the null distribution. That makes sense because we got a high value of the median for a sample. What if our median was 2.3, however? In that instance, it would likely make more sense to compute the probability that we would see a value as low as 2.3 if the data was generated by the null distribution. We can compute this too.

Counts@Thread[Greater[medians,2.3]]

In[]:=

True9612,False388

Out[]=

Here we see there is only a slim chance that we would see a value this low. This is again a one-sided P-Value because we are looking only at the probability of values on one side of a particular value.

Notice that in neither instance is 8.34 % or 3.88 % the probability that the data came from null distribution. We have NO IDEA what that number is. Nor can we say we are 91.66% certain or 96.12% certain that the data did not come from the null distribution. We also have NO IDEA what that number is. When you see people interpreting a P-Value this way, they are wrong, wrong, wrong. They are committing the fallacy of thinking in PDF terms. Also, remember, the P-Value is attached to a statistic. All we can say is that if the data came from the null distribution, we would expect to see a median as large (or small) as the one we have x% of the time.

Two - sided P - Values

So far I’ve looked at one-sided P-Value statistics. But there are also two-sided statistics. To see this, consider another statistic we could compute : the difference the median of each set of data and the median of the entire data.

medianOfWholeThing=Median[Flatten[rv]]

In[]:=

3.0052

Out[]=

(medianDifferences=Map[Median[#]-medianOfWholeThing&,rv])//Short

In[]:=

{0.194436,0.375453,9997,0.0571668}

Out[]//Short=

Let' s do the same thing of creating a distribution and constructing various Histograms with labels :

medianDifferencesSKD=SmoothKernelDistribution[medianDifferences]

In[]:=

DataDistribution

Type: SmoothKernel

Data points: 10000



Out[]=

Now let' s compute the same median difference statistic on this data as we did on the data from the null distribution.

medianDifferenceSample=Median[sample]-medianOfWholeThing

In[]:=

0.544797

Out[]=

We can generate decorated Histograms for the new statistic just as I did for the median statistic. Here I show the “PDF”, “CDF” and “Survival Function” (“SF”) of the median difference distribution. Note, by the way, that the distribution appears to be roughly symmetric about zero. I designed my statistic that way deliberately. Generally speaking, we don’t use two-sided P-Values where the distribution of the statistic is not symmetric about some modal value, and usually that modal value is 0.

sampleLineGraphicDifferences=Graphics[{Dashed,Text["statistic on our sample",{medianDifferenceSample,1/2},{-1,0},{1,1}],InfiniteLine[{{medianDifferenceSample,0},{medianDifferenceSample,1}}]}];

In[]:=

Column@KeyValueMap[Curry[Labeled]/*Framed][AssociationMap[hspecShow[Histogram[medianDifferences,50,hspec,ImageSize300,BaseStyle14],sampleLineGraphicDifferences],{"PDF","CDF","SF"}]]

In[]:=

PDF

CDF

Out[]=

I could now do the same thing as before and compute one - sided P - Values. But now there is a natural alternative. What if I wanted to compute the probability that one would see a statistic as “extreme” as 0.544797 if the median difference statistic had been generated by the null distribution. To do this, I need a notion of extreme. Here’s one possibility. I look at the difference between my statistic and 0. Equivalently, I want to look at the absolute value of the difference between my statistic and 0. So, what I want is the probability of getting a median difference statistic as large as 0.544797 or as small as -0.544797. Let’s compute this.

Counts[Thread[Greater[Abs[medianDifferences],Abs[medianDifferenceSample]]]]

In[]:=

False8347,True1653

Out[]=

So, we could say that if the null hypothesis were true, we would see a median statistic “as extreme” as 0.544797 about 16.53% of the time. And that number 16.53%, is a two-sided value.

Here' s another way to visualize the matter . The red zone represents sample statistics that are farther from the center of the distribution at zero than the negative of our sample median. The blue zone represents sample statistics that are farther from the center of the distribution at zero than our sample median. And the sum of the red and blue areas is equal to the two-sided P-Value.

Plot[PDF[medianDifferencesSKD,x],{x,-2,2},ColorFunction({x,y}Which[x<-0.54,Red,x>0.54,Blue,True,Green]),ColorFunctionScalingFalse,FillingAxis,FillingStyleAutomatic]

In[]:=

Out[]=

Here' s yet another way to visualize matters. Here is the CDF of the median difference distribution. The area marked in red are for those values less than the negative of the sample median difference. The area marked in blue are for those values more than the sample median difference. The orange arrow marks the difference between the value of the CDF at the negative of the sample median difference and 0. The purple arrow marks the differences between 1 and the the value of the CDF at the sample median difference . The sum of the lengths of the orange and purple arrows represents a two-sided P-Value.

Show[Plot[CDF[medianDifferencesSKD,x],{x,-2,2},ColorFunction({x,y}Which[y<CDF[medianDifferencesSKD,-0.54],Red,y>CDF[medianDifferencesSKD,0.54],Blue,True,Green]),ColorFunctionScalingFalse,ImageSize500],Graphics[GraphicsComplex[{{medianDifferenceSample,0},{medianDifferenceSample,CDF[medianDifferencesSKD,medianDifferenceSample]},{0,CDF[medianDifferencesSKD,medianDifferenceSample]},{0,1},{-medianDifferenceSample,0},{-medianDifferenceSample,CDF[medianDifferencesSKD,-medianDifferenceSample]},{0,CDF[medianDifferencesSKD,-medianDifferenceSample]},{0,0}},{Dashed,Line[{1,2}],Line[{2,3}],Arrowheads[{-.02,.02}],Purple,Arrow[{3,4}],Line[{5,6}],Line[{6,7}],Arrowheads[{-.02,.02}],Orange,Arrow[{7,8}]}]]]

In[]:=

Out[]=

Some complications

If you' ve been following this discussion, you may realize that my definition of extremeness only works sensibly where the PDF of the statistic’s distribution is symmetric about zero. And, indeed, when two-sided P-Values are used, most of the time a statistic is computed whose PDF meets that requirement. One should realize, however, that the two-sided P-Value can be adapted to more complex settings.

Symmetric case

To see an adaptation of P - Values to symmetric distributions about something other than zero, consider a statistic that computes (for some obscure reason) one plus the difference between the median of the sample and the median of the entire distribution.

(medianDifferencesPlusOne=Map[1+Median[#]-medianOfWholeThing&,rv])//Short

In[]:=

{1.19444,9998,1.05717}

Out[]//Short=

medianDifferencePlusOneSample=1+Median[sample]-medianOfWholeThing

In[]:=

1.5448

Out[]=

Here' s a visualization of the PDF, CDF, and SurvivalFunction of the distribution of the new statistic with the value of the statistic on our sample marked with a line.

sampleLineGraphicDifferencesPlusOne=Graphics[{Dashed,Text["statistic on our sample",{medianDifferencePlusOneSample,1/2},{-1,0},{1,1}],InfiniteLine[{{medianDifferencePlusOneSample,0},{medianDifferencePlusOneSample,1}}]}];

In[]:=

Column@KeyValueMap[Curry[Labeled]/*Framed][AssociationMap[hspecShow[Histogram[medianDifferencesPlusOne,50,hspec,ImageSize300,BaseStyle14],sampleLineGraphicDifferencesPlusOne],{"PDF","CDF","SF"}]]

In[]:=

PDF

CDF

Out[]=

Clearly, the distribution is no longer symmetric about zero. But we could proceed exactly as before if we if we transformed our distribution just a tiny bit. Consider a transformed distribution that subtracts a certain number (say 1) from each output of the distribution of our new statistic. We could then proceed exactly as before but work on the transformed distribution. Here’s the code and the picture of the transformed distribution and transformed statistic.

medianDifferencePlusOneSampleTransformed=medianDifferencePlusOneSample-1

In[]:=

0.544797

Out[]=

sampleLineGraphicDifferencesPlusOneTransformed=Graphics[{Dashed,Text["transformed statistic\non our sample",{medianDifferencePlusOneSampleTransformed,1/2},{-1,0},{1,1}],InfiniteLine[{{medianDifferencePlusOneSampleTransformed,0},{medianDifferencePlusOneSampleTransformed,1}}]}];

In[]:=

Column@KeyValueMap[Curry[Labeled]/*Framed][AssociationMap[hspecShow[Histogram[medianDifferencesPlusOne-1,50,hspec,ImageSize300,BaseStyle14],sampleLineGraphicDifferencesPlusOneTransformed],{"PDF","CDF","SF"}]]

In[]:=

PDF

CDF

Out[]=

And so the analysis is now exactly the same.

Asymmetric case

There' s a more complicated approach that can be taken if the distribution of the statistic is not symmetric. The idea is to again to come up with a definition as extreme. One could compute the CDF of the statistic on the sample relative to the distribution of the statistic. Call one minus that value the “right hand side.” And one could compute the SurvivalFunction of the statistic on the sample relative to the distribution of the statistic. Call that the “left hand side.” The sum of the left hand side and the right hand side is the two-sided P-Value. Perhaps fortunately, this extension is seldom used because two-sided P-Values are generally not used where the distribution of the underlying statistic is not symmetric. Put otherwise, statisticians often try to develop statistics that are symmetric (preferably around zero) precisely so two-sided P-Values can be used.

POSTED BY: Seth Chandler

Answer

5 Replies

Sort By:

Moderation Team

Moderation Team, WOLFRAM

Posted 7 months ago

- Congratulations! This post is now featured in our Staff Pick column as distinguished by a badge on your profile of a Featured Contributor! Thank you, keep it coming, and consider contributing your work to the The Notebook Archive!

POSTED BY: Moderation Team

Answer

Seth Chandler

Seth Chandler, University of Houston

Posted 7 months ago

Thanks for the recognition. Just so people understand the context for this notebook, I developed it as part of a class I teach to students at the University of Houston Law Center, Analytic Methods for Lawyers, so that they have a good conceptual understanding of statistics. I hope it is helpful for others. Also, ignore the last paragraph of the notebook; it should not be there. (maybe the moderators could delete it).

POSTED BY: Seth Chandler

Answer

Diego Zviovich

Posted 7 months ago

Hi @Seth Chandler , excellent post. Do you have something similar for conditional probability? It is a topic that always raises its ugly head in court and may have sent many innocent people to jail.

POSTED BY: Diego Zviovich

Answer

Murray Eisenberg

Murray Eisenberg, University of Massachusetts Amherst

Posted 4 months ago

The first sentence is "We are now ready for a crucial concept : the p - value." This seems to say that there are materials that precede this document. Where can one find those materials?

POSTED BY: Murray Eisenberg

Answer

Scott Cinsavich

Posted 3 months ago

Hi, this is the very first time I have encountered this site and I have sort of an off topic question How do I cite your notebook for Mendeley or Reference Manager?

POSTED BY: Scott Cinsavich

Answer

Reply to this discussion

Reply Preview

Attachments

Remove Add a file to this post

Follow this discussion

or Discard

Group Abstract

Feedback