[WSG23] Daily Study Group: Introduction to Statistics

A Wolfram U Daily Study Group on “Introduction to Statistics” begins on April 3, 2023.

Join a cohort of fellow statistics enthusiasts to learn about collecting, describing, analyzing and interpreting data and trends in science, industry and society. Learn about techniques for data visualization and descriptive statistics, methods for calculating confidence intervals and tools for hypothesis testing from video lessons created by veteran instructor and developer David Withoff. Participate in live Q&A and review your understanding through interactive in-session polls. Complete quizzes at the end of the study group to get your certificate of program completion.

April 3-14, 2023, 11am-12pm CT (4-5pm GMT)

REGISTER HERE

Please feel free to use this thread to collaborate and share ideas, additional resources and questions with the instructors as well as with other members of the study group.

11 Likes

Reminder that the statistics group starts Monday! Author @David Withoff will join us as we kickoff our latest Daily Study Group. A pre-release version of the interactive course framework will be shared with participants. Sign up here.

2 Likes

Show does not work properly in Mathematica 13.2.x. The overlay shown in the downloaded notebook 8.UsingHistogramData.nb fails to show the overlay!

8.UsingHistogramData.nb (51.3 KB)

How to take the Quizzes and the Online Course Exam?

Hi Marvin,

The reason is that Show takes PlotRange from its first argument (Histogram) and you will notice that the second argument (ListLinePlot) has a very different PlotRange. Why???

Because, unexpectedly, MovingAverage performs a unit conversion from Celcius to Kelvin.

MovingAverage[{Quantity[22, "DegreesCelsius"], Quantity[24, "DegreesCelsius"]}, 2]
(* {Quantity[5923/20, "Kelvins"]} *)

To get the plots to overlay correctly, convert back to Celsius.

Show[Histogram[data, Automatic, "PDF"], 
 ListLinePlot[
  Transpose[
   MapAt[UnitConvert[MovingAverage[#, 2], "DegreesCelcius"] &, 
    HistogramList[data, Automatic, "PDF"], 1]], PlotMarkers -> All]]

This looks like a bug to me, or at least unexpected, undocumented (as far as I can tell) and can result in unexpected errors. You should report it to Wolfram Support.

1 Like

There was an update to how temperature units are handled in 13.2. This should provide an explanation: Temperature Units—Wolfram Documentation

Thank you very much for diagnosing the problem, Rohit! I got a response from support about the units on the x-axis not corresponding between the ListLinePlot and the graphic, and that was also not making sense to me at the time. Your response made it all very clear and, as you indicate, I do not think that this modification in implicit conversion was not documented in the prerelease documents I received for 3.2 or 3.3. It certainly is not compatible with the documentation and is possibly an easily corrected bug.

I truly appreciate your persisting in analysis of this conundrum! I’ll re-report it to Support.

-Marv Schaefer

Thank you, Abrita. Seeing the documentation is what I badly needed !

-Marv

We will share links to quizzes and final exam during the DSG.

1 Like

Hi;

I am interested in finding the value of “x” in a probability when I know the probability in which I am interested. Intuitively one would assume that the Solve function should produce the answer by simply solving for x - see below. However, Solve, SolveValue or Reduce does not return the desired results. In some situations, I know the probability in which I am interested but I do not know the value of x that gives that probability without a lot of trial and error, so I hope someone could point me in the right direction.

Thanks,
Mitch Sandlin

Solve[0.6 ==
Probability[x, x [Distributed] NormalDistribution[998, 202]], x]

Dear Mr. Sandlin:

I am NO expert on Wolfram or Probability but the following may help you.

Sandlin-solve-probability.nb (18.6 KB)

Hi Juan;

Thanks so much. It was actually the InverseCDF function that performed the correct calculation.

Mitch Sandlin

In the notebook “11.NumericalSummariesOfData.nb” there are images of this kind:

You animated these diagrams in the accompanying video and I wondered how you did it. Unfortunately, there are only pictures in the notebook (which makes sense, since you have explanations in the notebook that are “on the soundtrack” in the video). Even if it’s a little off-topic, but: would you be so kind and share with us the code of this stunning animation?

Hi;

In using both functions NormalCI and MeanCI, I am getting some unexpected results - please see attached notebook.

Thanks,

Mitch Sandlin

NormalCI_Problem.nb (11 KB)

The animations are done using dynamic displays with off-screen controls. The programming is off-screen to avoid the potential distraction from the topic of the video, and avoid cluttering up the screen with a bunch of code that has nothing to do with statistics.

For a simple example of the basic idea, evaluate:

x = 0; Slider[Dynamic[x], {0, 10}]

to get a slider for controlling the global value of x (which is initially set to zero) and then evaluate

Dynamic[AngularGauge[x, {0, 10}]]

to get an angular gauge that shows the value of x. Moving the position of the slider changes the angular gauge.

The animation in the video is done by putting the slider in an off-screen notebook, and copying the dynamic AngularGauge display to the notebook shown in the video. The off-screen slider then controls the dynamic display in the video.

Other than that, the only difference in that particular animation is that there are two off-screen controls (one for use when describing “skewness” and one for use when describing “outliers”) and the display is a labeled NumberLinePlot graphic rather than an AngularGauge display.

There are several other animated illustrations in the statistics course that are done in this same way.

A disclaimer regarding the code for this animation is that this code is ad hoc and not intended for public release. This code is also not intended to show exemplary programming practice. It’s single-use code written to get the desired animation in this one example. There are lots of details here for getting the display to be exactly what is wanted. None of those details are especially tricky or complex, but there are nevertheless lots of details.

With that disclaimer, here is the code that was used to generate the animation in this video.

x = 0; Slider[Dynamic[x], {-4.3, 3.3}]
y = 0; Slider[Dynamic[y], {-30, 26}]
data = {45.4, 48.1, 49.7, 50.4, 51.5, 52.5, 53.4, 54.3, 54.8, 55.9, 
  57.8, 60.9}; Dynamic[
 md = Median[data];
 d1 = Take[data, 5] - md;
 d2 = Take[data, -5] - md;
 pts = If[x > 0, Join[md + d1, data[[{6, 7}]], md + (1 + x) d2], 
    Join[md + (1 - x) d1, data[[{6, 7}]], md + d2]] + {UnitStep[-y] y,
     0, 0, 0, 0, 0, 0, 0, 0, 0, 0, UnitStep[y] y};
 m = Mean[pts];
 md = Median[pts]; 
 Show[NumberLinePlot[pts, BaseStyle -> 14, ImageSize -> 600,
   Epilog -> {Text[
      StringTemplate["mean = ``"][Round[m, 0.001]], {m, 4.8}], 
     Arrow[{{m, 2}, {m, 1.5}}], 
     Text[StringTemplate["median = ``"][md], {md, -3.2}], 
     Arrow[{{md, -1}, {md, 0}}]}, AspectRatio -> 0.2, 
   PlotStyle -> PointSize[0.01]], 
  Graphics[{Opacity[0], Point[{45, 5}], Point[{45, -5}]}], 
  PlotRange -> {{20, 80}, Automatic}]]
3 Likes

Hi Mitch,

I suspect it has something to do with the handling of temperatures and temperature differences, see the link in Abrita’s post.

I’m always a bit cautious when it comes to dimensionally significant quantities. Especially with large data sets, I have the feeling that the calculations are faster with pure numbers. Therefore, just like you, I would have loaded the temperatures first and put them into a consistent format. However, then I would have dropped the units. You can do that for example like this:

wdAllFlat = QuantityMagnitude@Flatten[wdAll];

With dimensionless numbers you get the expected results (as you have already shown with your “copy&paste approach”).

A comment on your second question: The variable wdAllFlatt still has units in your case - therefore you get a normal distribution with units (http://reference.wolfram.com/language/ref/QuantityDistribution.html). However, if you work with pure numbers, then you will get the result with pure numbers only - this makes plotting much easier, for example…

By the way, this is not a speech against the use of units in WL - on the contrary: I am a big fan of it; but not in every context.

Dear Mr. Withoff,

Thank you very much for your answer.

You are absolutely right, the code does distract a lot from the actual topic. There is also a separate course on the topic of “animations” here on WolframU.

Still, thanks for sharing, I found your method of moving the five points or the single point via the join function (pts = …) very enlightening. Also that you were able to assemble the final image with only “Epilog” was not something I would have expected.

Please post the link to the course framework on this community site. The chat pane does not appear in the recording of the course so if I miss that lecture I won’t see the link. Thanks!

In the lecture today on the Multiple Testing Problem, a website was mentioned of someone who made it his hobby to collect false positives, or “spurious correlations”, e.g., a time series of people drowning in swimming pools over successive years that appears to be very similar to the time series of movies coming out starring Nicholas Cage. The website shows many more funny coincidences.

This is the website:
Spurious Correlations

There’s a book as well by the same author:
Amazon.com

1 Like

Unfortunately the course framework is not ready for release yet. We are only sharing a beta version with our study group attendees. We can include the link in our reminder emails, so you can get to it, even if you miss the live session.