Message Boards Message Boards

[LiVE] Quantile Regression Workflows (WL Live-Stream Series)

Posted 5 years ago

The live-coding sessions

  1. In the first live-streaming / live-coding session I demonstrated how to make Quantile Regression workflows using the software monad QRMon and some of the underlying software design principles. (Namely "monadic programming".)

  2. In the follow up live-coding session I discussed topics like outliers removal (data cleaning), anomaly detection, and structural breaks.

  3. In the third live-coding session:

    • First, we demonstrate and explain how to do QR-based time series simulations and their applications in Operations Research.
    • Next, we discuss QR in 2D and 3D and a related application.
  4. In the fourth live-coding session we discussed the following the topics.

    • Review of previous sessions.
    • Proclaiming the upcoming ResourceFunction["QuantileRegression"].
    • Predict tomorrow from today's data.
    • Using NLP techniques on time series.
    • Generation of QR workflows with natural language commands.
  5. In the final, fifth live-coding session different questions are going to be answered.

    • Can we apply Quantile Regression (QR) without the package QRMon?
    • Yes, see the Wolfram Functions Repository item “QuantileRegression”.
    • Are you computing QR using a moving window?
    • Related, how QR compares to Local regression? Or LOESS?
    • What do you do when the QR fitted curves (regression quantiles) intersect?
    • How does QR fitted curve looks like over 3 points?
    • What approximation to pick for reconstructing the conditional CDF’s?
    • Why use QR? Is it just for better visualization of the signal?
    • What is the point of using those anomaly detection methods -- a human can easily do it?
    • Can we use Neural Networks instead of QR?
    • Are there implementations in other popular DS languages?

Notebooks

  • I did not save the notebook I made during the first live coding session, but I attached to this post a modified version of the notebook I used for a Meetup workshop 6-7 months ago. (See for more details the GitHub MathematicaVsR project "Quantile Regression Workflows".)

  • The notebook of the 2nd session is also attached. (I added a "References" section to it.)

Resource function

Instead of the software monad QRMon used in the live-coding sessions the Wolfram Resource Function QuantileRegression can be used. The resource page has many example applications that are also discussed in the live-coding sessions.

POSTED BY: Anton Antonov
11 Replies

Great job Anton! Looking forward to your future live-streams (next one will be today Sep 13 at 1:30pm PT, Wolfram TwitchTV channel)

POSTED BY: Mohammad Bahrami

Just attached/uploaded the "live" notebook of the second session.

POSTED BY: Anton Antonov

Join for the 3rd session now: https://www.twitch.tv/wolfram

POSTED BY: Mohammad Bahrami

Just added/attached the notebook of the 3rd live-coding session.

POSTED BY: Anton Antonov

Join us for the 4th session on Tuesday Oct. 22nd, 2pm PT (https://www.twitch.tv/wolfram) 4th session: Quantile Regression workflow: additional topics. - Predictions for today from yesterday’s data. - Conversational agent generated workflows. - Application of NLP techniques to time series processed with QRMon.

POSTED BY: Mohammad Bahrami

Just uploaded the notebook from the 4th live-coding session.

POSTED BY: Anton Antonov

A few days ago the Wolfram Function Repository entry QuantileRegression was approved. The resource description page has many of the topics discussed in the live-coding sessions on Quantile regression.

Here is the code result of the neat examples section:

SeedRandom[20];
distData = Table[{x, Exp[-x^2] + RandomVariate[NormalDistribution[0, .15 \[Sqrt](Abs[1.5 - x]/1.5)]]}, {x, -3, 3, .01}];
Grid[Table[
  Block[{tempData = RandomSample[distData, 100], data, probs}, 
   data = Join[tempData, {#[[1]], -#[[2]]} & /@ tempData];
   data = SortBy[Transpose[Rescale /@ Transpose[data]], #[[1]] &];
   probs = {0.02, 0.48, 0.52, 0.98};
   qFuncs = 
    ResourceFunction["QuantileRegression"][data, 5, probs, Method -> {LinearProgramming, Method -> "InteriorPoint",Tolerance -> 10^(-2)}]; 
   ListPlot[Prepend[(Transpose[{data[[All, 1]], #1 /@ data[[All, 1]]}] &) /@ qFuncs, data], 
   Joined -> Prepend[Table[True, Length[probs]], False], 
    Filling -> Prepend[Table[(i + 1) -> {i + 2}, {i, Length[probs] - 1}], 1 -> None], Frame -> True, FrameTicks -> False, Axes -> False, 
    ImageSize -> Small, AspectRatio -> 1]], 3, 3]]

enter image description here

POSTED BY: Anton Antonov

Great work, Anton. Having the ResourceFunction available should help a lot of people.

POSTED BY: Seth Chandler

Thank you, Seth! WFR is a great way to introduce new, user derived functions in the way, say, CRAN does for R.

POSTED BY: Anton Antonov

Just uploaded the notebook from the final, 5th live-coding session.

POSTED BY: Anton Antonov
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract