- another post of yours has been selected for the Staff Picks group, congratulations !

We are happy to see you at the top of the "Featured Contributor" board. Thank you for your wonderful contributions, and please keep them coming!

Just to give another example, I saw this R code under the first search result for "dplyr":

by_tailnum <- group_by(flights, tailnum)
delay <- summarise(by_tailnum,
  count = n(),
  dist = mean(distance, na.rm = TRUE),
  delay = mean(arr_delay, na.rm = TRUE))
delay <- filter(delay, count > 20, dist < 2000)

# Interestingly, the average delay is only slightly related to the
# average distance flown by a plane.
ggplot(delay, aes(dist, delay)) +
  geom_point(aes(size = count), alpha = 1/2) +
  geom_smooth() +
  scale_size_area()

Here is the output:

Here is equivalent Mathematica 10.1 code and a similar chart.

flights[GroupBy@
     "tailnum", <|"count" -> Length@#, 
      "dist" -> N@Mean[#[[All, "distance"]]], 
      "delay" -> N@Mean[#[[All, "arr_delay"]]]|> &] // 
   Select[#count > 20 && #dist < 2000 &] // 
  Map[{#dist, #delay, #count} &] // 
 BubbleChart[#, ChartBaseStyle -> {Black, Opacity@.5, EdgeForm[None]},
    BubbleSizes -> {0.005, 0.1}, AspectRatio -> 1/2, 
   GridLines -> Automatic, FrameLabel -> {"Dist", "Delay"}] &

Hi Sam, Yes, I exported the sample dataset from R with write.csv() and then I used SemanticImport in Mathematica to convert it to a dataset. For a dataset of that size SemanticImport takes a minute. Maybe you could speed it up by manually specifying the column types, but a minute is fine for this. The blue curve and shading are a predicted mean and confidence interval. It's produced by the geom_smooth() function in the ggplot R package. According to their documentation, they use a generalized additive model for data sets larger than 2000 points, confidence bands of 95%, and some heuristics/meta-algorithms to select smoothness that I didn't dig into. I think a linear model of degree 10 is fine for this illustration.

flights = SemanticImport["E:\\flights.csv"];

summary = 
  flights[GroupBy@
     "tailnum", <|"count" -> Length@#, 
      "dist" -> N@Mean[#[[All, "distance"]]], 
      "delay" -> N@Mean[#[[All, "arr_delay"]]]|> &] // 
   Select[#count > 20 && #dist < 2000 && NumberQ@#delay &];

lm = LinearModelFit[{#dist, #delay} & /@ Values@Normal@summary, 
   Table[x^n, {n, 10}], x];
bands = lm["MeanPredictionBands"];

bc = summary // Map[{#dist, #delay, #count} &] // 
   BubbleChart[#, 
     ChartBaseStyle -> {Black, Opacity@.5, EdgeForm[None]}, 
     BubbleSizes -> {0.005, 0.1}, AspectRatio -> 1/2, 
     GridLines -> Automatic, FrameLabel -> {"Dist", "Delay"}] &;

Show[bc, Plot[bands, {x, 170, 2000}, PlotStyle -> None, 
  FillingStyle -> {Blue, Opacity@.75}, Filling -> {1 -> {2}}], 
 Plot[lm[x], {x, 170, 2000}]]

As it turns out, the Wolfram Language now has a convenient way of chaining: the right composition operator /*. I'm currently producing a L O N G work on how to extract information from lists of Associations, nested Associations and Dataset that makes significant use of the RightComposition construct.

Seth, this is great, thanks for sharing. For such a nice content I felt it is worth mirroring the the notebook right in the post and went ahead with it. I am glad you are taking advantage of the Dataset!