I appreciate what must have been a huge challenge to combine all of the "fits" into a single function but the ability for obtaining confidence or prediction bands, parameter correlations and covariances, and other measures of goodness-of-fit statistics seems almost hidden because of the meager description of "Report" in the ModelFit documentation.

Finally, I'm not so thrilled with the use of the CrossEntropy term under CriterionFunction, when that is equivalent to Maximum Likelihood which is the standard statistical term for many decades. That seems to be an appeasement to the machine language folks. At least include MaximumLikelihood as one of the options.

If you are going to create additional features, please consider adding "mixed models" where a specific error model can be specified (i.e., more than one error term).

OK. Enough complaining. Mathematica is still by far my favorite software package since Version 1.

Hi Jim,

your complaint is perfectly valid, and I'm well aware we're missing a lot, not just relative to R or statsmodels, but to the old ModelFit as well.

The good news is that most of what you're asking for is already on the roadmap for 15.1. Given the interest (which I'm genuinely happy about), let me elaborate a bit.

And you're right about the documentation: the inference and goodness-of-fit machinery is real but buried behind a one-line mention of "Report". That's on us to fix.

CriterionFunction

I see exactly where you're coming from, and we're going to explain this far better as we flesh the system out. We're trying to walk a fine line between the ML and statistics worlds, so it's going to be a little bumpy at the start.

The key idea is that two things often get conflated. One is the objective the optimizer minimizes when fitting a single model — for most of our models that's the negative log-likelihood, i.e. maximum likelihood. The other is the criterion used to choose among competing candidate models (different hyperparameters, different model classes). That second job is what CriterionFunction controls. The plain is for it to be an in-sample measure like RSquared or AIC, or a cross-validation loss that estimates out-of-sample performance (what you currently see).

So when you see CrossEntropy under CriterionFunction, that's the held-out loss used for selection, not the fitting objective — even though, as you note, it's the same log-likelihood the statistician already knows by name. Point taken on the naming; making MaximumLikelihood (and other familiar statistical names) recognized options is exactly the kind of thing I want to settle when I streamline this in 15.1, alongside a larger built-in selection and support for custom functions.

Inference and diagnostics

We plan to drop the internal bridge to FittedModel entirely and compute every property natively in the new framework. That transition is nearly done, so I don't see a blocker.

As a teaser, here are prediction error bands at several confidence levels for linear, nonlinear, k-NN and decision tree models:

This is currently exposed as a "runtime" property, model[input, "DataAround"]. You can also ask for the full predictive distribution and make some nice plots:

Feature parity

Bringing back the remaining FittedModel capabilities:

• weights
• support for Around
• categorical data via one-hot encoding
• the missing options

New models

Here the sky is the limit, but at a minimum I want:

• GLM (with a new multinomial link, and likely more)
• GAM
• mixed models
• GAMM

Mixed models in particular are squarely aimed at what you describe — specifying an explicit error structure with more than one error term.

All of this is still pending review, so please treat it as a rough direction rather than a commitment.

Let me close by saying I really appreciate the time and effort you put into this feedback. The framework is very much a work in progress, and comments like yours are exactly what we want!

I really like the new model fitting framework, it makes understanding different model types much more streamlined. Are GLMs covered in the new model fitting framework in this first release? Which function is the equivalent of GeneralizedLinearModelFit with a link function?

As an example, how do i do the following in the new framework:

GeneralizedLinearModelFit[$data,{x,x^2},{x},
    IncludeConstantBasis->True,
    LinearOffsetFunction->(Sqrt[#]&),
    ExponentialFamily->"Gaussian",
    LinkFunction->"LogLink"
]

The parametric function, I think the above translate to is as follows:

"Response" = Exp[Sqrt[x]+C[1]+x C[2]+x^2 C[3]] + "GaussianError"

How do i define the same model in the new framework?

Individually I can do the following:

This defines the linear part of the model:

LinearModel[{1,x,x^2,Sqrt[x]},{C[1],C[2],C[3],1},x]

Sqrt[x]+C[1]+x C[2]+x^2 C[3] + "GaussianError"

This defines the non-linear log-link part of the model:

ExponentialModel[{0,1,1},x]

Exp[x] + "GaussianError"

But how do I compose a linked model like the GeneralizedLinearModelFit?

Actually an idea but I don't know how doable that would be. Since the theoretical framework of these models are so disperse. But here goes:

Instead of making a GeneralizedLinearModel[] or LinearModel[..., LinkFunction->...] in this new framework, wolfram could make a ComposedModel[model1, model2, model3, ...] similar to mathematical Compose[...] which could give a more general composed model which would naturally include GLM as a special case. Also since all these classical models are uni-valued so there wouldn't be a need to make complex chains or graphs, a simple composition might do it for them, I think maybe.

As an example, the following would replicate the GLM I mentioned:

ComposedModel[
    ExponentialModel[{0,1,1},x],
    LinearModel[{1,x,x^2,Sqrt[x]},{C[1],C[2],C[3],1},x]
]

But it would go beyond and truly exploit the nice architecture of having a single function ModelFit doing the fitting for all these and allow for things like composition of DecisionTreeModel with a LinearModel and so on. Imagine the possibilities! Interpreting the results would be a challenge though since people know how to interpret the parameters of each of these models in isolation but together might be more difficult. But I think with neural networks people have already gotten used to not being able to assign meaning to every parameter so it wouldn't be such a difficult pill to swallow for the experts.

-- you have earned Featured Contributor Badge Your exceptional post has been selected for our editorial column Staff Picks http://wolfr.am/StaffPicks and Your Profile is now distinguished by a Featured Contributor Badge and is displayed on the Featured Contributor Board. Thank you!