Message Boards Message Boards

Parameter estimation for nonlinear difference equation

Posted 8 years ago

Given input-output sequences with uniform sampling of a natural growth process, I would like to estimate model parameters in a nonlinear difference equation. Here are example input-output sequences:

in = {16.969, 15.151, 6.83824, 2.51202, 7.62716, 15.7341, 16.6111, 9.15233, 2.76235, 5.4981, 13.9102, 17.392, 11.5535, 3.75623, 3.83153, 11.6845, 17.4135, 13.7951, 5.39155, 2.79867, 9.28573, 16.6734, 15.6468, 7.50031, 2.50563, 6.96037, 15.2477, 16.9184, 9.86588,  2.98251}
out = {0.1, 0.235755, 0.520125, 1.58736, 5.30637, 10.8286, 14.5398, 17.9214, 24.4315, 36.8878, 47.4856, 51.8436, 54.3319, 57.7956, 63.7485, 68.5763, 70.3306, 71.0822, 71.8903, 73.2932, 74.6665, 75.2082, 75.4013, 75.5603, 75.8267, 76.1532, 76.3087, 76.3591, 76.3908, 76.4382}

input-output sequences for this example

The model I would like to fit by estimating parameters a, b, & c and initial condition out[0]=out0 is

iter[a_, b_, c_, y0_] := Module[{}, Clear[y];
out[n_] := 
out[n] = out[n - 1] (1 + a*Exp[-(b in[[n - 1]] + c (n - 1))]); 
out[0] := out0;
y /@ Range[Length[in]]];

FindFit would be ideal, as it allows setting initial parameter estimates, gradient method, as well as the norm function. However, I could not find any examples of FindFit using discrete time models such as this and I'm not sure of its applicability or required syntax.

I would appreciate any suggestions for a parameter estimation scheme suitable for this problem.

POSTED BY: Craig Randall
10 Replies
Posted 8 years ago

Hi Craig,

I changed your difference relation slightly so y[n] is a function of in[[n]], otherwise the count on in and out does not match. The technique below uses the difference relation build a sum-of-squares error function between symbolic terms and the list of out values, and then uses FindMinimum to determine parameter values which minimize the error. It's a useful technique.

Best regards, David

In[1]:= in = {16.969, 15.151, 6.83824, 2.51202, 7.62716, 15.7341, 
   16.6111, 9.15233, 2.76235, 5.4981, 13.9102, 17.392, 11.5535, 
   3.75623, 3.83153, 11.6845, 17.4135, 13.7951, 5.39155, 2.79867, 
   9.28573, 16.6734, 15.6468, 7.50031, 2.50563, 6.96037, 15.2477, 
   16.9184, 9.86588, 2.98251};

In[2]:= out = {0.1, 0.235755, 0.520125, 1.58736, 5.30637, 10.8286, 
   14.5398, 17.9214, 24.4315, 36.8878, 47.4856, 51.8436, 54.3319, 
   57.7956, 63.7485, 68.5763, 70.3306, 71.0822, 71.8903, 73.2932, 
   74.6665, 75.2082, 75.4013, 75.5603, 75.8267, 76.1532, 76.3087, 
   76.3591, 76.3908, 76.4382};

In[3]:= Length /@ {in, out}

Out[3]= {30, 30}

In[4]:= (* mofified to give out[[n]] as a function of in[[n]]*)
y[n_] := y[n] = y[n - 1] (1 + a Exp[-(b in[[n]] + c (n))]);

In[5]:= y[0] = y0; (* to be determined *)

In[6]:= (* sum of squares error from measured out *)
(* to big to print conveniently *)
error = Sum[(y[n] - out[[n]])^2, {n, Length[out]}];

In[7]:= (* the first term of the sum *)
error[[1]]

Out[7]= (-0.1 + (1 + a E^(-16.969 b - c)) y0)^2

In[8]:= (* the second term *)
error[[2]]

Out[8]= (-0.235755 + (1 + a E^(-15.151 b - 2 c)) (1 + 
     a E^(-16.969 b - c)) y0)^2

In[9]:= (* minimize the error *)
sol = FindMinimum[error, {a, b, c, y0}]

Out[9]= {54.7861, {a -> 7.49923, b -> 0.0349481, c -> 0.296997, 
  y0 -> 0.0143244}}

In[10]:= (* a table of calculated out values *)
check = Table[y[n] /. sol[[2]], {n, Length[out]}];

In[11]:= plot = 
 ListPlot[{out, check}, PlotLegends -> {"given", "modeled"}]

enter image description here

Attachments:
POSTED BY: David Keith

This is great. The problem I want to warn you of though is that you're inventing your own statistics. You won't know any of the formal properties of that statistic. If I were reviewing your work and asked you questions about the values, you won't be able to say anything more than they seemed to minimize the squared error.

That may or may not be sufficient for your needs.

POSTED BY: Sean Clarke
Posted 8 years ago

I agree completely. The fit is reasonable, and it is easy to look at the error on a point-by-point basis. But there is no information regarding the fit parameters. There is no confidence interval associated with their values. In fact, one or more of the parameters could be insignificant. I usually think of curve fitting as a process for determining the values of parameters which are physically meaningful. It is the knowledge about the parameter values that is meaningful -- the graph is just a picture. So I think the deficiency is serious. I am spending some time looking at TimeSeriesModelFit in the documentation. I'm finding the water a bit murky. But it is definitely worth studying.

POSTED BY: David Keith

The problem is that the issue is being approached backwards. This is pretty common. I see mostly with differential equations.

Scientists/Researchers/Grads/Undergrads will often come up with a differential equation and then want to solve it or do something significant with it. That's not how things work. There are a few kinds of differential equations that are possible to work with and you can use in your analysis. The job is understanding how to use the already studied models.

I think the situation is even more extreme in time series analysis. There are very few fundamental, simple models you can use and sometimes you have to be a bit creative. It's very much possible that this isn't expressible as any of the common kinds of statistical processes. That means you probably want to simplify your model to use one of them.

POSTED BY: Sean Clarke

David, thank you very much for walking me through the FindMinimum approach to fitting parameters in my model. I played with your nb, and found that it does everything I wanted in terms of setting initial parameter estimates, gradient method, norm function ("error" in your nb). I'm looking at changing my modeling approach based on Sean's considered advice regarding well-known time series models. For now, though, you've provided the solution I was looking for. I appreciate your taking the time to help.

POSTED BY: Craig Randall

You're doing time series analysis. FindFit is not for time series analysis.

The function for fitting a time series to a time series process is called TimeSeriesModelFit

Are you familiar with what ARIMA, or ARMA, or MA processes are? You'll first want to know what these are before continuing. These are the very basic kinds of processes people work with.

The function that describes your process is this:

out[n] = out[n - 1] (1 + a*Exp[-(b in[[n - 1]] + c (n - 1))])

That is a monster of process. First make sure you understand the simple kinds of process. You should be at least able to approximate your process using ARIMA or something. ARIMA has ... known properties and stuff. It's useful.

POSTED BY: Sean Clarke
Posted 8 years ago

Sean, I saw your response after posting my method. It looks very interesting. I have not worked with TimeSeriesModelFit -- but I will certainly be investigating it. Thanks!

POSTED BY: David Keith

Sean, thank you for your valuable feedback and for pointing out that FindFit is not appropriate for time series analysis. I am modeling a biological growth process with exogenous inputs (e.g., temperature) that generate an output (e.g., size). I had looked at TimeSeriesModelFit, however, that function does not seem to allow for exogenous inputs, nor do there appear to be other time series functions in Mathematica that allow for exogenous inputs.

If I understand you correctly, you are recommending identifying a well-studied generic time series model, which with some creative adaptation might be applied to this problem, rather than beginning with a relatively unknown, nonlinear system model. I did a cursory check for time series models with exogenous inputs, and found there is a body of work in this area. I appreciate your turning my approach on its head--your strategy is something I at least need to understand before proceeding.

POSTED BY: Craig Randall
Posted 8 years ago

I suggest you have a look at your iter function. y is used as afunction, but nowhere defined. in[[n-1]] is used, which means the lowest value of n that can be used is 2, which will extract the first value of in. out is defined as a function, but out was previously defined as the output list.

POSTED BY: David Keith

David, thank you for the corrections. With your revisions to the iter function, i.e.,

iter[a_, b_, c_, y1_] := Module[{}, Clear[y];
y[n_] := y[n] = y[n - 1] (1 + a*Exp[-(b in[[n - 1]] + c (n - 1))]); y[1] := y1; y /@ Range[Length[in]]];

how can iter be used in expr in a curve fitting function such as FindFit? This seems to be where I'm stuck now.

POSTED BY: Craig Randall
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract