Re: Model building algorithm

From: James <J.G.Wright@ncl.ac.uk> Subject: Re: Model building algorithm Date: Tue, 25 Apr 2000 13:37:13 +0100 Dear Model builders, The statistical model defines how the differences between the predictions and observations should be "priced". In this context, interindividual ETA's allow us to pay less for persistent differences between individuals, in terms of the parameters to which they are applied. The problem with backward elimination approaches is that we may end up starting from an overparameterized model and may not be able to make reliable decisions about the components we remove. Although the general textbook advice is to take a stepwise approach to modelling, I too have personal experience of situations were this would be misleading. It was interesting that in Mark's original example, the parameterisation to which the random effects were applied affected the eventual decision. This isn't surprising as random effects on say rate constants or their product with volumes allow for the interindividual ETA to discount serial correlations in different ways. And if we allow off-diagonal elements, this gets even more complicated. Although for example CL and V are almost invariably correlated in practise, I am personally reluctant to introduce a parameter that can enforce such a correlation before attempting to explain noise with (fixed) covariate effects. Incidentally, I was under the impression that asymptotically we can test the change in objective function against a chi-squared distribution to assess STATISTICAL significance. However, this is probably irrelevant as part of model building, which I do not believe should be a sequential decision making process (which ignores interactions between model components and multiple testing issues). And I worry about applying asymptotic tests to a non-infinite sample. Returning to the original point, I add thetas and then etas usually and objective function drops reassure me a little. This approach could fail if the fixed effect alone did not seem to explain noise, which could easily happen, if for example there was a lot of interindividual noise and especially of the residual error model can slide itself around to downweight the points in the "distribution" phase, and probably for many other reasons which I cannot even conceive of (I'm guessing at situations where there is a small mean difference between the misspecified null model and the more complex alternative but a lot of variability which can be discounted). Or perhaps there a samples in the "distribution" phase in only a subset of individuals. Whilst generally, I would say "Don't let hypothesis testing procedures boss you around", if they don't back you up, you need to be able to justify your position by explaining why. A good way to do this would be to say there were banana-shaped residual patterns for example, although this does not necessarily mean we can characterise the more complex model (and hence continue reliably with that model) although we suspect it is more appropriate. If we honestly believe our objective function is missing something, then maybe we should be double-checking how it has been constructed on the null model- for example is the residual model appropriate? ELS is notorious for its vulnerabibility to misspecification of variance functions, and in this situation objective function changes may not be trustworthy. And even if we believe they are chi-squared distributed (I personally think they have higher variance) and used a five percent alpha, we would make the wrong decision 5% of the time (colloquially speaking). And who knows what effect this owuld have on subsequent tests...thinking about this gives me nightmares. Hopefully, there are clues in other aspects of the model - residuals, estimates, and how they have changed - becuase the objective function alone is a very blunt tool. James Wright