RE: $OMEGA blocks and log-likelihood profiling

From: "Hutmacher, Matt" Subject: RE:[NMusers] $OMEGA blocks and log-likelihood profiling Date: Fri, June 11, 2004 9:53 am Hello all, I have some general overall comments on the discussion based on my understanding of the discussion and on the bootstrap. It seems to me, that for the bootstrap to perform correctly, one must have a "proper" estimate. In the simple case of a univariate random identically distributed random variable (not necessarily known), the sample mean is an estimator of the population mean. This population may be bootstrapped, the sample means for each bootstrap calculated, and the sample means can be ordered to provide an estimate of the confidence interval (which I believe is asymptotically consistent with the true confidence interval for very general situations). In calculating the sample mean, no iteration is necessary, so we are always assured of a "proper" estimate of the mean. When things get more complicated in the non-linear mixed effects world, it would seem to me - to ensure good estimates of the confidence intervals that one would want the "proper" (i.e. global minimum) parameter estimates for each bootstrapped data set. The problem in the practical setting is determining when you have "proper" estimates. This can be difficult to determine even for the original data set. Even when the COV step completes, we may not be sure that we are at a global minimum. To help ensure that we get there, we build stable models and use good starting values. Both of these practices can be applied to the model run on the original data set, but it becomes difficult when bootstrapping because of the large number of runs. Thus, I agree with Ken, that to ensure the best overall performance, not only of the original model run, but the bootstrap procedure, that a stable model should be built and used for inference. As Mats points out, computationally, the COV step is cheap (compared to the bootstrap) in that it gives us an indication of the stability of the model. If the COV step runs, and the model condition number is low, then I would have good faith in proceeding with the bootstrap. However, if the COV step does not run, as Ken indicated, I would be concerned about whether the model has found a saddle point (indicating the estimates are not proper), is overparameterized, etc. I do agree with you Nick, that sometimes the COV may not run for a good model. One often used example is when TLAG is estimated near a data point. Here, the derivative is undefined, so no covariance estimate is available. This model may still yield good estimates and be suitable for inference by bootstrapping. However, in my opinion, for general situations some detective work needs to be done to determine why the COV failed. Otherwise, I would think, some suspicion as to whether the estimates are "proper" would exist in a reviewer's mind. When the COV step fails on the original model, I think the bootstrap could be valuable tool to help show that the estimates are proper. If one looks at the distributions and finds they are reasonable (smooth, unimodal, [and "ideally" symmetric due to the central limit theorem]) then one has some credible evidence that the original model/data is robust. However, one must be careful. In a case like Nick's, where, hypothetically, a large number of runs do not converge, the parameter estimates from the runs that didn't converge could be unnecessarily widening the confidence intervals. Thus, in my opinion, it is good to always include the distribution of the bootstrap estimates in the appendix of one's report. For rounding errors, I agree with Leonid that to some extent, selection of significant digits is arbitrary. However, in my mind, if one can't get three significant digits, one must ask why can't I achieve them. Perhaps if the lowest is 2.9 you could proceed. However, in this case if you change sigdigits to 2 and you get 1.9, wouldn't that be concerning (in the sense of instability)? If the parameter estimates were identical then maybe they are ok, but in my experience this is not usually the case. [I know usually when this happens we increase the sigdigits and trick NONMEM to run the COV using the MSFI option - I state this as somewhat of an old argument or thought process]. If a sigdigit is less than 1, would you trust the estimates. Nick, I would liked to have gone to PAGE and discussed this with you over a pint, as I think you have an interesting problem - that being, "how do you know you have good estimates". There is always room for debate there. But, I have to say, I disagree with the general tone of your 2nd paragraph below. I believe this statement implies (or at least I infer) that this one dataset/model provides proof that the COV step is meaningless. You are using the data to argue your point here, and you claim it is generalizable, which I think is not a scientific argument. You say the proof is in the data and unarguable. However, when data drives a correlation estimate between CL and V to 95%, I believe that you would state that the model is wrong and that this result was in error, even if the model fit the data well (correct me if this is inconsistent with previous emails). You would discount the data in that situation based on an "opinion", wouldn't you? Or is your "opinion" subject matter knowledge... Model fitting is not 100% composed of subject matter knowledge (pharmacology/pharamcokinetics). When one wants to "realize" the data by fitting a model to the data and perform inference, estimation is necessary. Estimation is statistical in nature. Statistical theory plays a valuable role in assessing estimates and ensuring proper inference. Statistics allows us to make probabilistic statements about the data, future predictions, etc. Some statistical rigor is necessary to ensure that the probabilistic statements made will be valid. These "hurdles" have been shown to be valuable over time. If this were not the case, statistics would be such a value part of interpreting clinical trials. Therefore, I would not call good statistical practice "arbitrary" hurdles. Matt