Re: T matrix

From:Nick Holford Subject:Re: [NMusers] T matrix Date:Mon, March 11, 2002 5:56 pm Leonid, Thanks for posting these references and thank you also for sending me separately PPT copies of your poster material. One of the posters (aripiprazole) has a table that allows numerical comparison of the symmetry and coverage produced by 95% confidence intervals derived from log likelihood profiling (LLP) and NONMEM standard errors (SE). Using your own numbers I find that the LLP method in several cases has substantially different coverage and is often asymmetrical. Parameter Sym Coverage q1 CL 84% 92% q2 V 107% 97% q3 Ka 107% 200% q4 CAT1 CL 102% 103% q5 LBW CL 103% 106% q6 AGE_V 92% 138% q7 WT_V 81% 158% q8 omega1 100% 86% q9 omega2 126% 77% q10 omega3 88% 103% q11 sigma 100% 40% Sym=(abs((Hi-Est)/(Lo-Est)) LLP)/(abs((Hi-Est)/(Lo-Est)) SE) Coverage=((Hi-Lo) LLP)/((Hi-Lo) SE) Est=parameter estimate; Lo=95% CI lower; Hi=95% CI upper bound This poster concludes "NONMEM standard errors and CI of the parameter estimates were similar to the ones obtained by more computer-intensive methods.". I would not draw the same conclusion. It depends on what you want to call similar but I don't consider the coverage on KA, AGE_V, WT_V to be similar. The asymmetry of CL and WT_V is also more than trivial. Nick TSR Matrices thread From:Leonid Gibiansky Subject:[NMusers] NONMEM SE and CI Date:Tue, March 12, 2002 9:06 am Nick, Let's put this discussion into the prospective: 1. We wrote "similar", but not identical, so I would not expect a one-to-one agreement. 2. Asymmetry of CL: if you look on the plots, asymmetry of profiling CI is in the different direction compared to the asymmetry of bootstrap CI. If so, it would be hard to argue which of them to trust. 3. Asymmetry of WT_V: yeas, I agree, there is some asymmetry that is not recognizable by NONMEM. But is it really important difference: 0.489-0.953 (profiling), 0.599-0.895 (NONMEM) with the estimate equal to 0.746 ? Is it really "not similar" when you see 20% difference in the symmetry of the confidence interval for the parameter if the covariate model ? 4. Coverage on KA: yes, difference is large here, but bootstrap CI are 1/3 way between the NONMEM CI and profiling CI. So NONMEM is not so bad. 5. Coverage on AGE_V, WT_V: again, there is a difference, but bootstrap CI are between NONMEM and profiling. So again NONMEM is reasonable. Let me also add that for the parameters that describe random effect variances, omega1-omega3, NONMEM CI are approximately half-way between the profiling and bootstrap CI, with a rather significant gap between these "better" methods. For the variance of the error term, sigma, the NONMEM CI coincides with the bootstrap CI whereas the profiling CI are twice more narrow. So if you would be forced to trust the results of just one method, I would trust NONMEM for this particular problem, with the other methods giving similar although not identical results. Another peace of information: this is FOCEI, so bootstrap runs took about 2 weeks of computer time if I remember correctly (500 out of 1000 runs converged). With profiling, there were many-many runs that were interrupted by numerical errors, and we started them again and again with the new initial values (I would estimate, 250 more runs were made). It took a lot of time even with our automated routines. I am not sure that this is an adequate price for the 20% improvement in the CI interval for WT_V, or even 40% improvement that we would get for KA CI. From CI, I can get the following qualitative information: - parameter is well-defined; - parameter is defined but not too well; - the NONMEM converged but the parameter value cannot be trusted. This qualitative information and also quite reliable quantitative information can be readily extracted from the NONMEM SE, and none of the more elaborate techniques will change it. It is surely sufficient during the model development and for most of the final models as well. If one need specific, up to 20-30% precise CI, then bootstrap, profiling, something else, can help sometimes. This may be needed if you plan to use the model for simulations. Even then, posterior predictive checks methods would be a reasonable alternative to the bootstrap or profiling. I am not trying to diminish the significance of the profiling or the bootstrap methods: I routinely did them on my projects, and they bring new more detailed information about the model. My main objection was the comment that NONMEM SE does not worth much. Actually, they allow you immediately and correctly capture "the big picture" and even most of the fine print, with the few details that can be studied, if really needed, by some more elaborate methods. Leonid