Model Comparison and Viewing of Output

Dear NMusers, I had a couple of questions for the group members: 1. Model Comparison: Everything else remaining the same in a model, should a drop in the objective function on addition of an eta, sigma or a covariance term reflect an improvement in a model. Or comparing AICs, considering eta's and sigma's as additional parameters a correct approach. I understand that these decisions are rather based on multiple criterias like goodness of fit plots, biological sense, physiological plausibility of the parameter estimates etc. But specifically would like to know whether comparing objective function among these models a logical approach? 2. Viewing Output via $TABLE option: When we do simulation using NONMEM,the output file contains both the dosing and observation records. Is there a way in NONMEM to specify a priori in the control stream, such that only observation records could be obtained in the simulation output file? Thanks and Regards Nitin Mehrotra Nitin Mehrotra, Ph.D Post Doctoral Research Fellow 874 Union Avenue, Suite4.5p/5p Department of Pharmaceutical Sciences University of Tennessee Health Science Center Memphis, TN, USA-38163 901-448-3385 (Lab) [EMAIL PROTECTED] --------------------------------- Shape Yahoo! in your own image. Join our Network Research Panel today!

Nitin, No one else has yet answered, so I'll give a try, opening myself to the wrath of pretty much everyone. Model Comparison: Everything else remaining the same in a model, should a drop in the objective function on addition of an eta, sigma or a covariance term reflect an improvement in a model. Or comparing AICs, considering eta's and sigma's as additional parameters a correct approach. I understand that these decisions are rather based on multiple criterias like goodness of fit plots, biological sense, physiological plausibility of the parameter estimates etc. But specifically would like to know whether comparing objective function among these models a logical approach? I suspect that no one would question that model selection should be based on multiple criteria. Which criteria, and how to weight them depends on what you hope to accomplish with the model. If your goal is to get a simulation of something you may care little about tests of hypothesis, objective function, estimation correlation and care more about simulation based methods such as PPC or NPDE. Conversely, if your goal is hypothesis testing, then you likely should be very interested in OBJ (and or bootstrap tests of hypothesis). I'd suggest that there are two issues with evaluating models: 1. How good is the model (in comparison to competing models)? 2. What opportunities are there for improving the model? I assume first, that you are only considering models that are biologically plausible (ignoring whether there are degrees of biological plausibility, and whether this should play ! a role in selection) #1 can often be answered to a significant degree with objective measures (OBJ, AIC, PPC, NPDE), although graphics certainly play an important role, we prefer a model that doesn't show bias in graphics. But, improving bias should improve objective measures. #2. Largely, if not entirely, the role of graphics - most useful graphics IMHO are Visual predictive check and post hoc quantitites vs things that might explain any non-randomness in these post hoc quantities (time vs WRES or CWRES, post hoc eta vs covariates etc), but there are many others (see excellent paper by Ene Ette in Pharm Res, Dec 1995.). I would suggest that a goal (a goal, not the goal) of #2 should be to improve #1. If you make an improvement in plots, without a corresponding improvement in some objective measure, you have to be concerned about what else in the model you've made worse. So, to answer your question, if I had two models, that differ o! nly in AIC (equally biologically plausible, plots are equivalent, same NPDE, same PPC), I would not hesitates to choose the model with the lower AIC. What justification could there be for choosing the other model? Of course, things are rarely that simply, and invariably, other things are not the same. Then one has to choose whether some subtle improvement in a plot, or the PPC, or the NPDE is most important for this model selection exercise. Viewing Output via $TABLE option: When we do simulation using NONMEM,the output file contains both the dosing and observation records. Is there a way in NONMEM to specify a priori in the control stream, such that only observation records could be obtained in the simulation output file? Sort of. You can use the BY option (BY = MDV). This will sort by MDV (with the MDV = 0, then MDV = 1). But this is probably not all that helpful. I'm not sure if Xpose ( http://xpose.sourceforge.net/ ) automatically deletes MDV = 1 records (I think it does), the excel macro at Next Level solutions ( http://www.nextlevelsolns.com/downloads.html ) does delete records where MDV <> 0, if you include this in the $TABLE output. Mark Sale MD Next Level Solutions, LLC www.NextLevelSolns.com > -------- Original Message -------- Subject: [NMusers] Model Comparison and Viewing of Output From: Nitin Mehrotra <[EMAIL PROTECTED]> Date: Tue, August 14, 2007 12:18 pm To: > > [email protected] > > Dear NMusers, I had a couple of questions for the group members: 1. Model Comparison: Everything else remaining the same in a model, should a drop in the objective function on addition of an eta, sigma or a covariance term reflect an improvement in a model. Or comparing AICs, considering eta's and sigma's as additional parameters a correct approach. I understand that these decisions are rather based on multiple criterias like goodness of fit plots, biological sense, physiological plausibility of the parameter estimates etc. But specifically would like to know whether comparing objective function among these models a logical approach? 2. Viewing Output via $TABLE option: When we do simulation using NONMEM,the output file contains both the dosing and observation records. Is there a way in NONMEM to specify a priori in the control stream, such that only observation record! > s could be obtained in the simulation output file? Thanks and Regards Nitin Mehrotra > > Nitin Mehrotra, Ph.D Post Doctoral Research Fellow 874 Union Avenue, Suite4.5p/5p Department of Pharmaceutical Sciences University of Tennessee Health Science Center Memphis, TN, USA-38163 901-448-3385 (Lab) [EMAIL PROTECTED] > > Shape Yahoo! in your own image. > > Join our Network Research Panel today!

Dear Nitin, If I interpret your first question narrowly, we did a study that at least partly addressed this (Wahlby U, Bouw MR, Jonsson EN, Karlsson MO. Assessment of type I error rates for the statistical sub-model in NONMEM. J Pharmacokinet Pharmacodyn 2002;29(3):251-69.) The conclusion was that there seems to be some basis for using the OFV in likelihood ratio tests for looking at aspects of the variability models. However, such tests are sensitive to aspects like method used, shape of the distribution, etc. The final paragraph was: "In conclusion, the LR test is not reliable for the statistical sub-model if the FO method is used. For the FOCE INTER method, the LR test is appropriate when the underlying assumptions of normality of residuals hold true and data per individual are not too sparse. Deviations from normality, which can be diagnosed from the modeling output, may cause estimated type I error rates that deviate markedly from the expected. The type I error rates for inclusion of variance and covariance parameters are the most affected by deviations from normality, while those for covariates on variance parameters are more robust." Best regards, Mats Mats Karlsson, PhD Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Faculty of Pharmacy Uppsala University Box 591 SE-751 24 Uppsala Sweden phone +46 18 471 4105 fax +46 18 471 4003 [EMAIL PROTECTED] _____

Dear Mats, Mark and Nick, Thanks for your response. Mats, thanks for refering to your paper which explains this issue. With regard to the second question, its true that we have plenty of tools which can filter out the observations from other type records post run. However, sometimes the output file size is very big for large simulations that it is difficult to open in programs like Excel (2003) because of row limitations. Thus, I was curious if we can specify something in the control stream a priori such that we only get observation records in the output file. That was the whole purpose of asking this question. Warm Regards Nitin Nick Holford <[EMAIL PROTECTED]> wrote: Nitin, I think Mark and Mats have given you sensible advice on your first question. The bottom line is that there are no magic numbers that are always the correct guide to the solution. Sometimes you just have to apply common sense. Your second question is more of a puzzle. Why are you wanting to do this? If you are planning to examine the NONMEM output table then almost any tool you would use should be able to filter out the records so that only the observations are left. Can you explain why you feel this simple filtering should be done by NONMEM? What are you trying to do with this table output when you have separated the observations from other event type records? Nick Nitin Mehrotra wrote: > > Dear NMusers, > > I had a couple of questions for the group members: > > 1. Model Comparison: Everything else remaining the same in a model, should a > drop in the objective function on addition of an eta, sigma or a covariance > term reflect an improvement in a model. Or comparing AICs, considering eta's > and sigma's as additional parameters a correct approach. I understand that > these decisions are rather based on multiple criterias like goodness of fit > plots, biological sense, physiological plausibility of the parameter > estimates etc. But specifically would like to know whether comparing > objective function among these models a logical approach? > > 2. Viewing Output via $TABLE option: When we do simulation using NONMEM,the > output file contains both the dosing and observation records. Is there a way > in NONMEM to specify a priori in the control stream, such that only > observation records could be obtained in the simulation output file? > > Thanks and Regards > > Nitin Mehrotra > > Nitin Mehrotra, Ph.D > Post Doctoral Research Fellow > 874 Union Avenue, Suite4.5p/5p > Department of Pharmaceutical Sciences > University of Tennessee Health Science Center > Memphis, TN, USA-38163 > 901-448-3385 (Lab) > [EMAIL PROTECTED] > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------- > Shape Yahoo! in your own image. Join our Network Research Panel today! -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand [EMAIL PROTECTED] tel:+64(9)373-7599x86730 fax:+64(9)373-7090 www.health.auckland.ac.nz/pharmacology/staff/nholford Nitin Mehrotra, Ph.D Post Doctoral Research Fellow 874 Union Avenue, Suite4.5p/5p Department of Pharmaceutical Sciences University of Tennessee Health Science Center Memphis, TN, USA-38163 901-448-3385 (Lab) [EMAIL PROTECTED] --------------------------------- Fussy? Opinionated? Impossible to please? Perfect. Join Yahoo!'s user panel and lay it on us.

Nitin, > With regard to the second question, its true that we have plenty of tools > which can filter out the observations from > other type records post run. OK. So you know how to do the filtering of NONMEM table output. > However, sometimes the output file size is very big for large simulations > that it is difficult > to open in programs like Excel (2003) because of row limitations. There is indeed a limitation in using Excel. But whether the filtering to select observation records is done in NONMEM (which as far as I know it cannot do) or done by a subsequent filtering step (which you say you know how to do) it will not change the Excel size limitation. So once again I dont understand why you want NONMEM to try to do this :-) Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand [EMAIL PROTECTED] tel:+64(9)373-7599x86730 fax:+64(9)373-7090 www.health.auckland.ac.nz/pharmacology/staff/nholford

Hi Nick, I think I should make myself more clear that I was not particulary addressing the Excel size limitation issue. Suppose for example, there are 200 records (50 observations+150 dose records) in a NONMEM data file and if we do 500 simulations, it creates a large output file which cannot be opened in Excel 2003. However, I thought if we could specify a priori in NONMEM control stream (Which you say is not possible) to filter out the dosing records leaving 50 X 500 observation records which can be opened in excel. This issue is not hampering my analysis as I can postprocess the table output. Rather, it was just a curiosity to know if it was possible in NONMEM control stream a priori. Sorry for creating any confusion :-) Regards Nitin Nick Holford <[EMAIL PROTECTED]> wrote: Nitin, > With regard to the second question, its true that we have plenty of tools > which can filter out the observations from > other type records post run. OK. So you know how to do the filtering of NONMEM table output. > However, sometimes the output file size is very big for large simulations > that it is difficult > to open in programs like Excel (2003) because of row limitations. There is indeed a limitation in using Excel. But whether the filtering to select observation records is done in NONMEM (which as far as I know it cannot do) or done by a subsequent filtering step (which you say you know how to do) it will not change the Excel size limitation. So once again I dont understand why you want NONMEM to try to do this :-) Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand [EMAIL PROTECTED] tel:+64(9)373-7599x86730 fax:+64(9)373-7090 www.health.auckland.ac.nz/pharmacology/staff/nholford Nitin Mehrotra, Ph.D Post Doctoral Research Fellow 874 Union Avenue, Suite4.5p/5p Department of Pharmaceutical Sciences University of Tennessee Health Science Center Memphis, TN, USA-38163 901-448-3385 (Lab) [EMAIL PROTECTED] --------------------------------- Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games.