VPC with Mixture Model

From: [email protected] [mailto:[email protected]] On Behalf Of Satyendra Suryawanshi Sent: Tuesday, April 14, 2009 6:50 PM To: [email protected] Subject: [NMusers] VPC with Mixture Model Dear all, I have a Mixture Model with 2 subpopulation. Now I want to check its prediction. One way to see this is a Visual Predictive Check. My question is, How to perform visual predictive check with mixture model? I will be thankful for your suggestion and references. Best regards Satyendra Suryawanshi, PhD University of Tennessee Health Science Center

> -----Original Message----- > > From: Leonid Gibiansky [ mailto: [email protected] ] Sent: Tuesday, April 14, 2009 9:06 PM > > To: Mats Karlsson > Cc: 'Satyendra Suryawanshi'; [email protected] > Subject: Re: [NMusers] VPC with Mixture Model > > Hi Mats, > > Could you elaborate why you would not stratify based on the subpopulations? It seems perfectly reasonable for me to simulate from the model (including random assignment of subpopulations), and then compare "apples to apples": observed subpopulation versus simulated subpopulations. In your example of 5% poor metabolizers, I would plot observed poor metabolizers (as assigned by the model) versus simulated poor metabolizers (as simulated from the model). Indeed, poor metabolizers assignment would be defined by the model, so this VPC will be conditioned on the model posthoc EST prediction, but the remaining parts of the model would be tested by this procedure. If the model is good, VPC should provide good results. It is unclear to me how sensitive this procedure is to model misspecification (in general, I think VPC is less sensitive to model misspecification than other model diagnostics) > > Thanks > Leonid > > -------------------------------------- > Leonid Gibiansky, Ph.D. > President, QuantPharm LLC > web: www.quantpharm.com > e-mail: LGibiansky at quantpharm.com > tel: (301) 767 5566 > > Mats Karlsson wrote: > > > Dear Satyendra, > > > > Interesting question. I don't think there is much written about this, but I may be wrong. What I would not do is to try to stratify based on estimated subpopulation allocation ("EST"). Rather I would use the same VPCs as if there had been no mixture model. Possibly what you could do is be more careful in your choice of prediction intervals to display. For example, if you have a subpopulation of poor metabolizers of about 5%, displaying only median and interquartile range PIs may not be a good idea. > > > > Best regards, > > > > Mats > > > > Mats Karlsson, PhD > > > > Professor of Pharmacometrics > > > > Dept of Pharmaceutical Biosciences > > > > Uppsala University > > > > Box 591 > > > > 751 24 Uppsala Sweden > > > > phone: +46 18 4714105 > > > > fax: +46 18 471 4003 > > > > *From:* [email protected] [ mailto: [email protected] ] *On Behalf Of *Satyendra Suryawanshi > > > > *Sent:* Tuesday, April 14, 2009 6:50 PM > > *To:* [email protected] > > *Subject:* [NMusers] VPC with Mixture Model > > > > Dear all, > > > > I have a Mixture Model with 2 subpopulation. Now I want to check its prediction. One way to see this is a Visual Predictive Check. My question is, How to perform visual predictive check with mixture model? > > > > I will be thankful for your suggestion and references. > > > > Best regards > > > > Satyendra Suryawanshi, PhD > > > > University of Tennessee Health Science Center -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand [email protected] tel:+64(9)923-6730 fax:+64(9)373-7090 mobile: +33 64 271-6369 (Apr 6-Jul 17 2009) http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Nick Holford Sent: Wednesday, April 15, 2009 8:33 AM To: [email protected] Subject: Re: [NMusers] VPC with Mixture Model Hi, I realized after I sent the comment below about mixture model **simulations** that Mats was no doubt referring to the problem of separating the two sub-populations in the **observed** data using MIXEST. This will of course be problematic for generating prediction percentiles for the 2 populations as Mats pointed out when there is shrinkage of the EBEs which lead to the 'decision' about MIXEST. This is an example of why one must be cautious interpreting a VPC when the observed data is not 'correct' for comparison with the simulation predictions. Another example is when there is data missing in the observations which is not missing completely at random (e.g. dropouts due to adverse effects). Adding a dropout model to the VPC simulation can help sometimes in 'correcting' the simulation so that it matches the observed data better. Mats, can you suggest a way to try to correct for the Bayesian shrinkage problem for obtaining MIXEST? Nick Nick Holford wrote: > Mats, > > A VPC relies on simulation alone - there is no estimation step. > Presumably if the population estimate of the % of poor metabolizers is > 5% then NONMEM will simulate 5% of the population as a poor > metaboliser. There is no Bayesian 'posthoc' step hidden in the way > that NONMEM does simulations is there? I had assumed (but have never > checked) that if one tabulates the value of MIXEST obtained when using > $SIM that the proportion of MIXEST values would not be biased compared > to the population value. > > Nck > > Mats Karlsson wrote: >> Hi Leonid, >> >> I would not do it for just the reason you mention. I would not want to >> condition my VPC on the model results. Especially as we know that the >> subpopulation assignment suffer from the same problems as other >> empirical >> Bayes estimates. "Shrinkage" when it comes to subpopulation >> assignment will >> have the consequence that the larger of (two) subpopulations having a >> higher >> fraction of POSTHOC assignments than the Pmix estimate for that >> subpopulation. This is expected and I have often seen it. So you may >> well >> have a situation when the population estimate of poor metabolizers is >> 5%, >> but only 2% are allocated to this subpopulation by the EBE step. >> >> Best regards, >> Mats >> >> >> Mats Karlsson, PhD >> Professor of Pharmacometrics >> Dept of Pharmaceutical Biosciences >> Uppsala University >> Box 591 >> 751 24 Uppsala Sweden >> phone: +46 18 4714105 >> fax: +46 18 471 4003 >> >> >> -----Original Message----- >> From: Leonid Gibiansky [mailto:[email protected]] Sent: >> Tuesday, April 14, 2009 9:06 PM >> To: Mats Karlsson >> Cc: 'Satyendra Suryawanshi'; [email protected] >> Subject: Re: [NMusers] VPC with Mixture Model >> >> Hi Mats, >> >> Could you elaborate why you would not stratify based on the >> subpopulations? It seems perfectly reasonable for me to simulate from >> the model (including random assignment of subpopulations), and then >> compare "apples to apples": observed subpopulation versus simulated >> subpopulations. In your example of 5% poor metabolizers, I would plot >> observed poor metabolizers (as assigned by the model) versus >> simulated poor metabolizers (as simulated from the model). Indeed, >> poor metabolizers assignment would be defined by the model, so this >> VPC will be conditioned on the model posthoc EST prediction, but the >> remaining parts of the model would be tested by this procedure. If >> the model is good, VPC should provide good results. It is unclear to >> me how sensitive this procedure is to model misspecification (in >> general, I think VPC is less sensitive to model misspecification than >> other model diagnostics) >> >> Thanks >> Leonid >> >> -------------------------------------- >> Leonid Gibiansky, Ph.D. >> President, QuantPharm LLC >> web: www.quantpharm.com >> e-mail: LGibiansky at quantpharm.com >> tel: (301) 767 5566 >> >> >> >> >> Mats Karlsson wrote: >> >>> Dear Satyendra, >>> >>> >>> >>> Interesting question. I don't think there is much written about >>> this, but I may be wrong. What I would not do is to try to stratify >>> based on estimated subpopulation allocation ("EST"). Rather I would >>> use the same VPCs as if there had been no mixture model. Possibly >>> what you could do is be more careful in your choice of prediction >>> intervals to display. For example, if you have a subpopulation of >>> poor metabolizers of about 5%, displaying only median and >>> interquartile range PIs may not be a good idea. >>> >>> >>> >>> Best regards, >>> >>> Mats >>> >>> >>> >>> Mats Karlsson, PhD >>> >>> Professor of Pharmacometrics >>> >>> Dept of Pharmaceutical Biosciences >>> >>> Uppsala University >>> >>> Box 591 >>> >>> 751 24 Uppsala Sweden >>> >>> phone: +46 18 4714105 >>> >>> fax: +46 18 471 4003 >>> >>> >>> >>> *From:* [email protected] >>> [mailto:[email protected]] *On Behalf Of *Satyendra >>> Suryawanshi >>> *Sent:* Tuesday, April 14, 2009 6:50 PM >>> *To:* [email protected] >>> *Subject:* [NMusers] VPC with Mixture Model >>> >>> >>> >>> Dear all, >>> >>> I have a Mixture Model with 2 subpopulation. Now I want to check its >>> prediction. One way to see this is a Visual Predictive Check. My >>> question is, How to perform visual predictive check with mixture model? >>> I will be thankful for your suggestion and references. >>> >>> >>> >>> Best regards >>> >>> >>> >>> Satyendra Suryawanshi, PhD >>> >>> University of Tennessee Health Science Center >>> >>> >>> >>> >> >> > -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand [email protected] tel:+64(9)923-6730 fax:+64(9)373-7090 mobile: +33 64 271-6369 (Apr 6-Jul 17 2009) http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford