WRES AND OUTLIER IDENTIFICATION/EXCLUSION

From: "NIYI ADEDOKUN" niyiadedokun@hotmail.com Subject: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Monday, September 25, 2006 3:50 PM Dear All, In conducting population PK analysis I have seen situations when outliers are excluded from a dataset based on weighted residuals>5. Is this justified and are there useful references to back up this practice. Regards, Jo

From: "Bachman, William (MYD)" bachmanw@iconus.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Mon, September 25, 2006 4:24 pm My particular preference is to remove the outliers (based on whatever criterion that you choose) and then, if warranted by the results without the outliers, be able to state that removing the outliers had NO significant influence on the results. Of course, if it does, I would start looking for WHY.

From: Nick Holford n.holford@auckland.ac.nz Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Monday, September 25, 2006 4:56 PM Bill, If removing outliers didnt make any difference to the results then why would you want to remove them? It seems to me that one only removes outliers if it really makes a difference to the results. I agree that you then have to try to explain why the differences exist and why results without outliers are preferable to other results with outliers. This is a game I really try not to play unless outliers can clearly be shown by independent means to be data errors. Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

From: Mark Sale - Next Level Solutions mark@nextlevelsolns.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Mon, 25 Sep 2006 14:38:15 -0700 I'd like to add to Bill's comment: 1. Look at a time vs dv/pred plot for that person, if you have an otherwise reasonable profile, but one strange point, I'd be more comfortable (not that CROs ever mislabel samples, or record times wrong, or mix up tubes) 2. If all samples from a person are strange, it isn't a statistical outlier. Either, that person is different, or something went wrong (wrong dose, sample put in red tube instead of green tube, samples sat on counter over the weekend, assay technician did the dilution wrong etc etc...) You can still delete them all if you want to, but you're sort of obligated to ask if this person is different and why. If we always through away data that isn't consistent with our models, we won't learn very much (think pre 2D6 genotyping studies for tricyclic antidepressants). We only learn when our models DO NOT explain the data. Sort of Learn..Confirm OR Fail to confirm..Learn, to paraphrase LBS. 3. It is very hard to justify deleting more than 3% of your data as statistical outliers. At some point you have to say that the site|lab|data management messed up. Or, this person is strange, for unknown reason, should be included in the estimate of OMEGA, and you should write a grant to study why this is happening. To answer your question about references, not that I'm aware of. But, outlier handleing should be specified in the analysis plan, then you're covered. Mark Mark Sale MD Next Level Solutions, LLC www.NextLevelSolns.com

From: "Bill Bachman" bachmanw@comcast.net Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Mon, 25 Sep 2006 20:26:14 -0400 That's exactly the point! It's better not to remove data!

From: "Andrew Hooker" andrew.hooker@farmbio.uu.se Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 10:53:17 +0200 Hi Jo, If you are using the FOCE method then I dont think that using the weighted residuals as a criterion to exclude outliers is justified. The WRES are based on the FO approximation and can be severely biased even when the data is simulated from the same model you estimate from. Using some other form of metric to identify outliers (such as DV/PRED as suggested by Mark) seems preferable in this case. -Andy Andrew Hooker, Ph.D. Assistant Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Uppsala University Box 591 751 24 Uppsala Sweden Tel: +46 18 471 4355 www.farmbio.uu.se/research.php?avd=5

From: Michael.J.Fossler@GSK.COM Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 08:29:34 -0400 My personal preference is not to exclude any points based on outlier criteria. By doing so, you may be excluding important information. To take an extreme example, if you were modeling consecutive games played in the majors, would you exclude Cal Ripkin? He is clearly an outlier, and yet excluding him from the data-set would bias your model significantly. You would be trading model relevance for a better fit, which is not a good trade-off. Excluding data which are in error should be done, but those data are not outliers, they are errors. Apologies to my European colleagues for the baseball reference. Insert your favorite soccer example above (:^)) Mike ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Michael J. Fossler, Pharm. D., Ph. D., F.C.P. Director Clinical Pharmacokinetics, Modeling & Simulation GlaxoSmithKline (610) 270 - 4797 FAX: (610) 270-5598 Cell: (443) 350-1194 Michael_J_Fossler@gsk.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From: Leonid Gibiansky leonidg@metrumrg.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 09:16:31 -0400 It is known that methods implemented by NONMEM are not robust to the outliers (or data errors). Few bad observations can significantly damage convergence and result in the parameter estimate that reflect data errors rather than system description. Phase 3 data are known to be prone to data errors from non-compliance to incorrect dosing to sample mixed up, you name it. On the other hand, it is not feasible to check several hundred records (say 300-400 patients with 3-4 samples each = 1-2K records) manually/visually. In this situation it could be a valid strategy to exclude 20-30 (say, 1-2-3%) of records based on some diagnostic. WRES is one of the indicators of the mis-fit; large WRES are indicative of outliers/possible data errors. If you do not like WRES, you may try CWRES or any other diagnostic. Often you see the problem immediately: large cloud of reasonable WRES and several points far up or down the scale. Those few suspicious observations can be manually checked for problems: more often that not you may see that the observation is likely to be an error (much higher than the previous one without extra dose, or nearly zero followed by something reasonable, etc.). If so, decision to exclude can be backed by WRES as diagnostic + explanation based on the plots. Leonid

From: Michael.J.Fossler@gsk.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 09:47:39 -0400 Hi Leonid; I have to point out that you are mixing up "outliers" with bad data. The two are not the same. Errors in the data should be tracked down and corrected, but these are not outliers. Outliers are data points that do not fit your "priors" of how the data should look. I still maintain that excluding data based on the fact that they don't fit into your pre-conceived notions of how they should look is a bad idea and should not be done. I'm not sure I agree with you about the relative quality of Phase 3 data. Electronic point-of-care data capture is definitely minimizing these kinds of errors (If not, why are we doing it?). Also, thorough training of nursing staff will go a long way toward minimizing mistakes. I'd argue for more of that and less outlier exclusion; I'm not sure I'd want to go to the FDA and say, "well, you know Phase 3 data, always full of errors, this is the best we can do." I have an idea of what the response would be to that... Mike ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Michael J. Fossler, Pharm. D., Ph. D., F.C.P. Director Clinical Pharmacokinetics, Modeling & Simulation GlaxoSmithKline (610) 270 - 4797 FAX: (610) 270-5598 Cell: (443) 350-1194 Michael_J_Fossler@gsk.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From: "Mats Karlsson" mats.karlsson@farmbio.uu.se Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 15:53:13 +0200 Hi, I believe that sensitivity to outliers/errors, when present, may well come from the assumption that residual error magnitude is the same across all subjects. This assumption is usually proven wrong when challenged and a model that allows variability in sigma between subjects is preferable. This can easily be accomplished with FOCEI and a coding of the type Y=F+EPS(1)*EXP(ETA(.)). Best regards, Mats Mats Karlsson, PhD Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Faculty of Pharmacy Uppsala University Box 591 SE-751 24 Uppsala Sweden phone +46 18 471 4105 fax +46 18 471 4003 mats.karlsson@farmbio.uu.se

From: "NIYI ADEDOKUN" niyiadedokun@hotmail.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 14:19:20 +0000 Many thanks for responses to this posting. Clearly "true" outliers may give important information in model building but suspected "errors" in data can be very detrimental to convergence as well as a succesful covariance step. I have noticed this phenomenon when using FOCEI. Runs with FO or FOCE would converge with the data 'errors' while with FOCEI convergence is only possible when the data errors are excluded. I suspect that sensitivity to data errors is estimation method dependent. More bothersome is the fact that when convergence does occur with FOCEI, parameters could differ by as much as 25%. Since I have more confidence using FOCEI one is left to make a decison on what to do with the data "errors". As for comparing models with and without the 'errors', how can one compare parameters obtained from a converging and a non converging NONMEM run? In addition if one were to use CWRES as a diagnostic to exclude data "errors" what would be the acceptable cut off point? Regards Jo

From: "Xiao, Alan" alan_xiao@merck.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 10:37:08 -0400 I think the key is when to exclude the "outliers". If they are excluded at the beginning, you might lose some important information about your drug (and consequently important findings) as Mike mentioned below. If they are excluded after you got a final model and you want to refine the final model with the exclusion, you then lose much less information - you can not really well characterize the lost part anyway since your model can not capture them either because they are simply data errors or non-representativeness for a new population or other reasons you can imagine, although you still need an interpretation about the "outliers". In this case, your refined final model might be more reliable to data which are representative, but not covering the outliers. Appropriate documentation and interpretation (and implication) are very important. Alan

From: Leonid Gibiansky leonidg@metrumrg.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 11:24:33 -0400 Hi Mike, You never know whether high WRES points are outliers or bad data points: these are not labeled on the tube "outliers". I am not talking about full profile - exclusion; the discussion is whether to exclude some points on the profile that for some unknown reasons cannot be described by the model, and mainly, on how to identify those points without looking on each and every PK profile of hundreds of patients. I tried the idea of using individual sigma values. Results of the quick experiments on the recent data set is below (number of excluded data points was about 2%): High WRES excluded, same sigma THETAs OM SIGMA Value 4.05E+00 2.90E-01 1.11E-02 1.29E-02 8.06E-01 4.82E-02 9.33E-02 SE 1.20E-01 1.21E-02 1.58E-03 4.74E-03 3.41E-02 6.61E-03 8.24E-03 High WRES included, same sigma THETAs OM SIGMA Value 4.37E+00 3.08E-01 1.08E-02 1.75E-02 7.68E-01 4.72E-02 2.04E-01 SE 2.12E-01 1.09E-02 1.11E-03 5.47E-03 3.65E-02 9.27E-03 4.84E-02 High WRES excluded, individual sigma THETAs OM SIGMA OM on SIGMA Value 4.04E+00 2.90E-01 1.11E-02 1.30E-02 8.06E-01 4.82E-02 9.20E-02 2.42E-03 SE 1.18E-01 1.22E-02 1.60E-03 4.82E-03 3.43E-02 6.61E-03 9.14E-03 1.18E-02 High WRES included, individual sigma THETAs OM SIGMA OM on SIGMA Value 4.04E+00 2.93E-01 1.05E-02 1.50E-02 8.21E-01 4.65E-02 8.27E-02 2.00E-01 SE 1.19E-01 1.20E-02 1.38E-03 6.23E-03 3.66E-02 7.49E-03 1.06E-02 7.41E-02 You may see that individualizing residual error is more or less equivalent to exclusion of high WRES: weight of those points is significantly reduced by assigning high residual error to patients with those points. It is more automatic, and do not require data exclusion, great idea, Mats, thanks. Leonid

From: Michael.J.Fossler@gsk.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 11:37:30 -0400 I routinely use the varying sigma residual model and usually see a great improvement in the fit. As Mats points out, it allows one to side-step the issue of data exclusion. Mike ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Michael J. Fossler, Pharm. D., Ph. D., F.C.P. Director Clinical Pharmacokinetics, Modeling & Simulation GlaxoSmithKline (610) 270 - 4797 FAX: (610) 270-5598 Cell: (443) 350-1194 Michael_J_Fossler@gsk.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From: Chuanpu.Hu@sanofi-aventis.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 17:08:06 -0400 I think Mike pointed out an important distinction. As Mats suggested, most of the time outliers are model-dependent; i.e., outliers occurred because the appropriate model wasn't or couldn't be fitted. If the quality of the apparent outliers are questionable, this becomes a robustness/sensitivity issue. This can only be assessed by analyzing data in two ways, including and excluding the outliers. In any case, I see no rationale for simply excluding. Of course, excluding outliers makes the result look better, which can be very appealing... Chuanpu

From: "Mats Karlsson" mats.karlsson@farmbio.uu.se Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 26 Sep 2006 23:44:00 +0200 Hi Chuanpu, You wrote "As Mats suggested, most of the time outliers are model-dependent; i.e., outliers occurred because the appropriate model wasn't or couldn't be fitted." Actually, I don't think I said this. My general impression, which doesn't particularly influence how I handle these cases, is that once we've done a good job on the modeling, the remaining outliers are most likely errors in data, which would occur under any reasonable model. Further, you wrote: "This can only be assessed by analyzing data in two ways, including and excluding the outliers." Although I think that such contrasting analyses can be of great help to understand the impact of anomalous data, my point was the opposite, you don't necessarily have to do these contrasting analyses if you use a model that is more robust to outliers/errors. Best regards, Mats Mats Karlsson, PhD Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Faculty of Pharmacy Uppsala University Box 591 SE-751 24 Uppsala Sweden phone +46 18 471 4105 fax +46 18 471 4003 mats.karlsson@farmbio.uu.se

From: Dr Sima Sadray Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Wed, 27 Sep 2006 Dear All, I think the article below will help. Likelihood-based diagnostics for Influential Individuals in nonlinear mixed effects model selection. S Sadray,E.N. Jonsson and M O Karlsson, Pharmaceutical Research, Vol 16, No.8,1999 Sima Dr Sima Sadray, PharmD, PhD Division of Pharmacokinetics and Biopharmaceutics Department of Pharmaceutics, Faculty of Pharmacy Tehran University of Medical Sciences P.O.Box 14155 /6451 Tehran - IRAN Telfax: +98 21 66959054 Mobile: 0912 2022793 Fax: +98 21 66461178 E-mail: sadrai@sina.tums.ac.ir

From: "Nandy, Partha [PRDUS]" PNandy@prdus.jnj.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Wed, 27 Sep 2006 07:15:30 -0400 Hi All, Thanks for discussing this outlier issue. It is always very difficult to decide whether to keep the outlying data points in or remove those. I have a question though; For Additive error models I think what Mats suggested works great. What is your suggestion for a ADD+PROP Error models? Should one use OMEGAs on both ADD and PROP Errors? Also, bear in mind that if one is using AIC or any other such criteria to select models, one needs to now account for additional parameters.. I am interested in your opinion... Kind Regards, Partha

From: Chuanpu.Hu@sanofi-aventis.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Wed, 27 Sep 2006 09:25:53 -0400 Hi Mats, Sorry for not interpreting you correctly - thank you for clearifying. I agree with your second point, and I generally don't go about deleting outliers. My point was that if one does delete outliers, then the impact should be assessed. In particular, I have seen "assessments" been reported with changes in -2LL, which I think is misleading. Regarding your first point, I think sometimes in practice a "good job" is not done, for various reasons. For example, we know that dosing or sample collection times are inexact in phase III trials. Modeling can be attempted fot this, but not easilly. Could you comment on searching and dealing with outliers in this situation. Best regards, Chuanpu

From: "Kowalski, Ken" Ken.Kowalski@pfizer.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Wed, 27 Sep 2006 15:27:55 -0400 Hi Mats, Nmusers, Here are my two cents on this discussion. 1) For individual data-point outliers wouldn't the 'ETA on Epsilon' residual error model you propose effectively down-weight all of the observations within an individual and not just the suspected outlier data point? I certainly see value in the 'ETA on Epsilon' residual error model when the magnitude of the residual variation does not appear to be the same across all subjects. However, in using this model I would want to assess whether the apparent change in magnitude of the residual variation across subjects is being unduly influenced by a single observation within the subject's data. If it is, I don't think I would use this approach. Of course, it may be a challenge to discrimate statistical outliers vs. misspecification of the residual error model (e.g., non-homogenous variation across subjects) vs. lack of fit of the structural model. Note that a change in residual error model to accommodate outliers rather than excluding outliers is making an implicit set of assumptions so I don't think we can 'side-step' the issue of outlier assessment...we are just trading one set of assumptions for another. 2) Matt Hutmacher and I have been toying with the following idea to address individual data outliers. First, based on a prespecified set of criteria, identify suspected individual data outliers. Second, create a flag variable on the data set to identify these data outliers (i.e., FLAG=1 denotes outlier, FLAG=0 denotes non-outlier). Third, fit a residual error model with different sigmas for outliers and non-outliers. The following code for a constant CV error model might be considered: Y=F*(1+(1-FLAG)*EPS(1)+FLAG*EPS(2)) (If the outliers appear to be independent of F then one might postulate EPS(2) as an additive effect.) With this model sigma2 would be larger than sigma1 effectively down-weighting the suspected outliers without having to formally exclude them (i.e., giving zero weight to them). The degree of down-weighting can be determined from the ratio of the estimates of sigma2 to sigma1 and would increase as the magnitude of outlier deviations increases. One could compare the parameter estimates (thetas and omegas) from this model to that of the usual CV error model, Y=F*(1+EPS(1)), to determine how much leverage these outliers collectively have on the estimation. Any thoughts on this approach? We don't have any direct experience in applying this approach so if anyone would like to try it and report back their experiences we would certainly be interested in hearing about it. 3) For detecting individual data-point outliers (as opposed to outlying subjects) wouldn't the IWRES be a better diagnostic than WRES or CWRES? It would seem to better fit with the sentiment that when assessing individual data point outliers, they should be evaluated in context with the other observations for that individual, presumably with respect to their deviations from the IPRED. 4) Outlier assessment is a very contextual thing. It is nearly impossible to be completely objective in this assessment but at the same time we should be systematic and use sound reasoning in evaluating outliers and the actions we take. While we need to be cautious when considering the impact of exclusion or down-weighting individual outliers we also shouldn't take the position that we should never exclude them. These outliers can unduly inflate the variance components and mask our ability to detect important determinants (covariate effects) of the PK and PD responses. We need to rigorously evaluate the adequacy of our models with various diagnostic plots and rule out (whenever possible) various forms of model (structural and statistical) misspecification before proposing to exclude outliers. The totality of our diagnostics should help inform our decision on the models we postulate and any actions (including no action) we take regarding outliers. Ken

From: "Mats Karlsson" mats.karlsson@farmbio.uu.se Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Wed, 27 Sep 2006 23:54:18 +0200 Ken, Regarding your comments: 1) I agree. According to the model I suggested, a single outlying data point would mean that the entire information content of that individual would be considered less than without that outlier. Of course this model makes assumptions too, even if it relaxes the assumption of everyone having the same residual variability. It still makes the assumption that residual error distribution is a (transformation of) normal distribution. 2) Maybe the idea has merits. Trying it and showing that the extra subjectivity and effort does pay off in terms of increased parameter precision is however something that I think needs to be shown. Also, with your approach I would think that even when you do identify the outliers correctly, the assumption of a normally distributed random error for the outliers is usually not appropriate. In my experience that is not what outliers/errors look like. E.g. often some observations are far too high (sampling in the wrong arm) or far too low (didn't take the dose), but rarely do the two equate to form a nice normal. Further, I'm not sure what you mean by "prespecified criteria". This could be tricky as outliers are usually not easy to foresee. Your suggestion seems to imply that these are identified before you fit a model to the data and then it is even harder to predict which are outliers. Last, it is not uncommon that one can see that one out of two data points are an outlier, but difficult to determine which of the two it is. 3) I tend to agree, but IWRES is not a panacea either. If data are sparse (compared to the number of parameters and especially etas), IWRES can be quite misleading due to overfit. 4) Your usual good advice that I would not want to disagree with. In relation to this, one of my former co-workers (Dr Sima Sadrai) reminded me in a mail that I think was intended for nmusers (copied below), that there may be some further help in inspecting the individual contribution to the likelihood. The idea is to investigate whether some individuals are driving or masking any model selection. The main idea was not in relation to errors/outlying data points, but maybe it has some merits there too. Best regards, Mats Mats Karlsson, PhD Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Faculty of Pharmacy Uppsala University Box 591 SE-751 24 Uppsala Sweden phone +46 18 471 4105 fax +46 18 471 4003 mats.karlsson@farmbio.uu.se

From: "Elassaiss - Schaap, J. (Jeroen)" jeroen.elassaiss@organon.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Thu, 28 Sep 2006 08:18:30 +0200 Hi Ken, To my opinion your idea of flagging with an extra epsilon is a first step towards complete iterative weighting as demonstrated by Jan Freijer, see http://www.page-meeting.org/page/page2005/PAGE2005P76.pdf (I guess he won't chime in himself). His implementation has two pros: i) it is unsupervised and ii) it is gradual. He specified the example of errors in dosing and sampling time, but the approach seems general and therefore also applicable to other causes of outliers. I have used a similar approach during my PhD but in another field with smoothing rather than fitting, i.e. adaptive smoothing. It worked really well in removal of (electronic) artifacts observed in noisy, densily sampled time series and was simple to implement (iteration on a linear regression). Best regards, Jeroen J. Elassaiss-Schaap Scientist PK/PD Organon NV PO Box 20, 5340 BH Oss, Netherlands Phone: + 31 412 66 9320 Fax: + 31 412 66 2506 e-mail: jeroen.elassaiss@organon.com

From: "Kowalski, Ken" Ken.Kowalski@pfizer.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Thu, 28 Sep 2006 17:08:59 -0400 Mats, We appear to be in good agreement on all points. Thank you for your kind words regarding (4) and info on Dr. Sadray's (et al.) paper...I will certainly take a look at it. I just have a follow up with regards to your responses to (2). 2) Certainly work needs to be done to evaluate whether this approach indeed has merit. I agree there is no reason necessarily to believe outliers are normal, however, we most likely will lack suitable power to assess the distribution of these outlying data. There is precedence to consider a mixture model of normal distributions, referred to in the statistical literature as a contaminated normal distribution where Y is distributed as (1-p)N(mu,sigma) + (p)N(mu,k(sigma)) where p represents the fraction of outliers and k is the scale parameter for the increased variation in the outliers (see Barnett and Lewis, Outliers in Statistical Data, Wiley, 1978, pp 31-33, 127-130). We propose a two-stage approach to this contanimated normal distribution by first estimating p by use of a prespecified outlier criteria and fixing this through the use of the FLAG variable. In the second stage we estimate k which is the ratio of sigma2 to sigma1. The outlier criteria, which would ideally be specified in the analysis plan before starting the model development, might be something like "flag all data points as potential outliers for further evaluation where abs(IWRES)>5" (perhaps a reasonable criteria with dense data). Of course, we could look at a full likelihood mixture model approach were p and k are simultaneously estimated. There are other contaminated normal mixture models that allow for asymmetry (a shift in mu as well as a scale increase in sigma) and of course mixtures of different distributions between non-outliers and outliers. Whether we have enough power to discern between various contaminated distributions and how well they may perform in the context of PK/PD is certainly an area that could benefit from some research. Kind regards, Ken

From: "Mats Karlsson" mats.karlsson@farmbio.uu.se Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Fri, 29 Sep 2006 16:45:56 +0200 Hi Ken, Good luck with the evaluations. When you write: " Of course, we could look at a full likelihood mixture model approach were p and k are simultaneously estimated.", do you know a software that could do that? NONMEM would not be able to handle it. Best regards, Mats Mats Karlsson, PhD Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Faculty of Pharmacy Uppsala University Box 591 SE-751 24 Uppsala Sweden phone +46 18 471 4105 fax +46 18 471 4003 mats.karlsson@farmbio.uu.se

From: "Kowalski, Ken" Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Fri, 29 Sep 2006 12:17:45 -0400 Mats, It can be done in NONMEM...credit goes to Matt Hutmacher for figuring this out. You need to use the LIKELIHOOD or -2LL option. Here is an example of a $PRED code segment that Matt Hutmacher prepared and tested: $PRED MU=THETA(1)+ETA(1) MP=THETA(2) SIG1=THETA(3) SIG2=THETA(4) IW1=(DV-MU)/SIG1 IW2=(DV-MU)/SIG2 L1=-0.5*LOG(2*3.14159265)-LOG(SIG1)-0.5*(IW1**2) L2=-0.5*LOG(2*3.14159265)-LOG(SIG2)-0.5*(IW2**2) L=(1-MP)*EXP(L1)+MP*EXP(L2) Y=-2*LOG(L) Note that MU can be replaced with a more complex PK/PD model, MP is the mixing probability, SIG1 is the sigma for non-outliers, and SIG2 is the sigma for outliers. Regards, Ken

From: "Mats Karlsson" mats.karlsson@farmbio.uu.se Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Sat, 30 Sep 2006 17:38:50 +0200 Ken, Nice. I guess it does not come without a price as the same mixture model is applied to subjects with and without outliers alike. If one were to take this estimation route, estimating a (interindividual) mixture model for the mixing component would be a way to address this: $MIX P(1)=THETA(5) ;Proportion of subjects with outliers P(2)=1-P(1) ;Proportion of subjects without outliers $PRED MU=THETA(1)+ETA(1) MP=THETA(2) ;MP for subjects with outliers IF(MIXNUM.EQ.2) MP=0 ;MP for subjects without outliers SIG1=THETA(3) SIG2=THETA(4) IW1=(DV-MU)/SIG1 IW2=(DV-MU)/SIG2 L1=-0.5*LOG(2*3.14159265)-LOG(SIG1)-0.5*(IW1**2) L2=-0.5*LOG(2*3.14159265)-LOG(SIG2)-0.5*(IW2**2) L=(1-MP)*EXP(L1)+MP*EXP(L2) Y=-2*LOG(L) If you have really contaminated data maybe you in addition want to add a (logit-transformed) ETA on MP for subjects with outliers. Best regards, Mats Mats Karlsson, PhD Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Faculty of Pharmacy Uppsala University Box 591 SE-751 24 Uppsala Sweden phone +46 18 471 4105 fax +46 18 471 4003 mats.karlsson@farmbio.uu.se

From: Nick Holford n.holford@auckland.ac.nz Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Mon, 02 Oct 2006 12:59:33 +1300 Mats, Ken, I must be missing some subtle issue here -- why do you think it is necessary to code this using -2LL? Why not code it like this? $THETA (0,0.1,1) ; P prob of being an outlier (0,1,) ; SD of additive residual error (0,1,) ; K fractional difference in SD in outlier population $OMEGA 0.5 ; between subject variability in MYPRED $SIGMA 1 FIX ; unit random effect $MIX NSPOP=2 P(1)=THETA(1) ; P prob of being an outlier P(2)=1-P(1) $PRED ;MYPRED= ...any PKPD model you like with random effects e.g. MYPRED=DOSE/V*EXP(-CL/V*TIME)+ETA(1) ;SD= ... any residual error model you want expressed with THETAs e.g SD=THETA(2) ; additive residual error SD IF (MIXNUM.EQ.1) THEN ; outlier KOUT=THETA(3) ELSE KOUT=1 ENDIF Y=MYPRED+KOUT*SD*EPS(1) Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

From: "Mats Karlsson" mats.karlsson@farmbio.uu.se Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Mon, 2 Oct 2006 08:25:22 +0200 Hi Nick, The intention is to have the mixture on the individual *observation* level, not on (or not only on) the individual subject level. Best regards, Mats Mats Karlsson, PhD Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Faculty of Pharmacy Uppsala University Box 591 SE-751 24 Uppsala Sweden phone +46 18 471 4105 fax +46 18 471 4003 mats.karlsson@farmbio.uu.se

From: Nick Holford n.holford@auckland.ac.nz Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Mon, 02 Oct 2006 22:55:36 +1300 Mats, So you are using $MIX to switch between outliers and non-outlier type subjects? The mixing probability of an outlier is P (Ken's nomenclature). If its an outlier type subject then the -2LL code returns a fractional residual error contribution for an outlier (SIG2) or non-outlier observation (SIG1) depending on an observation level mixing fraction (MP). If its a non-outlier type subject then the residual error is determined by SIG1 alone (MP=0). SIG2/SIG1 is K (Ken's nomenclature). >>> > L1=-0.5*LOG(2*3.14159265)-LOG(SIG1)-0.5*(IW1**2) >>> > L2=-0.5*LOG(2*3.14159265)-LOG(SIG2)-0.5*(IW2**2) >>> > L=(1-MP)*EXP(L1)+MP*EXP(L2) >>> > Y=-2*LOG(L) PS The -0.5*LOG(2*3.14159265) is not really needed (because its a constant that doesn't change the minimum of -2LL) and indeed is inconsistent with the way that NONMEM typically calculates its objective function. -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

From: Mats Karlsson [mailto:mats.karlsson@farmbio.uu.se] Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Monday, October 02, 2006 5:08 PM Hi Ken, Allowing varying residual variability between subjects provide you with an estimate of which individuals' data for which the model doesn't fit, whether it is through error, outlier or model misspecification. If you use a prespecified criteria, I don't think this will change - your criteria (if it is any good) will still identify observations for which the model doesn't fit. These will represent errors, outliers or data where model misspecification is pronounced. I'm not sure why you think you could, through the somewhat ad hoc procedure of e.g. IWRES>x strike a better balance and only get errors and not model misspecification. Regarding classification of data points as outliers or not, I guess classification is always destroying information. Why would you want to classify? However, I guess that you could get the information to classified by "just" evaluating the OFV for the 2**Ni possibilities, where Ni is the number of observations for subject i. On the practical side, a drawback of the individual observation mixture model is of course that it requires the LAPLACIAN method (unless Matt has pulled off another stunt!). Thanks for the entertaining discussion, but tomorrow morning I'm off for vacation so don't expect any quick reply on further comments. Best regards, Mats Mats Karlsson, PhD Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Faculty of Pharmacy Uppsala University Box 591 SE-751 24 Uppsala Sweden phone +46 18 471 4105 fax +46 18 471 4003 mats.karlsson@farmbio.uu.se

From: Nick Holford n.holford@auckland.ac.nz Subject: Re: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Monday, October 02, 2006 5:59 PM Ken, Thanks for confirming the distinction between outlier subjects (i.e. large ETA) and subjects with outliers (i.e. large EPS). While attempting to contribute to this thread I have realized the importance of distinguishing between 'outlier subjects' and 'subjects with outliers'. The $MIX code I posted earlier attempts to separate subjects with outliers from those without outliers based on the size of EPS (see again below). This is in the spirit of the discussion to identify subjects who do not fit well and therefore may be subjects with outliers regardless of any ETA adjustment. The fractional likelihood method you and Matt H. propose estimates some average fraction of outlier observations. I am still not very comfortable about the fractional likelihood method because it assumes a similar fraction of outlier and non-outlier observations in each subject (which might be addressed by having an ETA on this fraction) but more importantly (for this thread) it doesn't distinguish in a discrete way between outlier and non-outlier observations. I think we need to have something like $MIX that works at the observation level (instead of only at the subject level) because it produces a discrete classification. You refer to a "two stage approach" in your earlier emails but I am afraid I didnt really follow all the details. Here is a specific proposal using NONMEM: Step 1: Use the simple $MIX method (See Step 1 code below) to identify subjects with outliers from subjects without outliers. Save posthoc estimates of the ETA specific parameters for the next step. This step estimates 'P' (your nomenclature) i.e. the proportion of subjects with outliers and also discretely identifies (using MIXEST) which subject belongs to each group (ME1). Step 2: Code the data with ID changing for each observation and use the individual posthoc parameters (FIXED) to make subject specific predictions. Set up a $MIX model to identify the overall proportion of observation level outliers in the total set of observations (See Step 2 code below). You can estimate different residual error model parameters for outlier and non-outlier observations ('K') during this second step but the main outcome would to use MIXEST at the observation level to distinguish outlier observations (ME2). The second step can be used to to test the hypothesis that there are indeed two populations of observations i.e. outliers and non-outliers by fitting with and without $MIX and 'K'. Overall this 2 step procedure is not as elegant as doing it in one step but it could maybe achieve the goal of this thread i.e. to identify outlier observations with a somewhat objective and automatable procedure. The main problem I think is in the posthoc estimates of the parameters in Step 1 which will be somewhat contaminated with the misspecification of the outlier vs nonoutlier residual error. Perhaps it is here that the fractional likelihood method might help. Best wishes, Nick Caution: The following code has not been tested. It is intended primarily to demonstrate the ideas being discused. It might easily contain errors! ************* STEP 1 ****************** $THETA (0,0.1,1) ; P prob of being subject with outlier 1 (0,3,); Pop clearance 2 (0,10,); Pop volume 3 (0,1,) ; SD of additive residual error 4 (0,1,) ; K fractional difference in SD in subjects with outliers 5 $OMEGA 0.5 ; between subject variability in CL 0.5 ; between subject variability in V $SIGMA 1 FIX ; unit random effect $MIX NSPOP=2 P(1)=THETA(1) ; P prob of being subject with outlier P(2)=1-P(1) $PRED IF (NEWIND.EQ.0) OBID=0 ; counter for observations CL=THETA(2)*EXP(ETA(1)) V=THETA(3)*EXP(ETA(2)) MYPRED=DOSE/V*EXP(-CL/V*TIME) ;SD= ... any residual error model you want expressed with THETAs e.g SD=THETA(4) ; additive residual error SD IF (MIXNUM.EQ.1) THEN ; subject with outliers KOUT=THETA(3) ELSE KOUT=1 ENDIF Y=MYPRED+KOUT*SD*EPS(1) IF (MDV.EQ.0) OBID=OBID+1 ; create observation level ID ME1=MIXEST $TABLE ID ME1 OBID TIME DOSE CL V DV MDV NOAPPEND NOPRINT ONEHEADER FILE=step1.fit ************* STEP 2 ****************** $DATA step1.fit IGNORE=@ ;OBID is an observation level ID type of data item ;ICL and IV are the posthoc individual estimates of CL and V from Step 1 $INPUT SID ME1 ID=OBID TIME DOSE ICL IV DV MDV $THETA (0,0.1,1) ; proportion of obs which are outliers 1 (0,1,) ; SD of additive residual error 2 (0,1,) ; K fractional difference in SD in outlier population 3 $OMEGA 0 FIX ; keep NM-TRAN happy (no OMEGAs needed in this step $SIGMA 1 FIX ; unit random effect $MIX NSPOP=2 P(1)=THETA(1) ; proportion of obs which are outliers P(2)=1-P(1) $PRED MYPRED=DOSE/IV*EXP(-ICL/IV*TIME) SD=THETA(2) ; additive residual error SD IF (MIXNUM.EQ.1) THEN ; outlier observation KOUT=THETA(3) ELSE KOUT=1 ENDIF Y=MYPRED+KOUT*SD*EPS(1) ME2=MIXEST $TABLE SID ID ME1 ME2 -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

From: "Kowalski, Ken" Ken.Kowalski@pfizer.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Mon, 2 Oct 2006 16:21:25 -0400 Mats, Nice suggestion. By incorporating an interindividual mixture model for subjects with at least one outlier we can output MIXEST for each individual as a diagnostic. This would allow us to identify which individuals have at least one outlier observation (and which individuals are "clean" of any outliers). Still, within an individual in the outlier population we wouldn't know which observations are being classified as outliers vs non-outliers without further post-processing of the individual observation contributions to the likelihood (don't ask me how to do this :) ). While this is an interesting approach (our original proposal with or without your interindividual mixture model suggestion) it is not clear to me how well this approach will work in practice. I suspect that any misspecification of the structural as well as statistical models may result in the estimation of subpopulations that may not exactly coincide with outlier vs non-outlier populations of observations. I would probably start with the two-stage approach and go to a full likelihood approach if one shows that the subpopulations coincide with the outlier vs non-outlier dichotomy based on a pre-specified criteria for outliers. If the outliers tend to have more positive rather than negative residuals or if they are not randomly distributed about the curve, this might be an indication of model misspecification which may have to be resolved before entertaining this full likelihood outlier approach and what form of contaminated residual error model to employ. Best regards, Ken

From: "Kowalski, Ken" Ken.Kowalski@pfizer.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Mon, 2 Oct 2006 16:31:43 -0400 Nick, Sorry for the confusion. Mats' suggestion to identify subjects with at least one outlier observation (data point) via the $MIX code combined with the residual error mixture model should not be confused with the distinct but separate issue of "outlier subjects". Note that an "outlier subject" whose individual curve deviates substantially from the typical individual curve may not have any individual data point outliers. That is, conditional on the subject's ETA values, the deviations from IPRED may not be extreme (i.e., no extreme IWRES values). We are not dealing with outlier subjects whose curves may represent extreme deviations from PRED (e.g., extreme ETA values) but whose individual observations are reasonably consistent within the individual. Ken

From: "Kowalski, Ken" Ken.Kowalski@pfizer.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 3 Oct 2006 17:31:45 -0400 Mats, Outlier detection requires some criteria for explicit assessment. In practice we often look at residual plots and make a judgement as to whether to classify an observation as an outlier or non-outlier. It is important to make the distinction that this classification is done only for diagnostic purposes. No matter how we choose to deal with outliers, we should be explicit, transparent, and systematic about our procedures for handling data outliers. Moreover, as we stated previously, we need to first rule out to the best of our ability any model misspecification before we take any actions with regards to outliers we detect (e.g., excluding outlier observations or fitting alternative models and down-weighting). We could use a similar approach to classify data outliers based on this full likelihood (residual error mixture model) to what NONMEM does with the MIXEST calculation to classify which subpopulation each subject belongs to. It is purely a post hoc calculation for diagnostic purposes; knowing which subpopulation each observation belongs to is not involved in the estimation. In this case it's just not built into NONMEM so we would have to do some post-processing to perform this post hoc classification. At this point we are certainly not recommending this interesting but untested approach. But it may be promising and worth evaluating with real data and simulation studies. Figuring out the post hoc calculations would certainly help improve the diagnostic value of this approach as well. I trust that you had or are having an enjoyable vacation. Best wishes, Ken

From: "Kowalski, Ken" Ken.Kowalski@pfizer.com Subject: RE: [NMusers] WRES AND OUTLIER IDENTIFICATION/EXCLUSION Date: Tue, 3 Oct 2006 17:38:09 -0400 Nick, You wrote: "I am still not very comfortable about the fractional likelihood method because it assumes a similar fraction of outlier and non-outlier observations in each subject..." The full likelihood approach using an observation-level mixture model (i.e., residual error mixture model) does not make this assumption. The original version without Mats' modification to use $MIX merely estimates the proportion of outlier observations from the total number of observations. With Mats' $MIX code modification, we estimate the proportion of 'subjects with outliers', thus, the mixing proportion in the residual error mixture model is now conditional on the total number of observations in the population of 'subjects with outliers' rather than the total number of observations. There is no constraint that forces the observation-level mixing proportion to be the same within each subject of the 'subjects with outliers' subpopulation. You wrote: "...more importantly (for this thread) it doesn't distinguish in a discrete way between outlier and non-outlier observations." True...but it doesn't mean we couldn't perform some post hoc calculations to classify observations as outlier or non-outlier based on this full likelihood approach. It is analogous to the situation with $MIX and the post hoc calculations that MIXEST performs. If we did not have the MIXEST capabilities built in to NONMEM we would have a harder time with our diagnostic evaluation of subject-level mixture models using the $MIX functionality. Same is true here with the observation-level mixture models. To fully evaluate and advocate this approach would require more work to determine the post-processing calculations that would allow us to classify the observations in the two populations (outliers vs non-outliers) for diagnostic purposes. Note that if we performed these post hoc calculations such that we could classify outliers vs non-outliers at the observation level we would no longer need to use the $MIX code to identify 'subjects with outliers'. The population of 'subjects with outliers' could be determined directly from these post hoc calculations of the observation-level classification of outliers and non-outliers. Kind regards, Ken _______________________________________________________