Distribution of CWRESI as a diagnostic

5 messages 4 people Latest: Nov 17, 2022

Distribution of CWRESI as a diagnostic

From: Rik Schoemaker Date: November 15, 2022 technical
Dear all, I have routinely generated a histogram of CWRESI residuals as a model diagnostic, expecting the standard deviation to be close to one, and then commenting 'the SD is close to one as would be expected in a well behaved model'. However, I have now run into a situation where the SD is 0.887 which I consider a lot lower than I'd expect, but I have no clue what that tells me... Any ideas? Should I stop making the plot? ;) Kind regards, Rik Rik Schoemaker, PhD Occams Coöperatie U.A. Malandolaan 10 1187 HE Amstelveen The Netherlands http://www.occams.com +31 20 441 6410 [email protected]<mailto:[email protected]> [image001.png]

RE: Distribution of CWRESI as a diagnostic

From: Buclin Thierry Date: November 15, 2022 technical
Hi Rik Isn’t this simply the expression of an ε-shrinkage phenomenon, bringing the individual predictions close to the observations, possibly due to model overparametrization? Beware of models trying to be too well behaved! Best regards Thierry CHUV University Hospital Centre of the Canton de Vaud Pr Thierry BUCLIN, MD, head physician Service of Clinical Pharmacology Institutes building, Bugnon 17, 1st floor 1011 Lausanne - SWITZERLAND
Quoted reply history
De : [email protected] <[email protected]> De la part de Rik Schoemaker Envoyé : mardi 15 novembre 2022 09:54 À : [email protected] Objet : [NMusers] Distribution of CWRESI as a diagnostic Dear all, I have routinely generated a histogram of CWRESI residuals as a model diagnostic, expecting the standard deviation to be close to one, and then commenting 'the SD is close to one as would be expected in a well behaved model'. However, I have now run into a situation where the SD is 0.887 which I consider a lot lower than I'd expect, but I have no clue what that tells me... Any ideas? Should I stop making the plot? ;) Kind regards, Rik Rik Schoemaker, PhD Occams Coöperatie U.A. Malandolaan 10 1187 HE Amstelveen The Netherlands http://www.occams.com +31 20 441 6410 [email protected]<mailto:[email protected]> [image001.png]

Re: Distribution of CWRESI as a diagnostic

From: Martin Bergstrand Date: November 15, 2022 technical
Dear Rik, Is the mean of the distribution deviating significantly from 0 and/or does the shape of the distribution deviate much from a normal distribution? Ps. I do not regularly assess if my CWRES distributions have a SD=1 so I can’t say if I regularly see a similar behavior. Kind regards, Martin
Quoted reply history
On Tue, 15 Nov 2022 at 16:07, Rik Schoemaker <[email protected]> wrote: > Dear all, > > > > I have routinely generated a histogram of CWRESI residuals as a model > diagnostic, expecting the standard deviation to be close to one, and then > commenting 'the SD is close to one as would be expected in a well behaved > model'. However, I have now run into a situation where the SD is 0.887 > which I consider a lot lower than I'd expect, but I have no clue what that > tells me... > > Any ideas? Should I stop making the plot? ;) > > > > Kind regards, > > > > Rik > > > > > > *Rik Schoemaker, PhD* > > *Occams *Coöperatie U.A. > Malandolaan 10 > https://www.google.com/maps/search/Malandolaan+10+%0D%0A1187+HE+Amstelveen+%0D%0AThe+Netherlands?entry=gmail&source=g > 1187 HE Amstelveen > https://www.google.com/maps/search/Malandolaan+10+%0D%0A1187+HE+Amstelveen+%0D%0AThe+Netherlands?entry=gmail&source=g > The Netherlands > https://www.google.com/maps/search/Malandolaan+10+%0D%0A1187+HE+Amstelveen+%0D%0AThe+Netherlands?entry=gmail&source=g > > www.occams.com > > +31 20 441 6410 > [email protected] > > > > [image: image001.png] > > > > > > > > > -- Martin Bergstrand, Ph.D. Principal Consultant Pharmetheus AB +46(0)709 994 396 [email protected] www.pharmetheus.com +46(0)18 513 328 U-A Science Park, Dag Hammarskjölds v. 36b 752 37 Uppsala, Sweden -- *This communication is confidential and is only intended for the use of the individual or entity to which it is directed. It may contain information that is privileged and exempt from disclosure under applicable law. If you are not the intended recipient please notify us immediately. Please do not copy it or disclose its contents to any other person.* *Any personal data will be processed in accordance with Pharmetheus' privacy notice, available here https://pharmetheus.com/privacy-policy/.** *

Re: Distribution of CWRESI as a diagnostic

From: Andrew Hooker Date: November 15, 2022 technical
Hi Rik, In some investigations that Joakim Nyberg and I did at PAGE a while back, we saw that CWRESI were less likely to be N(0,1) even if the model is correct, compared to CWRES (Nyberg, Bauer, Hooker, PAGE, 2010). So I might recommend creating the plot with CWRES. Otherwise, you could do multiple simulation-evaluations with the model that you have to get an empirical understanding of the expected CWRESI distribution, if the model you are currently using was the model that generated the data. This could be done using the “simeval” tool in NONMEM and “tricking” PsN into thinking that the CWRESI are actually the CWRES by setting (CWRES=CWRESI) in the table file (not tested, but I suspect this will work). On a side note, I think my NMUSERS account is associated with my old “farmbio” email address, so maybe you could forward this to note to NMUSERS? Best regards, Andy Andrew Hooker, Ph.D. Professor of Pharmacometrics Dept. of Pharmacy Uppsala University Box 580, 751 23, Uppsala, Sweden Phone: +46 18 471 4355 https://katalog.uu.se/profile/?id=N4-631 https://katalog.uu.se/profile/?id=N4-631 http://www.farmbio.uu.se/research/researchgroups/pharmacometrics/
Quoted reply history
On 15 Nov 2022, at 09:53, Rik Schoemaker <[email protected]<mailto:[email protected]>> wrote: Dear all, I have routinely generated a histogram of CWRESI residuals as a model diagnostic, expecting the standard deviation to be close to one, and then commenting 'the SD is close to one as would be expected in a well behaved model'. However, I have now run into a situation where the SD is 0.887 which I consider a lot lower than I'd expect, but I have no clue what that tells me... Any ideas? Should I stop making the plot? ;) Kind regards, Rik Rik Schoemaker, PhD Occams Coöperatie U.A. Malandolaan 10 1187 HE Amstelveen The Netherlands http://www.occams.com/ +31 20 441 6410 [email protected]<mailto:[email protected]> <image001.png> När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här: http://www.uu.se/om-uu/dataskydd-personuppgifter/ E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: http://www.uu.se/en/about-uu/data-protection-policy

RE: Distribution of CWRESI as a diagnostic

From: Rik Schoemaker Date: November 17, 2022 technical
Dear Martin and Andy, Thank you very much for your input! It appears that my diagnostic distribution that I’d been using for ages is not such a great diagnostic after all; always good to toss out something of limited use (if it doesn’t spark joy… 😊). Btw, the distribution is nicely symmetrical around almost zero, just a bit too peaked. Kind regards, Rik Rik Schoemaker, PhD Occams Coöperatie U.A. Malandolaan 10 1187 HE Amstelveen The Netherlands http://www.occams.com +31 20 441 6410 [email protected]<mailto:[email protected]> [image001.png]
Quoted reply history
From: Andrew Hooker <[email protected]> Sent: 15 November 2022 18:12 To: Rik Schoemaker <[email protected]> Cc: Nmusers <[email protected]> Subject: Re: [NMusers] Distribution of CWRESI as a diagnostic Hi Rik, In some investigations that Joakim Nyberg and I did at PAGE a while back, we saw that CWRESI were less likely to be N(0,1) even if the model is correct, compared to CWRES (Nyberg, Bauer, Hooker, PAGE, 2010). So I might recommend creating the plot with CWRES. Otherwise, you could do multiple simulation-evaluations with the model that you have to get an empirical understanding of the expected CWRESI distribution, if the model you are currently using was the model that generated the data. This could be done using the “simeval” tool in NONMEM and “tricking” PsN into thinking that the CWRESI are actually the CWRES by setting (CWRES=CWRESI) in the table file (not tested, but I suspect this will work). On a side note, I think my NMUSERS account is associated with my old “farmbio” email address, so maybe you could forward this to note to NMUSERS? Best regards, Andy Andrew Hooker, Ph.D. Professor of Pharmacometrics Dept. of Pharmacy Uppsala University Box 580, 751 23, Uppsala, Sweden Phone: +46 18 471 4355 https://katalog.uu.se/profile/?id=N4-631 https://katalog.uu.se/profile/?id=N4-631 http://www.farmbio.uu.se/research/researchgroups/pharmacometrics/ From: Martin Bergstrand [email protected]<mailto:[email protected]> Sent: 15 November 2022 10:33 To: Rik Schoemaker [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: Re: [NMusers] Distribution of CWRESI as a diagnostic Dear Rik, Is the mean of the distribution deviating significantly from 0 and/or does the shape of the distribution deviate much from a normal distribution? Ps. I do not regularly assess if my CWRES distributions have a SD=1 so I can’t say if I regularly see a similar behavior. Kind regards, Martin On 15 Nov 2022, at 09:53, Rik Schoemaker <[email protected]<mailto:[email protected]>> wrote: Dear all, I have routinely generated a histogram of CWRESI residuals as a model diagnostic, expecting the standard deviation to be close to one, and then commenting 'the SD is close to one as would be expected in a well behaved model'. However, I have now run into a situation where the SD is 0.887 which I consider a lot lower than I'd expect, but I have no clue what that tells me... Any ideas? Should I stop making the plot? ;) Kind regards, Rik Rik Schoemaker, PhD Occams Coöperatie U.A. Malandolaan 10 1187 HE Amstelveen The Netherlands http://www.occams.com/ +31 20 441 6410 [email protected]<mailto:[email protected]> <image001.png> När du har kontakt med oss på Uppsala universitet med e-post så innebär det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör det kan du läsa här: http://www.uu.se/om-uu/dataskydd-personuppgifter/ E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: http://www.uu.se/en/about-uu/data-protection-policy