Hi, I am in the process of developing a PK/PD model and have a naive
question regarding model building.
I currently do not have all the PD data and more data would be available in
the future. For the current data set, I have tried models say A to D.
Now model A (cell kill) and B (cell kill +transduction) converges using
FOCE (no CV% but I am willing to live with that) but from the RES and WRES
plots we can clearly see that there is some bias. The fit is OK but not
great. The ofv values are around 400.
Now models B (cell cycle specific kill), C (cell + precursor cell kill
+transduction) and D (cell cycle specific + indirect response model) do not
converge using FO or FOCE methods but when I look at the fits from the
terminated runs, the fits are much better than those obtained with Model A,
and there seems very little bias. Also the ofv values are between 160-250.
So my question is whether the fits, RES, WRES plots and the ofv values have
meaning even when the minimization terminates.
Regards
Neil
--
Indranil Bhattacharya
Model building
Dear Neil,
Two questions:
1) Could you please give us some information on the number of thetas and omegas,
number of cellular sub-populations with different drug susceptibility, and typical
rates of growth and killing (e.g. n Log10 killing per hour or day) in your model?
2) Could you please let us know what your observed variables are? (i.e. are you just
measuring total cell concentration or are you specifically measuring resistant cells)
Especially for the FOCE method, too many omegas on initial conditions of resistant
subpopulation(s) that only show up later after the susceptible cells have been killed
may cause the FOCE method to not finish successfully. I am not overly worried about
this and tend to go for the better curve fits. Two solutions would be to
1) use MC-PEM instead of FOCE or 2) to reduce the number of OMEGAs.
At least for in vitro studies, the run to run variability may be small.
Best wishes
Juergen
Quoted reply history
From: owner-nmusers
Behalf Of Indranil Bhattacharya
Sent: Thursday, December 03, 2009 10:41 AM
To: nmusers
Subject: [NMusers] Model building
Hi, I am in the process of developing a PK/PD model and have a naive question regarding model building.
I currently do not have all the PD data and more data would be available in the future. For the current data set, I have tried models say A to D.
Now model A (cell kill) and B (cell kill +transduction) converges using FOCE (no CV% but I am willing to live with that) but from the RES and WRES plots we can clearly see that there is some bias. The fit is OK but not great. The ofv values are around 400.
Now models B (cell cycle specific kill), C (cell + precursor cell kill +transduction) and D (cell cycle specific + indirect response model) do not converge using FO or FOCE methods but when I look at the fits from the terminated runs, the fits are much better than those obtained with Model A, and there seems very little bias. Also the ofv values are between 160-250.
So my question is whether the fits, RES, WRES plots and the ofv values have meaning even when the minimization terminates.
Regards
Neil
--
Indranil Bhattacharya
Dear Neil,
Two questions:
1) Could you please give us some information on the number of thetas and omegas,
number of cellular sub-populations with different drug susceptibility, and
typical
rates of growth and killing (e.g. n Log10 killing per hour or day) in your
model?
2) Could you please let us know what your observed variables are? (i.e. are you
just
measuring total cell concentration or are you specifically measuring resistant
cells)
Especially for the FOCE method, too many omegas on initial conditions of
resistant
subpopulation(s) that only show up later after the susceptible cells have been
killed
may cause the FOCE method to not finish successfully. I am not overly worried
about
this and tend to go for the better curve fits. Two solutions would be to
1) use MC-PEM instead of FOCE or 2) to reduce the number of OMEGAs.
At least for in vitro studies, the run to run variability may be small.
Best wishes
Juergen
Quoted reply history
From: [email protected] [mailto:[email protected]] On
Behalf Of Indranil Bhattacharya
Sent: Thursday, December 03, 2009 10:41 AM
To: [email protected]
Subject: [NMusers] Model building
Hi, I am in the process of developing a PK/PD model and have a naive question
regarding model building.
I currently do not have all the PD data and more data would be available in the
future. For the current data set, I have tried models say A to D.
Now model A (cell kill) and B (cell kill +transduction) converges using FOCE
(no CV% but I am willing to live with that) but from the RES and WRES plots we
can clearly see that there is some bias. The fit is OK but not great. The ofv
values are around 400.
Now models B (cell cycle specific kill), C (cell + precursor cell kill
+transduction) and D (cell cycle specific + indirect response model) do not
converge using FO or FOCE methods but when I look at the fits from the
terminated runs, the fits are much better than those obtained with Model A, and
there seems very little bias. Also the ofv values are between 160-250.
So my question is whether the fits, RES, WRES plots and the ofv values have
meaning even when the minimization terminates.
Regards
Neil
--
Indranil Bhattacharya
Hi Neil,
You ask:
"So my question is whether the fits, RES, WRES plots and the ofv values
have meaning even when the minimization terminates"
The answer: you bet they matter! Residual plots are the most informative
output NONMEM gives you. They should guide you when you determine the
basic structure of your model that is supported by the data. Successful
termination, covariance step, standard errors, Eigen values, messages,
warnings... are just icing on the cake vis-a-vis the residual plots.
These are my two pennies worth of advice,
Joachim
Joachim Grevel
Senior Pharmacometrician
_____________________________________________________________________
AstraZeneca R&D Charnwood
Clinical Pharmacology & DMPK, Charnwood
Bakewell Road, Loughborough, Leics., LE11 5RH, England
Tel +44 (0) 1509 644035 Fax +44 (0) 1509 645576 Mobile +44 (0) 7920
285905
[email protected]
P Please consider the environment before printing this e-mail
Quoted reply history
From: [email protected] [mailto:[email protected]]
On Behalf Of Indranil Bhattacharya
Sent: 03 December 2009 15:41
To: [email protected]
Subject: [NMusers] Model building
Hi, I am in the process of developing a PK/PD model and have a naive
question regarding model building.
I currently do not have all the PD data and more data would be available
in the future. For the current data set, I have tried models say A to D.
Now model A (cell kill) and B (cell kill +transduction) converges using
FOCE (no CV% but I am willing to live with that) but from the RES and
WRES plots we can clearly see that there is some bias. The fit is OK but
not great. The ofv values are around 400.
Now models B (cell cycle specific kill), C (cell + precursor cell kill
+transduction) and D (cell cycle specific + indirect response model) do
not converge using FO or FOCE methods but when I look at the fits from
the terminated runs, the fits are much better than those obtained with
Model A, and there seems very little bias. Also the ofv values are
between 160-250.
So my question is whether the fits, RES, WRES plots and the ofv values
have meaning even when the minimization terminates.
Regards
Neil
--
Indranil Bhattacharya
--------------------------------------------------------------------------
AstraZeneca UK Limited is a company incorporated in England and Wales with
registered number: 03674842 and a registered office at 15 Stanhope Gate, London
W1K 1LN.
Confidentiality Notice: This message is private and may contain confidential,
proprietary and legally privileged information. If you have received this
message in error, please notify us and remove it from your system and note that
you must not copy, distribute or take any action in reliance on it. Any
unauthorised use or disclosure of the contents of this message is not permitted
and may be unlawful.
Disclaimer: Email messages may be subject to delays, interception, non-delivery
and unauthorised alterations. Therefore, information expressed in this message
is not given or endorsed by AstraZeneca UK Limited unless otherwise notified by
an authorised representative independent of this message. No contractual
relationship is created by this message by any person unless specifically
indicated by agreement in writing other than email.
Monitoring: AstraZeneca UK Limited may monitor email traffic data and content
for the purposes of the prevention and detection of crime, ensuring the
security of our computer systems and checking Compliance with our Code of
Conduct and Policies.
Thanks Joachim, that is what I thought. I wanted to be sure that I invest
time building the right model and not just some model which works
(converges) but is biased.
Neil
Quoted reply history
On Fri, Dec 4, 2009 at 3:31 AM, Grevel, Joachim <
[email protected]> wrote:
> Hi Neil,
>
>
>
> You ask:
>
>
>
> “So my question is whether the fits, RES, WRES plots and the ofv values
> have meaning even when the minimization terminates”
>
>
>
> The answer: you bet they matter! Residual plots are the most informative
> output NONMEM gives you. They should guide you when you determine the basic
> structure of your model that is supported by the data. Successful
> termination, covariance step, standard errors, Eigen values, messages,
> warnings... are just icing on the cake vis-a-vis the residual plots.
>
>
>
> These are my two pennies worth of advice,
>
>
>
> Joachim
>
>
>
>
>
> *Joachim Grevel *
>
> Senior Pharmacometrician
>
> _____________________________________________________________________
>
> *AstraZeneca R&D Charnwood*
>
> Clinical Pharmacology & DMPK, Charnwood
>
> Bakewell Road, Loughborough, Leics., LE11 5RH, England
>
> Tel +44 (0) 1509 644035 Fax +44 (0) 1509 645576 Mobile +44 (0) 7920
> 285905
>
> *[email protected]*
>
> * *
>
> P Please consider the environment before printing this e-mail
>
>
>
>
>
>
>
> ------------------------------
>
> AstraZeneca UK Limited is a company incorporated in England and Wales with
> registered number: 03674842 and a registered office at 15 Stanhope Gate,
> London W1K 1LN.
>
> *Confidentiality Notice: *This message is private and may contain
> confidential, proprietary and legally privileged information. If you have
> received this message in error, please notify us and remove it from your
> system and note that you must not copy, distribute or take any action in
> reliance on it. Any unauthorised use or disclosure of the contents of this
> message is not permitted and may be unlawful.
>
> *Disclaimer:* Email messages may be subject to delays, interception,
> non-delivery and unauthorised alterations. Therefore, information expressed
> in this message is not given or endorsed by AstraZeneca UK Limited unless
> otherwise notified by an authorised representative independent of this
> message. No contractual relationship is created by this message by any
> person unless specifically indicated by agreement in writing other than
> email.
>
> *Monitoring: *AstraZeneca UK Limited may monitor email traffic data and
> content for the purposes of the prevention and detection of crime, ensuring
> the security of our computer systems and checking compliance with our Code
> of Conduct and policies.
>
> *From:* [email protected] [mailto:[email protected]]
> *On Behalf Of *Indranil Bhattacharya
> *Sent:* 03 December 2009 15:41
>
> *To:* [email protected]
> *Subject:* [NMusers] Model building
>
>
>
> Hi, I am in the process of developing a PK/PD model and have a naive
> question regarding model building.
>
> I currently do not have all the PD data and more data would be available in
> the future. For the current data set, I have tried models say A to D.
>
> Now model A (cell kill) and B (cell kill +transduction) converges using
> FOCE (no CV% but I am willing to live with that) but from the RES and WRES
> plots we can clearly see that there is some bias. The fit is OK but not
> great. The ofv values are around 400.
>
> Now models B (cell cycle specific kill), C (cell + precursor cell kill
> +transduction) and D (cell cycle specific + indirect response model) do not
> converge using FO or FOCE methods but when I look at the fits from the
> terminated runs, the fits are much better than those obtained with Model A,
> and there seems very little bias. Also the ofv values are between 160-250.
>
>
>
> So my question is whether the fits, RES, WRES plots and the ofv values have
> meaning even when the minimization terminates.
>
>
>
> Regards
>
>
>
> Neil
>
> --
> Indranil Bhattacharya
>
>
--
Indranil Bhattacharya
Indranil Bhattacharya wrote:
“So my question is whether the fits, RES, WRES plots and the ofv values have meaning even when the minimization terminates”
I do not agree with Joachim that RES, WRES are useful. IMHO these have almost no diagnostic merit except for the most extreme cases of a bad model. Simulation based diagnostics (VPC, SPC) have better diagnostic properties and are being actively evaluated by many groups interested in modelling methodology.
See Karlsson MO, Savic RM. Diagnosing model diagnostics. Clin Pharmacol Ther. 2007 Jul;82(1):17-20. for a demonstration of the problems.
Nick
> Thanks Joachim, that is what I thought. I wanted to be sure that I invest time building the right model and not just some model which works (converges) but is biased. Neil On Fri, Dec 4, 2009 at 3:31 AM, Grevel, Joachim < [email protected] < mailto: [email protected] >> wrote:
>
> Hi Neil,
>
> You ask:
>
> “So my question is whether the fits, RES, WRES plots and the ofv
> values have meaning even when the minimization terminates”
>
> The answer: you bet they matter! Residual plots are the most
> informative output NONMEM gives you. They should guide you when
> you determine the basic structure of your model that is supported
> by the data. Successful termination, covariance step, standard
> errors, Eigen values, messages, warnings... are just icing on the
> cake vis-a-vis the residual plots.
>
> These are my two pennies worth of advice,
>
> Joachim
>
> *Joachim Grevel *
>
> Senior Pharmacometrician
>
> _____________________________________________________________________
>
> *AstraZeneca R&D Charnwood*
>
> Clinical Pharmacology & DMPK, Charnwood
>
> Bakewell Road, Loughborough, Leics., LE11 5RH, England
>
> Tel +44 (0) 1509 644035 Fax +44 (0) 1509 645576 Mobile +44 (0)
> 7920 285905
>
> [email protected]
> <mailto:[email protected]>_
>
> _ _
>
> P Please consider the environment before printing this e-mail
>
> ------------------------------------------------------------------------
>
> AstraZeneca UK Limited is a company incorporated in England and
> Wales with registered number: 03674842 and a registered office at
> 15 Stanhope Gate, London W1K 1LN.
>
> *Confidentiality Notice: *This message is private and may contain
> confidential, proprietary and legally privileged information. If
> you have received this message in error, please notify us and
> remove it from your system and note that you must not copy,
> distribute or take any action in reliance on it. Any unauthorised
> use or disclosure of the contents of this message is not permitted
> and may be unlawful.
>
> *Disclaimer:* Email messages may be subject to delays,
> interception, non-delivery and unauthorised alterations.
> Therefore, information expressed in this message is not given or
> endorsed by AstraZeneca UK Limited unless otherwise notified by an
> authorised representative independent of this message. No
> contractual relationship is created by this message by any person
> unless specifically indicated by agreement in writing other than
> email.
>
> *Monitoring: *AstraZeneca UK Limited may monitor email traffic
> data and content for the purposes of the prevention and detection
> of crime, ensuring the security of our computer systems and
> checking compliance with our Code of Conduct and policies.
>
> *From:* [email protected]
> <mailto:[email protected]>
> [mailto:[email protected]
> <mailto:[email protected]>] *On Behalf Of *Indranil
> Bhattacharya
> *Sent:* 03 December 2009 15:41
>
> *To:* [email protected] <mailto:[email protected]>
> *Subject:* [NMusers] Model building
>
> Hi, I am in the process of developing a PK/PD model and have a
> naive question regarding model building.
>
> I currently do not have all the PD data and more data would be
> available in the future. For the current data set, I have tried
> models say A to D.
>
> Now model A (cell kill) and B (cell kill +transduction) converges
> using FOCE (no CV% but I am willing to live with that) but from
> the RES and WRES plots we can clearly see that there is some bias.
> The fit is OK but not great. The ofv values are around 400.
>
> Now models B (cell cycle specific kill), C (cell + precursor cell
> kill +transduction) and D (cell cycle specific + indirect response
> model) do not converge using FO or FOCE methods but when I look at
> the fits from the terminated runs, the fits are much better than
> those obtained with Model A, and there seems very little bias.
> Also the ofv values are between 160-250.
>
> So my question is whether the fits, RES, WRES plots and the ofv
> values have meaning even when the minimization terminates.
>
> Regards
>
> Neil
>
> -- Indranil Bhattacharya
>
> --
> Indranil Bhattacharya
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Hi Nick,
As usual, you are very extreme. VPC could be more sensitive in some cases but the first step is to get the model with good fits, RES, WRES plots. The original question was whether to choose the model with numerical problems but good WRES plots versus converged problem with bad WRES plots. Your answer effectively means: do VPC fists, then decide.
Let me disagree and recommend the model with better WRES plots even if this model does not converge.
Thanks
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
Nick Holford wrote:
> Indranil Bhattacharya wrote:
>
> “So my question is whether the fits, RES, WRES plots and the ofv values have meaning even when the minimization terminates”
>
> I do not agree with Joachim that RES, WRES are useful. IMHO these have almost no diagnostic merit except for the most extreme cases of a bad model. Simulation based diagnostics (VPC, SPC) have better diagnostic properties and are being actively evaluated by many groups interested in modelling methodology.
>
> See Karlsson MO, Savic RM. Diagnosing model diagnostics. Clin Pharmacol Ther. 2007 Jul;82(1):17-20. for a demonstration of the problems.
>
> Nick
>
> > Thanks Joachim, that is what I thought. I wanted to be sure that I invest time building the right model and not just some model which works (converges) but is biased. Neil On Fri, Dec 4, 2009 at 3:31 AM, Grevel, Joachim < [email protected] < mailto: [email protected] >> wrote:
> >
> > Hi Neil,
> >
> > You ask:
> >
> > “So my question is whether the fits, RES, WRES plots and the ofv
> > values have meaning even when the minimization terminates”
> >
> > The answer: you bet they matter! Residual plots are the most
> > informative output NONMEM gives you. They should guide you when
> > you determine the basic structure of your model that is supported
> > by the data. Successful termination, covariance step, standard
> > errors, Eigen values, messages, warnings... are just icing on the
> > cake vis-a-vis the residual plots.
> >
> > These are my two pennies worth of advice,
> >
> > Joachim
> >
> > *Joachim Grevel *
> >
> > Senior Pharmacometrician
> >
> > _____________________________________________________________________
> >
> > *AstraZeneca R&D Charnwood*
> >
> > Clinical Pharmacology & DMPK, Charnwood
> >
> > Bakewell Road, Loughborough, Leics., LE11 5RH, England
> >
> > Tel +44 (0) 1509 644035 Fax +44 (0) 1509 645576 Mobile +44 (0)
> > 7920 285905
> >
> > [email protected]
> > <mailto:[email protected]>_
> >
> > _ _
> >
> > P Please consider the environment before printing this e-mail
> >
> > ------------------------------------------------------------------------
> >
> > AstraZeneca UK Limited is a company incorporated in England and
> > Wales with registered number: 03674842 and a registered office at
> > 15 Stanhope Gate, London W1K 1LN.
> >
> > *Confidentiality Notice: *This message is private and may contain
> > confidential, proprietary and legally privileged information. If
> > you have received this message in error, please notify us and
> > remove it from your system and note that you must not copy,
> > distribute or take any action in reliance on it. Any unauthorised
> > use or disclosure of the contents of this message is not permitted
> > and may be unlawful.
> >
> > *Disclaimer:* Email messages may be subject to delays,
> > interception, non-delivery and unauthorised alterations.
> > Therefore, information expressed in this message is not given or
> > endorsed by AstraZeneca UK Limited unless otherwise notified by an
> > authorised representative independent of this message. No
> > contractual relationship is created by this message by any person
> > unless specifically indicated by agreement in writing other than
> > email.
> >
> > *Monitoring: *AstraZeneca UK Limited may monitor email traffic
> > data and content for the purposes of the prevention and detection
> > of crime, ensuring the security of our computer systems and
> > checking compliance with our Code of Conduct and policies.
> >
> > *From:* [email protected]
> > <mailto:[email protected]>
> > [mailto:[email protected]
> > <mailto:[email protected]>] *On Behalf Of *Indranil
> > Bhattacharya
> > *Sent:* 03 December 2009 15:41
> >
> > *To:* [email protected] <mailto:[email protected]>
> > *Subject:* [NMusers] Model building
> >
> > Hi, I am in the process of developing a PK/PD model and have a
> > naive question regarding model building.
> >
> > I currently do not have all the PD data and more data would be
> > available in the future. For the current data set, I have tried
> > models say A to D.
> >
> > Now model A (cell kill) and B (cell kill +transduction) converges
> > using FOCE (no CV% but I am willing to live with that) but from
> > the RES and WRES plots we can clearly see that there is some bias.
> > The fit is OK but not great. The ofv values are around 400.
> >
> > Now models B (cell cycle specific kill), C (cell + precursor cell
> > kill +transduction) and D (cell cycle specific + indirect response
> > model) do not converge using FO or FOCE methods but when I look at
> > the fits from the terminated runs, the fits are much better than
> > those obtained with Model A, and there seems very little bias.
> > Also the ofv values are between 160-250.
> >
> > So my question is whether the fits, RES, WRES plots and the ofv
> > values have meaning even when the minimization terminates.
> >
> > Regards
> >
> > Neil
> >
> > -- Indranil Bhattacharya
> >
> > --
> > Indranil Bhattacharya
>
> --
> Nick Holford, Professor Clinical Pharmacology
> Dept Pharmacology & Clinical Pharmacology
> University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
> tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
> email: [email protected]
> http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Leonid,
I rely on the objective function for model development. Note the word objective.
I have never looked at a RES or WRES plot except to laugh at the subjective foolishness one can imagine there. Note the word subjective.
Of course one can run into problems by looking only at the objective function but that is when the VPC is most helpful. I like to use the VPC to decide which model(s) are fit for purpose.
Thank you for recognizing my extreme views. I prefer to be an outlier than lost among the pseudo-random residuals :-)
Nick
Leonid Gibiansky wrote:
> Hi Nick,
>
> As usual, you are very extreme. VPC could be more sensitive in some cases but the first step is to get the model with good fits, RES, WRES plots. The original question was whether to choose the model with numerical problems but good WRES plots versus converged problem with bad WRES plots. Your answer effectively means: do VPC fists, then decide.
>
> Let me disagree and recommend the model with better WRES plots even if this model does not converge.
>
> Thanks
> Leonid
>
> --------------------------------------
> Leonid Gibiansky, Ph.D.
> President, QuantPharm LLC
> web: www.quantpharm.com
> e-mail: LGibiansky at quantpharm.com
> tel: (301) 767 5566
>
> Nick Holford wrote:
>
> > Indranil Bhattacharya wrote:
> >
> > “So my question is whether the fits, RES, WRES plots and the ofv values have meaning even when the minimization terminates”
> >
> > I do not agree with Joachim that RES, WRES are useful. IMHO these have almost no diagnostic merit except for the most extreme cases of a bad model. Simulation based diagnostics (VPC, SPC) have better diagnostic properties and are being actively evaluated by many groups interested in modelling methodology.
> >
> > See Karlsson MO, Savic RM. Diagnosing model diagnostics. Clin Pharmacol Ther. 2007 Jul;82(1):17-20. for a demonstration of the problems.
> >
> > Nick
> >
> > > Thanks Joachim, that is what I thought. I wanted to be sure that I invest time building the right model and not just some model which works (converges) but is biased. Neil On Fri, Dec 4, 2009 at 3:31 AM, Grevel, Joachim < [email protected] < mailto: [email protected] >> wrote:
> > >
> > > Hi Neil,
> > >
> > > You ask:
> > >
> > > “So my question is whether the fits, RES, WRES plots and the ofv
> > >
> > > values have meaning even when the minimization terminates”
> > >
> > > The answer: you bet they matter! Residual plots are the most
> > >
> > > informative output NONMEM gives you. They should guide you when
> > > you determine the basic structure of your model that is supported
> > > by the data. Successful termination, covariance step, standard
> > > errors, Eigen values, messages, warnings... are just icing on the
> > > cake vis-a-vis the residual plots.
> > >
> > > These are my two pennies worth of advice,
> > >
> > > Joachim
> > >
> > > *Joachim Grevel *
> > >
> > > Senior Pharmacometrician
> > >
> > > _____________________________________________________________________
> > >
> > > *AstraZeneca R&D Charnwood*
> > >
> > > Clinical Pharmacology & DMPK, Charnwood
> > >
> > > Bakewell Road, Loughborough, Leics., LE11 5RH, England
> > >
> > > Tel +44 (0) 1509 644035 Fax +44 (0) 1509 645576 Mobile +44 (0)
> > > 7920 285905
> > >
> > > [email protected]
> > > <mailto:[email protected]>_
> > >
> > > _ _
> > >
> > > P Please consider the environment before printing this e-mail
> > >
> > > ------------------------------------------------------------------------
> > >
> > > AstraZeneca UK Limited is a company incorporated in England and
> > > Wales with registered number: 03674842 and a registered office at
> > > 15 Stanhope Gate, London W1K 1LN.
> > >
> > > *Confidentiality Notice: *This message is private and may contain
> > > confidential, proprietary and legally privileged information. If
> > > you have received this message in error, please notify us and
> > > remove it from your system and note that you must not copy,
> > > distribute or take any action in reliance on it. Any unauthorised
> > > use or disclosure of the contents of this message is not permitted
> > > and may be unlawful.
> > >
> > > *Disclaimer:* Email messages may be subject to delays,
> > > interception, non-delivery and unauthorised alterations.
> > > Therefore, information expressed in this message is not given or
> > > endorsed by AstraZeneca UK Limited unless otherwise notified by an
> > > authorised representative independent of this message. No
> > > contractual relationship is created by this message by any person
> > > unless specifically indicated by agreement in writing other than
> > > email.
> > >
> > > *Monitoring: *AstraZeneca UK Limited may monitor email traffic
> > > data and content for the purposes of the prevention and detection
> > > of crime, ensuring the security of our computer systems and
> > > checking compliance with our Code of Conduct and policies.
> > >
> > > *From:* [email protected]
> > > <mailto:[email protected]>
> > > [mailto:[email protected]
> > > <mailto:[email protected]>] *On Behalf Of *Indranil
> > > Bhattacharya
> > > *Sent:* 03 December 2009 15:41
> > >
> > > *To:* [email protected] <mailto:[email protected]>
> > > *Subject:* [NMusers] Model building
> > >
> > > Hi, I am in the process of developing a PK/PD model and have a
> > >
> > > naive question regarding model building.
> > >
> > > I currently do not have all the PD data and more data would be
> > > available in the future. For the current data set, I have tried
> > > models say A to D.
> > >
> > > Now model A (cell kill) and B (cell kill +transduction) converges
> > > using FOCE (no CV% but I am willing to live with that) but from
> > > the RES and WRES plots we can clearly see that there is some bias.
> > > The fit is OK but not great. The ofv values are around 400.
> > >
> > > Now models B (cell cycle specific kill), C (cell + precursor cell
> > > kill +transduction) and D (cell cycle specific + indirect response
> > > model) do not converge using FO or FOCE methods but when I look at
> > > the fits from the terminated runs, the fits are much better than
> > > those obtained with Model A, and there seems very little bias.
> > > Also the ofv values are between 160-250.
> > >
> > > So my question is whether the fits, RES, WRES plots and the ofv
> > >
> > > values have meaning even when the minimization terminates.
> > >
> > > Regards
> > >
> > > Neil
> > >
> > > -- Indranil Bhattacharya
> > >
> > > --
> > > Indranil Bhattacharya
> >
> > --
> > Nick Holford, Professor Clinical Pharmacology
> > Dept Pharmacology & Clinical Pharmacology
> > University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
> > tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
> > email: [email protected]
> > http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Dear All,
For the sake of correctness, Residuals Analysis is one of the most
relevant topics in the field of Regression. The very notion of objective
function is nothing more than a single characteristic of a type of
residuals. So, raw residuals, standardized residuals, partial residuals,
studentized residuals and so on, are essential to assess serial
correlation, collinearity, leverage, hat-matrix, scedasticity,
transformations, and on, and on, particularly in multivariable and
nonlinear regression. The fact that Nonmem only provides some residuals
by default does not mean we shouldn't look at them and even calculate
others. Using just the value of the objective function and a predictive
check measure is like stirring a boat in the fog with the eyes just
glued on the bow. The picture is much broader. Please check the
extensive literature on Regression and model identification.
Cheers
Luis
-----------------------
Luis Pereira, PhD
Associate Professor
Childrens' Hospital Boston
Harvard Medical School
Boston MA02115
Quoted reply history
________________________________
From: [email protected] on behalf of Nick Holford
Sent: Fri 12/4/2009 10:14 PM
To: nmusers
Subject: Re: [NMusers] Model building
Leonid,
I rely on the objective function for model development. Note the word
objective.
I have never looked at a RES or WRES plot except to laugh at the
subjective foolishness one can imagine there. Note the word subjective.
Of course one can run into problems by looking only at the objective
function but that is when the VPC is most helpful. I like to use the VPC
to decide which model(s) are fit for purpose.
Thank you for recognizing my extreme views. I prefer to be an outlier
than lost among the pseudo-random residuals :-)
Nick
Leonid Gibiansky wrote:
Hi Nick,
As usual, you are very extreme. VPC could be more sensitive in some
cases but the first step is to get the model with good fits, RES, WRES
plots. The original question was whether to choose the model with
numerical problems but good WRES plots versus converged problem with bad
WRES plots. Your answer effectively means: do VPC fists, then decide.
Let me disagree and recommend the model with better WRES plots even if
this model does not converge.
Thanks
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com http://www.quantpharm.com/
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
Nick Holford wrote:
Indranil Bhattacharya wrote:
"So my question is whether the fits, RES, WRES plots and the ofv values
have meaning even when the minimization terminates"
I do not agree with Joachim that RES, WRES are useful. IMHO these have
almost no diagnostic merit except for the most extreme cases of a bad
model. Simulation based diagnostics (VPC, SPC) have better diagnostic
properties and are being actively evaluated by many groups interested in
modelling methodology.
See Karlsson MO, Savic RM. Diagnosing model diagnostics. Clin Pharmacol
Ther. 2007 Jul;82(1):17-20. for a demonstration of the problems.
Nick
Thanks Joachim, that is what I thought. I wanted to be sure that I
invest time building the right model and not just some model which works
(converges) but is biased.
Neil
On Fri, Dec 4, 2009 at 3:31 AM, Grevel, Joachim
<[email protected] <mailto:[email protected]>
<mailto:[email protected]> > wrote:
Hi Neil,
You ask:
"So my question is whether the fits, RES, WRES plots and the ofv
values have meaning even when the minimization terminates"
The answer: you bet they matter! Residual plots are the most
informative output NONMEM gives you. They should guide you when
you determine the basic structure of your model that is supported
by the data. Successful termination, covariance step, standard
errors, Eigen values, messages, warnings... are just icing on the
cake vis-a-vis the residual plots.
These are my two pennies worth of advice,
Joachim
*Joachim Grevel *
Senior Pharmacometrician
_____________________________________________________________________
*AstraZeneca R&D Charnwood*
Clinical Pharmacology & DMPK, Charnwood
Bakewell Road, Loughborough, Leics., LE11 5RH, England
Tel +44 (0) 1509 644035 Fax +44 (0) 1509 645576 Mobile +44 (0)
7920 285905
[email protected]
<mailto:[email protected]>
<mailto:[email protected]> _
_ _
P Please consider the environment before printing this e-mail
------------------------------------------------------------------------
AstraZeneca UK Limited is a company incorporated in England and
Wales with registered number: 03674842 and a registered office at
15 Stanhope Gate, London W1K 1LN.
*Confidentiality Notice: *This message is private and may contain
confidential, proprietary and legally privileged information. If
you have received this message in error, please notify us and
remove it from your system and note that you must not copy,
distribute or take any action in reliance on it. Any unauthorised
use or disclosure of the contents of this message is not permitted
and may be unlawful.
*Disclaimer:* Email messages may be subject to delays,
interception, non-delivery and unauthorised alterations.
Therefore, information expressed in this message is not given or
endorsed by AstraZeneca UK Limited unless otherwise notified by an
authorised representative independent of this message. No
contractual relationship is created by this message by any person
unless specifically indicated by agreement in writing other than
email.
*Monitoring: *AstraZeneca UK Limited may monitor email traffic
data and content for the purposes of the prevention and detection
of crime, ensuring the security of our computer systems and
checking compliance with our Code of Conduct and policies.
*From:* [email protected]
<mailto:[email protected]>
<mailto:[email protected]>
[mailto:[email protected]
<mailto:[email protected]>
<mailto:[email protected]> ] *On Behalf Of *Indranil
Bhattacharya
*Sent:* 03 December 2009 15:41
*To:* [email protected] <mailto:[email protected]>
<mailto:[email protected]>
*Subject:* [NMusers] Model building
Hi, I am in the process of developing a PK/PD model and have a
naive question regarding model building.
I currently do not have all the PD data and more data would be
available in the future. For the current data set, I have tried
models say A to D.
Now model A (cell kill) and B (cell kill +transduction) converges
using FOCE (no CV% but I am willing to live with that) but from
the RES and WRES plots we can clearly see that there is some bias.
The fit is OK but not great. The ofv values are around 400.
Now models B (cell cycle specific kill), C (cell + precursor cell
kill +transduction) and D (cell cycle specific + indirect response
model) do not converge using FO or FOCE methods but when I look at
the fits from the terminated runs, the fits are much better than
those obtained with Model A, and there seems very little bias.
Also the ofv values are between 160-250.
So my question is whether the fits, RES, WRES plots and the ofv
values have meaning even when the minimization terminates.
Regards
Neil
-- Indranil Bhattacharya
--
Indranil Bhattacharya
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Dear Luis,
I guess your biggest concern regarding the residuals is whether residuals are
independent and identically distributed (i.i.d.). I think residual analysis
becomes more important, if you have only a few samples
per subject, since it is much easier to judge the goodness of fit on a DV vs.
TIME plot on linear and semi-log scale in the case of many samples per subject.
Could you please give us some examples where you made a real-life decision
based on one of the types of residual plots which you would not have been able
to make from a thorough analysis of individual fit vs. time plots or from basic
and advanced (visual) predictive checks?
I would be especially interested in the type of residuals you found most
helpful, types of observed variables, the study design, and software for
calculation of the residuals when you found those residuals were most helpful.
Best wishes
Juergen
Jurgen Bulitta, Ph.D., Senior Scientist
Ordway Research Institute, Albany, NY
Quoted reply history
From: [email protected] [mailto:[email protected]] On
Behalf Of [email protected]
Sent: Wednesday, December 09, 2009 12:29 PM
To: [email protected]
Subject: RE: [NMusers] Model building
Dear All,
For the sake of correctness, Residuals Analysis is one of the most relevant
topics in the field of Regression. The very notion of objective function is
nothing more than a single characteristic of a type of residuals. So, raw
residuals, standardized residuals, partial residuals, studentized residuals and
so on, are essential to assess serial correlation, collinearity, leverage,
hat-matrix, scedasticity, transformations, and on, and on, particularly in
multivariable and nonlinear regression. The fact that Nonmem only provides some
residuals by default does not mean we should look at them and even calculate
others. Using just the value of the objective function and a predictive check
measure is like stirring a boat in the fog with the eyes just glued on the bow.
The picture is much broader. Please check the extensive literature on
Regression and model identification.
Cheers
Luis
-----------------------
Luis Pereira, PhD
Associate Professor
Childrens' Hospital Boston
Harvard Medical School
Boston MA02115
________________________________
From: [email protected] on behalf of Nick Holford
Sent: Fri 12/4/2009 10:14 PM
To: nmusers
Subject: Re: [NMusers] Model building
Leonid,
I rely on the objective function for model development. Note the word objective.
I have never looked at a RES or WRES plot except to laugh at the subjective
foolishness one can imagine there. Note the word subjective.
Of course one can run into problems by looking only at the objective function
but that is when the VPC is most helpful. I like to use the VPC to decide which
model(s) are fit for purpose.
Thank you for recognizing my extreme views. I prefer to be an outlier than lost
among the pseudo-random residuals :-)
Nick
Leonid Gibiansky wrote:
Hi Nick,
As usual, you are very extreme. VPC could be more sensitive in some cases but
the first step is to get the model with good fits, RES, WRES plots. The
original question was whether to choose the model with numerical problems but
good WRES plots versus converged problem with bad WRES plots. Your answer
effectively means: do VPC fists, then decide.
Let me disagree and recommend the model with better WRES plots even if this
model does not converge.
Thanks
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: http://www.quantpharm.com/
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
Nick Holford wrote:
Indranil Bhattacharya wrote:
"So my question is whether the fits, RES, WRES plots and the ofv values have
meaning even when the minimization terminates"
I do not agree with Joachim that RES, WRES are useful. IMHO these have almost
no diagnostic merit except for the most extreme cases of a bad model.
Simulation based diagnostics (VPC, SPC) have better diagnostic properties and
are being actively evaluated by many groups interested in modelling methodology.
See Karlsson MO, Savic RM. Diagnosing model diagnostics. Clin Pharmacol Ther.
2007 Jul;82(1):17-20. for a demonstration of the problems.
Nick
Thanks Joachim, that is what I thought. I wanted to be sure that I invest time
building the right model and not just some model which works (converges) but is
biased.
Neil
On Fri, Dec 4, 2009 at 3:31 AM, Grevel, Joachim
<[email protected]<mailto:[email protected]>
<mailto:[email protected]><mailto:[email protected]>>
wrote:
Hi Neil,
You ask:
"So my question is whether the fits, RES, WRES plots and the ofv
values have meaning even when the minimization terminates"
The answer: you bet they matter! Residual plots are the most
informative output NONMEM gives you. They should guide you when
you determine the basic structure of your model that is supported
by the data. Successful termination, covariance step, standard
errors, Eigen values, messages, warnings... are just icing on the
cake vis-a-vis the residual plots.
These are my two pennies worth of advice,
Joachim
*Joachim Grevel *
Senior Pharmacometrician
_____________________________________________________________________
*AstraZeneca R&D Charnwood*
Clinical Pharmacology & DMPK, Charnwood
Bakewell Road, Loughborough, Leics., LE11 5RH, England
Tel +44 (0) 1509 644035 Fax +44 (0) 1509 645576 Mobile +44 (0)
7920 285905
[email protected]
<mailto:[email protected]><mailto:[email protected]>_
_ _
P Please consider the environment before printing this e-mail
------------------------------------------------------------------------
AstraZeneca UK Limited is a company incorporated in England and
Wales with registered number: 03674842 and a registered office at
15 Stanhope Gate, London W1K 1LN.
*Confidentiality Notice: *This message is private and may contain
confidential, proprietary and legally privileged information. If
you have received this message in error, please notify us and
remove it from your system and note that you must not copy,
distribute or take any action in reliance on it. Any unauthorised
use or disclosure of the contents of this message is not permitted
and may be unlawful.
*Disclaimer:* Email messages may be subject to delays,
interception, non-delivery and unauthorised alterations.
Therefore, information expressed in this message is not given or
endorsed by AstraZeneca UK Limited unless otherwise notified by an
authorised representative independent of this message. No
contractual relationship is created by this message by any person
unless specifically indicated by agreement in writing other than
email.
*Monitoring: *AstraZeneca UK Limited may monitor email traffic
data and content for the purposes of the prevention and detection
of crime, ensuring the security of our computer systems and
checking compliance with our Code of Conduct and policies.
*From:* [email protected]<mailto:[email protected]>
<mailto:[email protected]><mailto:[email protected]>
[mailto:[email protected]
<mailto:[email protected]><mailto:[email protected]>]
*On Behalf Of *Indranil
Bhattacharya
*Sent:* 03 December 2009 15:41
*To:* [email protected]<mailto:[email protected]>
<mailto:[email protected]><mailto:[email protected]>
*Subject:* [NMusers] Model building
Hi, I am in the process of developing a PK/PD model and have a
naive question regarding model building.
I currently do not have all the PD data and more data would be
available in the future. For the current data set, I have tried
models say A to D.
Now model A (cell kill) and B (cell kill +transduction) converges
using FOCE (no CV% but I am willing to live with that) but from
the RES and WRES plots we can clearly see that there is some bias.
The fit is OK but not great. The ofv values are around 400.
Now models B (cell cycle specific kill), C (cell + precursor cell
kill +transduction) and D (cell cycle specific + indirect response
model) do not converge using FO or FOCE methods but when I look at
the fits from the terminated runs, the fits are much better than
those obtained with Model A, and there seems very little bias.
Also the ofv values are between 160-250.
So my question is whether the fits, RES, WRES plots and the ofv
values have meaning even when the minimization terminates.
Regards
Neil
-- Indranil Bhattacharya
--
Indranil Bhattacharya
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]<mailto:[email protected]>
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]<mailto:[email protected]>
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford