Dear Friends – Observations versus population predicted is considered a
standard diagnostic plot in our field. I used to place observations on the
x-axis and predictions on the yaxis. Then I was pointed to a publication from
ISOP
( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/)
which recommended plotting predictions on the xaxis and observations on the
yaxis. To the best of my knowledge, there was no justification provided. It did
question my decades old practice, so I did some thinking and digging. Thought
to share it here so others might benefit from it. If this is obvious to you
all, then I can say I am caught up!
1. We write our models as observed = predicted + random error; which can be
interpreted to be in the form: y = f(x) + random error. It is technically not
though. Hence predicted goes on the xaxis, as it is free of random error. It is
considered a correlation plot, which makes plotting either way acceptable. This
is not so critical as the next one.
2. However, there is a statistical reason why it is important to keep
predictions on the xaxis. Invariably we always add a loess trend line for these
diagnostic plots. To demonstrate the impact, I took a simple iv bolus single
dose dataset and compared both approaches. The results are available at this
link: https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf. I
used Pumas software, but the scientific underpinning is agnostic to software.
See the two plots on Pages 5 and 6. The interpretation of the bias between the
two approaches is different. This is the statistical reason why it matters to
plot predictions on the xaxis.
Joga Gobburu
University of Maryland
Observed (yaxis) vs Predicted (xaxis) Diagnostic Plot - Scientific basis.
So whichever axis the observed data is plotted on is parallel to the direction of noise (random residual error). When you fit the loess line, I think it will generally assume noise is vertical i.e. parallel to the y-axis. So the problem is really that the loess line is fitting noise in the wrong direction if the observed is actually on the x-axis ... which means you are right, the observed needs to go on the y-axis and deviations need to be interpreted parallel to the y-axis.
Kind regards, James
https://product.popypkpd.com/
PS Of course, if you were to fit a loess line with horizontal noise and observed data on the x-axis, you should reach identical conclusions to the conventional vertical noise and observed data on the y-axis.
Quoted reply history
On 17/08/2023 11:35, Gobburu, Joga wrote:
> Dear Friends – Observations versus population predicted is considered a standard diagnostic plot in our field. I used to place observations on the x-axis and predictions on the yaxis. Then I was pointed to a publication from ISOP ( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/ ) which recommended plotting predictions on the xaxis and observations on the yaxis. To the best of my knowledge, there was no justification provided. It did question my decades old practice, so I did some thinking and digging. Thought to share it here so others might benefit from it. If this is obvious to you all, then I can say I am caught up!
>
> 1. We write our models as observed = predicted + random error; which
> can be interpreted to be in the form: y = f(x) + random error. It
> is technically not though. Hence predicted goes on the xaxis, as
> it is free of random error. It is considered a correlation plot,
> which makes plotting either way acceptable. This is not so
> critical as the next one.
> 2. However, there is a statistical reason why it is important to keep
> predictions on the xaxis. Invariably we always add a loess trend
> line for these diagnostic plots. To demonstrate the impact, I took
> a simple iv bolus single dose dataset and compared both
> approaches. The results are available at this link:
> https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf.
> I used Pumas software, but the scientific underpinning is agnostic
> to software. See the two plots on Pages 5 and 6. The
> interpretation of the bias between the two approaches is
> different. This is the statistical reason why it matters to plot
> predictions on the xaxis.
>
> Joga Gobburu
>
> University of Maryland
--
James G Wright PhD,
Scientist, Wright Dose Ltd
Tel: UK (0)772 5636914
Hi Joga,
Fully agree on this, unfortunately it is still often shown the other way
around which is at least confusing.
There is a publication on this very topic here
https://www.sciencedirect.com/science/article/abs/pii/S0304380008002305 that
arrives at the same conclusion and can be helpful.
Best,
Wilbert
Op do 17 aug 2023 om 19:47 schreef Gobburu, Joga <[email protected]
>:
> Dear James – how have you been?
>
>
>
> Yes, you said it most eloquently. Its not about plotting per se but “the
> problem is really that the loess line is fitting noise in the wrong
> direction if the observed is actually on the x-axis”. Thank you…J
>
>
>
> *From: *James G Wright <[email protected]>
> *Date: *Thursday, August 17, 2023 at 7:16 AM
> *To: *Gobburu, Joga <[email protected]>, [email protected] <
> [email protected]>
> *Subject: *Re: [NMusers] Observed (yaxis) vs Predicted (xaxis) Diagnostic
> Plot - Scientific basis.
>
> You don't often get email from [email protected]. Learn why this is
> important https://aka.ms/LearnAboutSenderIdentification
>
> *CAUTION: *This message originated from a non-UMB email system. Hover
> over any links before clicking and use caution opening attachments.
>
> So whichever axis the observed data is plotted on is parallel to the
> direction of noise (random residual error). When you fit the loess line, I
> think it will generally assume noise is vertical i.e. parallel to the
> y-axis. So the problem is really that the loess line is fitting noise in
> the wrong direction if the observed is actually on the x-axis ... which
> means you are right, the observed needs to go on the y-axis and deviations
> need to be interpreted parallel to the y-axis.
>
>
>
> Kind regards, James
>
>
>
> https://product.popypkpd.com/
>
>
>
> PS Of course, if you were to fit a loess line with horizontal noise and
> observed data on the x-axis, you should reach identical conclusions to the
> conventional vertical noise and observed data on the y-axis.
>
>
>
Quoted reply history
> On 17/08/2023 11:35, Gobburu, Joga wrote:
>
> Dear Friends – Observations versus population predicted is considered a
> standard diagnostic plot in our field. I used to place observations on the
> x-axis and predictions on the yaxis. Then I was pointed to a publication
> from ISOP (
> https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/)
> which recommended plotting predictions on the xaxis and observations on the
> yaxis. To the best of my knowledge, there was no justification provided. It
> did question my decades old practice, so I did some thinking and digging.
> Thought to share it here so others might benefit from it. If this is
> obvious to you all, then I can say I am caught up!
>
>
>
> 1. We write our models as observed = predicted + random error; which
> can be interpreted to be in the form: y = f(x) + random error. It is
> technically not though. Hence predicted goes on the xaxis, as it is free of
> random error. It is considered a correlation plot, which makes plotting
> either way acceptable. This is not so critical as the next one.
> 2. However, there is a statistical reason why it is important to keep
> predictions on the xaxis. Invariably we always add a loess trend line for
> these diagnostic plots. To demonstrate the impact, I took a simple iv bolus
> single dose dataset and compared both approaches. The results are available
> at this link:
> https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf.
> I used Pumas software, but the scientific underpinning is agnostic to
> software. See the two plots on Pages 5 and 6. The interpretation of the
> bias between the two approaches is different. This is the statistical
> reason why it matters to plot predictions on the xaxis.
>
>
>
> Joga Gobburu
>
> University of Maryland
>
>
>
> --
>
> James G Wright PhD,
>
> Scientist, Wright Dose Ltd
>
> Tel: UK (0)772 5636914
>
>
Dear Joga and all,
Joga makes a valuable point that all pharmacometricians should be aware of.
Standard methodology for regression assumes that the x-variable is without
error (loess, linear regression etc.). Note that it is the same for NLME
models i.e. we generally assume that our independent variables e.g. time,
covariates etc. are without error.
For DV vs. PRED plots it is common practice, even among those that do not
know why, to plot PRED on the x-axis and DV on the y-axis. A greater
problem with these plots is the commonly held expectation that for a "good
model" a smooth or regression line should align with the line of unity.
Though this seems intuitive it is a flawed assumption. This issue was
clearly pointed out by Mats Karlsson and Rada Savic in their 2007 paper
titled "Diagnosing Model Diagnostics''. For simple well-behaved examples
you will see an alignment around the line of unity for DV vs. PRED plots.
However, there are several factors that contribute to an expected deviation
from this expectation:
(1) Censoring (e.g. censoring of observations < LLOQ)
- In this case DVs are capped at LLOQ but PRED values are not. This makes
it perfectly expected that there will be a deviation from alignment around
the line of unity in the lower range.
(2) Strong non-linearities
- The more nonlinear the modelled system is, the greater the expected
deviation from the line of unity. Especially in combination with
significant ETA correlations.
(3) High variability
- With higher between/within subject variability (e.g. IIV and RUV) that
isn't normally distributed (e.g. exponential distributions) will result in
an expected deviation from the line of unity. Note: this is a form of
non-linearity so it may fall under the above category.
(4) Adaptive designs (e.g. TDM dosing)
- Listed in the original paper by Karlsson & Savic but I have not been able
to recreate an issue in this case.
I am rather sure that many thousands of hours have been spent on modeling
trying to correct for perceived model misspecifications that are not really
there. This is why I recommend relying primarily on simulation-based model
diagnostics (e.g. VPCs) and as far as possible account for censoring that
affects the original dataset. As pointed out by Karlsson & Savic a
simulation/re-estimation based approach can also be used to investigate the
expected behavior for DV vs. PRED plots for a particular model and dataset
(e.g. mirror plots in Xpose). Note that to my knowledge there is yet
no automated way to handle censoring in this context (clearly doable if
anyone wants to develop a nifty implementation of that).
If we leave the DV vs. PRED plot case, there are many other instances where
we use scatter plots where it is much less clear what can be considered the
independent variable and yet other cases where the assumption that the
x-variable is without error is violated in a way that makes the results
hard to interpret. One instance of the latter is when exposure-response is
studied by plotting observed PD response versus observed trough plasma
concentrations. This is already a way too long email so I will not deep
dive into that problem as well.
Best regards,
Martin Bergstrand, Ph.D.
Principal Consultant
Pharmetheus AB
[email protected]
www.pharmetheus.com
Quoted reply history
On Thu, Aug 17, 2023 at 12:44 PM Gobburu, Joga <[email protected]>
wrote:
> Dear Friends – Observations versus population predicted is considered a
> standard diagnostic plot in our field. I used to place observations on the
> x-axis and predictions on the yaxis. Then I was pointed to a publication
> from ISOP (
> https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/)
> which recommended plotting predictions on the xaxis and observations on the
> yaxis. To the best of my knowledge, there was no justification provided. It
> did question my decades old practice, so I did some thinking and digging.
> Thought to share it here so others might benefit from it. If this is
> obvious to you all, then I can say I am caught up!
>
>
>
> 1. We write our models as observed = predicted + random error; which
> can be interpreted to be in the form: y = f(x) + random error. It is
> technically not though. Hence predicted goes on the xaxis, as it is free of
> random error. It is considered a correlation plot, which makes plotting
> either way acceptable. This is not so critical as the next one.
> 2. However, there is a statistical reason why it is important to keep
> predictions on the xaxis. Invariably we always add a loess trend line for
> these diagnostic plots. To demonstrate the impact, I took a simple iv bolus
> single dose dataset and compared both approaches. The results are available
> at this link:
> https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf.
> I used Pumas software, but the scientific underpinning is agnostic to
> software. See the two plots on Pages 5 and 6. The interpretation of the
> bias between the two approaches is different. This is the statistical
> reason why it matters to plot predictions on the xaxis.
>
>
>
> Joga Gobburu
>
> University of Maryland
>
--
*This communication is confidential and is only intended for the use of the
individual or entity to which it is directed. It may contain information
that is privileged and exempt from disclosure under applicable law. If you
are not the intended recipient please notify us immediately. Please do not
copy it or disclose its contents to any other person.*
*Any personal data
will be processed in accordance with Pharmetheus' privacy notice, available
here https://pharmetheus.com/privacy-policy/.**
*
Thanks Joga for raising the issue of so called diagnostic plots and Martin’s
reminder that they are not reliable as diagnostics.
The gold standard tool for model evaluation, which may also help diagnose model
problems, it the VPC. Martin - it is not a “for example” method -- it is the
primary model evaluation tool.
Comparison of the median observed percentile with the median predicted
percentile is the first step in using a VPC. Unfortunately, there are still
VPCs being produced that show only the observed percentiles without the
corresponding predicted percentiles.
All so called diagnostic plots and VPCs that do not show observed AND predicted
percentiles belong in the bin.
Best wishes,
Nick
--
Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP
mobile:NZ+64(21)46 23 53 ; FR+33(6)62 32 46 72
email: [email protected]<mailto:[email protected]>
web: http://holford.fmhs.auckland.ac.nz/
Quoted reply history
From: [email protected] <[email protected]> On Behalf Of
Martin Bergstrand
Sent: Friday, August 18, 2023 9:48 AM
To: Gobburu, Joga <[email protected]>
Cc: [email protected]
Subject: Re: [NMusers] Observed (yaxis) vs Predicted (xaxis) Diagnostic Plot -
Scientific basis.
Dear Joga and all,
Joga makes a valuable point that all pharmacometricians should be aware of.
Standard methodology for regression assumes that the x-variable is without
error (loess, linear regression etc.). Note that it is the same for NLME models
i.e. we generally assume that our independent variables e.g. time, covariates
etc. are without error.
For DV vs. PRED plots it is common practice, even among those that do not know
why, to plot PRED on the x-axis and DV on the y-axis. A greater problem with
these plots is the commonly held expectation that for a "good model" a smooth
or regression line should align with the line of unity. Though this seems
intuitive it is a flawed assumption. This issue was clearly pointed out by Mats
Karlsson and Rada Savic in their 2007 paper titled "Diagnosing Model
Diagnostics''. For simple well-behaved examples you will see an alignment
around the line of unity for DV vs. PRED plots. However, there are several
factors that contribute to an expected deviation from this expectation:
(1) Censoring (e.g. censoring of observations < LLOQ)
- In this case DVs are capped at LLOQ but PRED values are not. This makes it
perfectly expected that there will be a deviation from alignment around the
line of unity in the lower range.
(2) Strong non-linearities
- The more nonlinear the modelled system is, the greater the expected deviation
from the line of unity. Especially in combination with significant ETA
correlations.
(3) High variability
- With higher between/within subject variability (e.g. IIV and RUV) that isn't
normally distributed (e.g. exponential distributions) will result in an
expected deviation from the line of unity. Note: this is a form of
non-linearity so it may fall under the above category.
(4) Adaptive designs (e.g. TDM dosing)
- Listed in the original paper by Karlsson & Savic but I have not been able to
recreate an issue in this case.
I am rather sure that many thousands of hours have been spent on modeling
trying to correct for perceived model misspecifications that are not really
there. This is why I recommend relying primarily on simulation-based model
diagnostics (e.g. VPCs) and as far as possible account for censoring that
affects the original dataset. As pointed out by Karlsson & Savic a
simulation/re-estimation based approach can also be used to investigate the
expected behavior for DV vs. PRED plots for a particular model and dataset
(e.g. mirror plots in Xpose). Note that to my knowledge there is yet no
automated way to handle censoring in this context (clearly doable if anyone
wants to develop a nifty implementation of that).
If we leave the DV vs. PRED plot case, there are many other instances where we
use scatter plots where it is much less clear what can be considered the
independent variable and yet other cases where the assumption that the
x-variable is without error is violated in a way that makes the results hard to
interpret. One instance of the latter is when exposure-response is studied by
plotting observed PD response versus observed trough plasma concentrations.
This is already a way too long email so I will not deep dive into that problem
as well.
Best regards,
Martin Bergstrand, Ph.D.
Principal Consultant
Pharmetheus AB
[email protected]<mailto:[email protected]>
http://www.pharmetheus.com
On Thu, Aug 17, 2023 at 12:44 PM Gobburu, Joga
<[email protected]<mailto:[email protected]>> wrote:
Dear Friends – Observations versus population predicted is considered a
standard diagnostic plot in our field. I used to place observations on the
x-axis and predictions on the yaxis. Then I was pointed to a publication from
ISOP
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/)
which recommended plotting predictions on the xaxis and observations on the
yaxis. To the best of my knowledge, there was no justification provided. It did
question my decades old practice, so I did some thinking and digging. Thought
to share it here so others might benefit from it. If this is obvious to you
all, then I can say I am caught up!
1. We write our models as observed = predicted + random error; which can be
interpreted to be in the form: y = f(x) + random error. It is technically not
though. Hence predicted goes on the xaxis, as it is free of random error. It is
considered a correlation plot, which makes plotting either way acceptable. This
is not so critical as the next one.
2. However, there is a statistical reason why it is important to keep
predictions on the xaxis. Invariably we always add a loess trend line for these
diagnostic plots. To demonstrate the impact, I took a simple iv bolus single
dose dataset and compared both approaches. The results are available at this
link:
https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf.
I used Pumas software, but the scientific underpinning is agnostic to
software. See the two plots on Pages 5 and 6. The interpretation of the bias
between the two approaches is different. This is the statistical reason why it
matters to plot predictions on the xaxis.
Joga Gobburu
University of Maryland
This communication is confidential and is only intended for the use of the
individual or entity to which it is directed. It may contain information that
is privileged and exempt from disclosure under applicable law. If you are not
the intended recipient please notify us immediately. Please do not copy it or
disclose its contents to any other person.
Any personal data will be processed in accordance with Pharmetheus' privacy
notice, available https://pharmetheus.com/privacy-policy/.
Hi Joga, Wilbert,
It indeed is an interesting aspect. I was triggered to think about this during my masters research (with dense time series), and for me it was helpful to think about orthogonal regression. One can find and compare the expressions in the wikipedia entries at https://en.wikipedia.org/wiki/Deming_regression#Orthogonal_regression and https://en.wikipedia.org/wiki/Simple_linear_regression to see how it impacts. [ Side note: As the authors in the paper you referenced Wilbert, express, the correlation coefficient r is symmetric for x and y and is not impacted.] A good example of how it changes the fit can be found in the last figure of this blog < https://www.r-bloggers.com/2018/10/about-a-curious-feature-and-interpretation-of-linear-regressions/ >: basically linear regression goes through the middle of the cloud at the edges in the y-direction, while orthogonal goes through them balanced perpendicular to the linear relation.
But in the end it also goes down to the general expectation in regression to put the independent variable without error on the x-axis and the dependent variable on the y-axis. From this we can derive it is best to put the observations on the y-axis (*). Therefore we have two reasons to adhere to the approach of putting observed on the y-axis and predicted on the x-axis.
Hope this helps,
Jeroen
(*) Whether or not the predictions are without (residual) error is a matter of debate and situation. If we go from PRED predictions to PRED when the model has a covariate, to post-hoc predictions, the amount of randomness increases. The observed values nevertheless will retain most randomness and therefore are expected on the y-axis.
http://pd-value.com
[email protected]
@PD_value
+31 6 23118438
-- More value out of your data!
Quoted reply history
On 18-08-2023 08:07, Wilbert de Witte wrote:
> Hi Joga,
>
> Fully agree on this, unfortunately it is still often shown the other way around which is at least confusing. There is a publication on this very topic here < https://www.sciencedirect.com/science/article/abs/pii/S0304380008002305 > that arrives at the same conclusion and can be helpful.
>
> Best,
>
> Wilbert
>
> Op do 17 aug 2023 om 19:47 schreef Gobburu, Joga < [email protected] >:
>
> Dear James – how have you been?
>
> Yes, you said it most eloquently. Its not about plotting per se
> but “the problem is really that the loess line is fitting noise in
> the wrong direction if the observed is actually on the x-axis”.
> Thank you…J
>
> *From: *James G Wright <[email protected]>
> *Date: *Thursday, August 17, 2023 at 7:16 AM
> *To: *Gobburu, Joga <[email protected]>,
> [email protected] <[email protected]>
> *Subject: *Re: [NMusers] Observed (yaxis) vs Predicted (xaxis)
> Diagnostic Plot - Scientific basis.
>
> You don't often get email from [email protected]. Learn why
> this is important https://aka.ms/LearnAboutSenderIdentification
>
> *CAUTION: *This message originated from a non-UMB email system.
> Hover over any links before clicking and use caution opening
> attachments.
>
> So whichever axis the observed data is plotted on is parallel to
> the direction of noise (random residual error). When you fit the
> loess line, I think it will generally assume noise is vertical
> i.e. parallel to the y-axis. So the problem is really that the
> loess line is fitting noise in the wrong direction if the observed
> is actually on the x-axis ... which means you are right, the
> observed needs to go on the y-axis and deviations need to be
> interpreted parallel to the y-axis.
>
> Kind regards, James
>
> https://product.popypkpd.com/
>
> PS Of course, if you were to fit a loess line with horizontal
> noise and observed data on the x-axis, you should reach identical
> conclusions to the conventional vertical noise and observed data
> on the y-axis.
>
> On 17/08/2023 11:35, Gobburu, Joga wrote:
>
> Dear Friends – Observations versus population predicted is
> considered a standard diagnostic plot in our field. I used to
> place observations on the x-axis and predictions on the yaxis.
> Then I was pointed to a publication from ISOP
>
> ( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/)
> which recommended plotting predictions on the xaxis and
> observations on the yaxis. To the best of my knowledge, there
> was no justification provided. It did question my decades old
> practice, so I did some thinking and digging. Thought to share
> it here so others might benefit from it. If this is obvious to
> you all, then I can say I am caught up!
>
> 1. We write our models as observed = predicted + random
> error; which can be interpreted to be in the form: y =
> f(x) + random error. It is technically not though. Hence
> predicted goes on the xaxis, as it is free of random
> error. It is considered a correlation plot, which makes
> plotting either way acceptable. This is not so critical as
> the next one.
> 2. However, there is a statistical reason why it is important
> to keep predictions on the xaxis. Invariably we always add
> a loess trend line for these diagnostic plots. To
> demonstrate the impact, I took a simple iv bolus single
> dose dataset and compared both approaches. The results are
> available at this link:
>
> https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf.
> I used Pumas software, but the scientific underpinning is
> agnostic to software. See the two plots on Pages 5 and 6.
> The interpretation of the bias between the two approaches
> is different. This is the statistical reason why it
> matters to plot predictions on the xaxis.
>
> Joga Gobburu
>
> University of Maryland
>
> --
>
> James G Wright PhD,
>
> Scientist, Wright Dose Ltd
>
> Tel: UK (0)772 5636914
Hi Nick,
I hope you are well! I think censoring is still a problem for VPCs unless you are including am explicit model component for the mechanism of censoring…?
I agree that the VPC is the most important method of assessing your model, and diagnostic plots are mainly to help you work out why your VPC isn’t adequate. Kind regards, James
Enviado do meu iPhone
> Em 18 de ago. de 2023, à(s) 09:27, Nick Holford < [email protected] > escreveu:
>
>
>
> Thanks Joga for raising the issue of so called diagnostic plots and Martin’s reminder that they are not reliable as diagnostics.
>
> The gold standard tool for model evaluation, which may also help diagnose model problems, it the VPC. Martin - it is not a “for example” method -- it is the primary model evaluation tool.
>
> Comparison of the median observed percentile with the median predicted percentile is the first step in using a VPC. Unfortunately, there are still VPCs being produced that show only the observed percentiles without the corresponding predicted percentiles.
>
> All so called diagnostic plots and VPCs that do not show observed AND predicted percentiles belong in the bin.
>
> Best wishes,
>
> Nick
>
> --
>
> Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP
>
> mobile:NZ+64(21)46 23 53 ; FR+33(6)62 32 46 72
>
> email: [email protected] <mailto:[email protected]>
>
> web: http://holford.fmhs.auckland.ac.nz/ < http://holford.fmhs.auckland.ac.nz/ >
>
> *From:* [email protected] < [email protected] > *On Behalf Of *Martin Bergstrand
>
> *Sent:* Friday, August 18, 2023 9:48 AM
> *To:* Gobburu, Joga <[email protected]>
> *Cc:* [email protected]
>
> *Subject:* Re: [NMusers] Observed (yaxis) vs Predicted (xaxis) Diagnostic Plot - Scientific basis.
>
> Dear Joga and all,
>
> Joga makes a valuable point that all pharmacometricians should be aware of. Standard methodology for regression assumes that the x-variable is without error (loess, linear regression etc.). Note that it is the same for NLME models i.e. we generally assume that our independent variables e.g. time, covariates etc. are without error.
>
> For DV vs. PRED plots it is common practice, even among those that do not know why, to plot PRED on the x-axis and DV on the y-axis. A greater problem with these plots is the commonly held expectation that for a "good model" a smooth or regression line should align with the line of unity. Though this seems intuitive it is a flawed assumption. This issue was clearly pointed out by Mats Karlsson and Rada Savic in their 2007 paper titled "Diagnosing Model Diagnostics''. For simple well-behaved examples you will see an alignment around the line of unity for DV vs. PRED plots. However, there are several factors that contribute to an expected deviation from this expectation:
>
> (1) Censoring (e.g. censoring of observations < LLOQ)
>
> - In this case DVs are capped at LLOQ but PRED values are not. This makes it perfectly expected that there will be a deviation from alignment around the line of unity in the lower range.
>
> (2) Strong non-linearities
>
> - The more nonlinear the modelled system is, the greater the expected deviation from the line of unity. Especially in combination with significant ETA correlations.
>
> (3) High variability
>
> - With higher between/within subject variability (e.g. IIV and RUV) that isn't normally distributed (e.g. exponential distributions) will result in an expected deviation from the line of unity. Note: this is a form of non-linearity so it may fall under the above category.
>
> (4) Adaptive designs (e.g. TDM dosing)
>
> - Listed in the original paper by Karlsson & Savic but I have not been able to recreate an issue in this case.
>
> I am rather sure that many thousands of hours have been spent on modeling trying to correct for perceived model misspecifications that are not really there. This is why I recommend relying primarily on simulation-based model diagnostics (e.g. VPCs) and as far as possible account for censoring that affects the original dataset. As pointed out by Karlsson & Savic a simulation/re-estimation based approach can also be used to investigate the expected behavior for DV vs. PRED plots for a particular model and dataset (e.g. mirror plots in Xpose). Note that to my knowledge there is yet no automated way to handle censoring in this context (clearly doable if anyone wants to develop a nifty implementation of that).
>
> If we leave the DV vs. PRED plot case, there are many other instances where we use scatter plots where it is much less clear what can be considered the independent variable and yet other cases where the assumption that the x-variable is without error is violated in a way that makes the results hard to interpret. One instance of the latter is when exposure-response is studied by plotting observed PD response versus observed trough plasma concentrations. This is already a way too long email so I will not deep dive into that problem as well.
>
> Best regards,
>
> Martin Bergstrand, Ph.D.
>
> Principal Consultant
>
> Pharmetheus AB
>
> [email protected]
>
> www.pharmetheus.com http://www.pharmetheus.com
>
Quoted reply history
> On Thu, Aug 17, 2023 at 12:44 PM Gobburu, Joga < [email protected] > wrote:
>
> Dear Friends – Observations versus population predicted is
> considered a standard diagnostic plot in our field. I used to
> place observations on the x-axis and predictions on the yaxis.
> Then I was pointed to a publication from ISOP
>
> ( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/)
> which recommended plotting predictions on the xaxis and
> observations on the yaxis. To the best of my knowledge, there was
> no justification provided. It did question my decades old
> practice, so I did some thinking and digging. Thought to share it
> here so others might benefit from it. If this is obvious to you
> all, then I can say I am caught up!
>
> 1. We write our models as observed = predicted + random error;
> which can be interpreted to be in the form: y = f(x) + random
> error. It is technically not though. Hence predicted goes on
> the xaxis, as it is free of random error. It is considered a
> correlation plot, which makes plotting either way acceptable.
> This is not so critical as the next one.
> 2. However, there is a statistical reason why it is important to
> keep predictions on the xaxis. Invariably we always add a
> loess trend line for these diagnostic plots. To demonstrate
> the impact, I took a simple iv bolus single dose dataset and
> compared both approaches. The results are available at this
> link:
> https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf.
> I used Pumas software, but the scientific underpinning is
> agnostic to software. See the two plots on Pages 5 and 6. The
> interpretation of the bias between the two approaches is
> different. This is the statistical reason why it matters to
> plot predictions on the xaxis.
>
> Joga Gobburu
>
> University of Maryland
>
> /This communication is confidential and is only intended for the use of the individual or entity to which it is directed. It may contain information that is privileged and exempt from disclosure under applicable law. If you are not the intended recipient please notify us immediately. Please do not copy it or disclose its contents to any other person./
>
> /Any personal data will be processed in accordance with Pharmetheus' privacy notice, available here < https://pharmetheus.com/privacy-policy/ >./
Thank you for the discussion. The obs vs pred are not the plots I usually look
at, at least at the beginning. However, these seem to have become common
practice. The first plots we ought to review are the PK plots (time vs obs,
ipred, pred conc for PK by id and overall; and similar plots for PKPD).
Unfortunately, most publications do not even include these basic plots these
days.
I would like to make a strong recommendation for these basic plots. VPCs are a
different matter for another day.
J
Quoted reply history
From: [email protected] <[email protected]> on behalf of
James G Wright <[email protected]>
Date: Friday, August 18, 2023 at 4:52 AM
To: NMusers <[email protected]>
Subject: Fwd: [NMusers] Observed (yaxis) vs Predicted (xaxis) Diagnostic Plot -
Scientific basis.
CAUTION: This message originated from a non-UMB email system. Hover over any
links before clicking and use caution opening attachments.
Hi Nick,
I hope you are well! I think censoring is still a problem for VPCs unless you
are including am explicit model component for the mechanism of censoring…?
I agree that the VPC is the most important method of assessing your model, and
diagnostic plots are mainly to help you work out why your VPC isn’t adequate.
Kind regards, James
Enviado do meu iPhone
Em 18 de ago. de 2023, à(s) 09:27, Nick Holford
<[email protected]><mailto:[email protected]> escreveu:
Thanks Joga for raising the issue of so called diagnostic plots and Martin’s
reminder that they are not reliable as diagnostics.
The gold standard tool for model evaluation, which may also help diagnose model
problems, it the VPC. Martin - it is not a “for example” method -- it is the
primary model evaluation tool.
Comparison of the median observed percentile with the median predicted
percentile is the first step in using a VPC. Unfortunately, there are still
VPCs being produced that show only the observed percentiles without the
corresponding predicted percentiles.
All so called diagnostic plots and VPCs that do not show observed AND predicted
percentiles belong in the bin.
Best wishes,
Nick
--
Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP
mobile:NZ+64(21)46 23 53 ; FR+33(6)62 32 46 72
email: [email protected]<mailto:[email protected]>
web: http://holford.fmhs.auckland.ac.nz/
From: [email protected]<mailto:[email protected]>
<[email protected]><mailto:[email protected]> On Behalf
Of Martin Bergstrand
Sent: Friday, August 18, 2023 9:48 AM
To: Gobburu, Joga <[email protected]><mailto:[email protected]>
Cc: [email protected]<mailto:[email protected]>
Subject: Re: [NMusers] Observed (yaxis) vs Predicted (xaxis) Diagnostic Plot -
Scientific basis.
Dear Joga and all,
Joga makes a valuable point that all pharmacometricians should be aware of.
Standard methodology for regression assumes that the x-variable is without
error (loess, linear regression etc.). Note that it is the same for NLME models
i.e. we generally assume that our independent variables e.g. time, covariates
etc. are without error.
For DV vs. PRED plots it is common practice, even among those that do not know
why, to plot PRED on the x-axis and DV on the y-axis. A greater problem with
these plots is the commonly held expectation that for a "good model" a smooth
or regression line should align with the line of unity. Though this seems
intuitive it is a flawed assumption. This issue was clearly pointed out by Mats
Karlsson and Rada Savic in their 2007 paper titled "Diagnosing Model
Diagnostics''. For simple well-behaved examples you will see an alignment
around the line of unity for DV vs. PRED plots. However, there are several
factors that contribute to an expected deviation from this expectation:
(1) Censoring (e.g. censoring of observations < LLOQ)
- In this case DVs are capped at LLOQ but PRED values are not. This makes it
perfectly expected that there will be a deviation from alignment around the
line of unity in the lower range.
(2) Strong non-linearities
- The more nonlinear the modelled system is, the greater the expected deviation
from the line of unity. Especially in combination with significant ETA
correlations.
(3) High variability
- With higher between/within subject variability (e.g. IIV and RUV) that isn't
normally distributed (e.g. exponential distributions) will result in an
expected deviation from the line of unity. Note: this is a form of
non-linearity so it may fall under the above category.
(4) Adaptive designs (e.g. TDM dosing)
- Listed in the original paper by Karlsson & Savic but I have not been able to
recreate an issue in this case.
I am rather sure that many thousands of hours have been spent on modeling
trying to correct for perceived model misspecifications that are not really
there. This is why I recommend relying primarily on simulation-based model
diagnostics (e.g. VPCs) and as far as possible account for censoring that
affects the original dataset. As pointed out by Karlsson & Savic a
simulation/re-estimation based approach can also be used to investigate the
expected behavior for DV vs. PRED plots for a particular model and dataset
(e.g. mirror plots in Xpose). Note that to my knowledge there is yet no
automated way to handle censoring in this context (clearly doable if anyone
wants to develop a nifty implementation of that).
If we leave the DV vs. PRED plot case, there are many other instances where we
use scatter plots where it is much less clear what can be considered the
independent variable and yet other cases where the assumption that the
x-variable is without error is violated in a way that makes the results hard to
interpret. One instance of the latter is when exposure-response is studied by
plotting observed PD response versus observed trough plasma concentrations.
This is already a way too long email so I will not deep dive into that problem
as well.
Best regards,
Martin Bergstrand, Ph.D.
Principal Consultant
Pharmetheus AB
[email protected]<mailto:[email protected]>
http://www.pharmetheus.com/
On Thu, Aug 17, 2023 at 12:44 PM Gobburu, Joga
<[email protected]<mailto:[email protected]>> wrote:
Dear Friends – Observations versus population predicted is considered a
standard diagnostic plot in our field. I used to place observations on the
x-axis and predictions on the yaxis. Then I was pointed to a publication from
ISOP
( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/)
which recommended plotting predictions on the xaxis and observations on the
yaxis. To the best of my knowledge, there was no justification provided. It did
question my decades old practice, so I did some thinking and digging. Thought
to share it here so others might benefit from it. If this is obvious to you
all, then I can say I am caught up!
1. We write our models as observed = predicted + random error; which can be
interpreted to be in the form: y = f(x) + random error. It is technically not
though. Hence predicted goes on the xaxis, as it is free of random error. It is
considered a correlation plot, which makes plotting either way acceptable. This
is not so critical as the next one.
2. However, there is a statistical reason why it is important to keep
predictions on the xaxis. Invariably we always add a loess trend line for these
diagnostic plots. To demonstrate the impact, I took a simple iv bolus single
dose dataset and compared both approaches. The results are available at this
link: https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf. I
used Pumas software, but the scientific underpinning is agnostic to software.
See the two plots on Pages 5 and 6. The interpretation of the bias between the
two approaches is different. This is the statistical reason why it matters to
plot predictions on the xaxis.
Joga Gobburu
University of Maryland
This communication is confidential and is only intended for the use of the
individual or entity to which it is directed. It may contain information that
is privileged and exempt from disclosure under applicable law. If you are not
the intended recipient please notify us immediately. Please do not copy it or
disclose its contents to any other person.
Any personal data will be processed in accordance with Pharmetheus' privacy
notice, available https://pharmetheus.com/privacy-policy/.
Dear Martin and NMusers,
With reference to Martin saying that “A greater problem with these plots is
the commonly held expectation that for a "good model" a smooth or
regression line should align with the line of unity. Though this seems
intuitive it is a flawed assumption”, I would like to defend the assumption
that in a relevant number of cases it is reasonable to assume that the plot
of observations (y-axis) versus predictions (x-axis) is expected to have a
regression line going through unity.
First, to be clear, I do not disagree with anything said in the classic
Karlsson and Savic 2007 paper. With any model where random effects enter
into the model nonlinearly, the plot of observations (y-axis) versus PRED
(x-axis) can have trends which look like model misspecification, even if
the data-generating model for observations has exactly the same parameter
values as the diagnostics-generating model. This is because the PRED
signifies the prediction for the median individual with random effect
values at zero, which is different from the mean prediction. And the local
regression line, as the name implies, trends around the local mean of the
observed data.
So basically we need a PRED-like data item that reflects the expected mean
population prediction, integrated over the possible individual random
effects values. Lucky for us, this data item exists, and is called EPRED.
The data item was not available at the time the Savic and Karlsson paper
was published. It is available now. The EPRED solves the problems caused by
model nonlinearity and high inter-individual variability by integrating
over the random effects, given the parameter estimates. I do note that it
does not solve the problems of censoring and dose adaptation.
So, because the local regression line reflects the mean values and the
EPRED data item reflects mean values, I contend that in the absence of
censoring and dose adaptation, the plot of observations (y-axis) versus
EPRED (x-axis) can be expected to have a regression line that mostly agrees
with the line of unity, with some caveats (see below). This expectation
holds even if the observations are not symmetrically distributed over the
mean, because the local regression simply follows the mean. Moreover, if
the model accounts for censoring and dose adaptation, then it would be
possible to manually code and calculate the simulation-predicted population
mean values (e.g. simulating 1000 datasets, and for each observation taking
the mean simulated value that accounts for censoring and dose adaptation),
and use those in x-axis. Also to note, in this NMusers message group I
focused on the EPRED data item because it is NONMEM-specific, but the
general concept is software-agnostic: Having Monte Carlo-generated
population mean predictions on the x-axis should result in the plot of
observations (y-axis) versus predictions (x-axis) trending through the line
of unity.
Caveat 1: Because of random variability, it cannot be expected that the
regression line always goes perfectly through the line of unity. This
should come as no surprise, e.g. it is also not expected that a VPC will
have observed data percentiles always perfectly in the middle of the
simulation-generated confidence intervals for prediction intervals.
Caveat 2: For small datasets, it is possible that there be additional bias
in the plot of observations (y-axis) versus EPRED (x-axis) if the
data-generating model is exactly the same as the diagnostics-generating
model, because the data-generating model is not necessarily the one that
best agrees with the data. Illustrative example: Suppose we simulate a
dataset of 10 individual concentration-profiles at steady-state with high
drug accumulation, thus the concentrations will be highly dependent on the
clearance parameter. It is entirely possible that the 10 simulated
clearance random effects (eta) values will have a mean that is either above
or below zero to some relevant extent, thus greatly affecting the
steady-state predictions. Thus, as a result there could be an apparent,
systematic disagreement between the simulated data (observations, y-axis)
and the EPRED (x-axis) because of clearance random effects trending above
or below zero due to random variability. This problem could be remedied by
fitting a model to the simulated data, and using that model for generating
the diagnostics. At larger dataset sizes, the problem disappears because it
becomes less and less likely for the mean of the random effects to deviate
from zero to a relevant extent. This same caveat also exists for the VPC
diagnostic; if one simulates a small dataset as observations, and then
produces a VPC from the same simulation model (without fitting the model to
the previously simulated data), then there may be apparent misspecification
in the resulting VPC figure.
Supplemental remark 1: To illustrate how the loess follows mean even if the
data are not symmetrically distributed, the following R code snippet may be
relevant. It simulates 100 observations from lognormal distribution, and
then compares the smoothing curves from "loess" and "mgcv::gam" functions
to the theoretically expected mean value. There is a close agreement
between the loess curves and the analytically calculated mean value.
library(tidyverse)
with(list(omega=0.6),
map_dfr(1:100,~tibble(x=1:10,y=exp(rnorm(10,0,omega)))) %>%
mutate(theoretical=exp(omega^2/2)) %>%
ggplot(aes(x,y))+geom_point()+
geom_smooth(method="loess",col=3)+geom_smooth(method=mgcv::gam),col=4)+
geom_line(aes(y=theoretical),col=2)
ps. The usual disclaimer, the opinions expressed in this message are mine
alone, and not necessarily those of my employer.
Best wishes,
Pyry Välitalo
PK Assessor at Finnish Medicines Agency
Quoted reply history
On Fri, 18 Aug 2023 at 10:59, Martin Bergstrand <
[email protected]> wrote:
> Dear Joga and all,
>
> Joga makes a valuable point that all pharmacometricians should be aware
> of. Standard methodology for regression assumes that the x-variable is
> without error (loess, linear regression etc.). Note that it is the same for
> NLME models i.e. we generally assume that our independent variables e.g.
> time, covariates etc. are without error.
>
> For DV vs. PRED plots it is common practice, even among those that do not
> know why, to plot PRED on the x-axis and DV on the y-axis. A greater
> problem with these plots is the commonly held expectation that for a "good
> model" a smooth or regression line should align with the line of unity.
> Though this seems intuitive it is a flawed assumption. This issue was
> clearly pointed out by Mats Karlsson and Rada Savic in their 2007 paper
> titled "Diagnosing Model Diagnostics''. For simple well-behaved examples
> you will see an alignment around the line of unity for DV vs. PRED plots.
> However, there are several factors that contribute to an expected deviation
> from this expectation:
> (1) Censoring (e.g. censoring of observations < LLOQ)
> - In this case DVs are capped at LLOQ but PRED values are not. This
> makes it perfectly expected that there will be a deviation from alignment
> around the line of unity in the lower range.
> (2) Strong non-linearities
> - The more nonlinear the modelled system is, the greater the expected
> deviation from the line of unity. Especially in combination with
> significant ETA correlations.
> (3) High variability
> - With higher between/within subject variability (e.g. IIV and RUV) that
> isn't normally distributed (e.g. exponential distributions) will result in
> an expected deviation from the line of unity. Note: this is a form of
> non-linearity so it may fall under the above category.
> (4) Adaptive designs (e.g. TDM dosing)
> - Listed in the original paper by Karlsson & Savic but I have not been
> able to recreate an issue in this case.
>
> I am rather sure that many thousands of hours have been spent on modeling
> trying to correct for perceived model misspecifications that are not really
> there. This is why I recommend relying primarily on simulation-based model
> diagnostics (e.g. VPCs) and as far as possible account for censoring that
> affects the original dataset. As pointed out by Karlsson & Savic a
> simulation/re-estimation based approach can also be used to investigate the
> expected behavior for DV vs. PRED plots for a particular model and dataset
> (e.g. mirror plots in Xpose). Note that to my knowledge there is yet
> no automated way to handle censoring in this context (clearly doable if
> anyone wants to develop a nifty implementation of that).
>
> If we leave the DV vs. PRED plot case, there are many other instances
> where we use scatter plots where it is much less clear what can be
> considered the independent variable and yet other cases where the
> assumption that the x-variable is without error is violated in a way that
> makes the results hard to interpret. One instance of the latter is when
> exposure-response is studied by plotting observed PD response versus
> observed trough plasma concentrations. This is already a way too long email
> so I will not deep dive into that problem as well.
>
> Best regards,
>
>
> Martin Bergstrand, Ph.D.
>
> Principal Consultant
>
> Pharmetheus AB
>
> [email protected]
>
> www.pharmetheus.com
>
>
> On Thu, Aug 17, 2023 at 12:44 PM Gobburu, Joga <[email protected]>
> wrote:
>
>> Dear Friends – Observations versus population predicted is considered a
>> standard diagnostic plot in our field. I used to place observations on the
>> x-axis and predictions on the yaxis. Then I was pointed to a publication
>> from ISOP (
>> https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/)
>> which recommended plotting predictions on the xaxis and observations on the
>> yaxis. To the best of my knowledge, there was no justification provided. It
>> did question my decades old practice, so I did some thinking and digging.
>> Thought to share it here so others might benefit from it. If this is
>> obvious to you all, then I can say I am caught up!
>>
>>
>>
>> 1. We write our models as observed = predicted + random error; which
>> can be interpreted to be in the form: y = f(x) + random error. It is
>> technically not though. Hence predicted goes on the xaxis, as it is free
>> of
>> random error. It is considered a correlation plot, which makes plotting
>> either way acceptable. This is not so critical as the next one.
>> 2. However, there is a statistical reason why it is important to keep
>> predictions on the xaxis. Invariably we always add a loess trend line for
>> these diagnostic plots. To demonstrate the impact, I took a simple iv
>> bolus
>> single dose dataset and compared both approaches. The results are
>> available
>> at this link:
>> https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf.
>> I used Pumas software, but the scientific underpinning is agnostic to
>> software. See the two plots on Pages 5 and 6. The interpretation of the
>> bias between the two approaches is different. This is the statistical
>> reason why it matters to plot predictions on the xaxis.
>>
>>
>>
>> Joga Gobburu
>>
>> University of Maryland
>>
>
> *This communication is confidential and is only intended for the use of
> the individual or entity to which it is directed. It may contain
> information that is privileged and exempt from disclosure under applicable
> law. If you are not the intended recipient please notify us immediately.
> Please do not copy it or disclose its contents to any other person.*
> *Any personal data will be processed in accordance with Pharmetheus'
> privacy notice, available here https://pharmetheus.com/privacy-policy/.*
>
Thank you, Wilbert. I was not aware of that publication – thank you for sharing!
J
Quoted reply history
From: Wilbert de Witte <[email protected]>
Date: Friday, August 18, 2023 at 2:08 AM
To: Gobburu, Joga <[email protected]>
Cc: James G Wright <[email protected]>, [email protected]
<[email protected]>
Subject: Re: [NMusers] Observed (yaxis) vs Predicted (xaxis) Diagnostic Plot -
Scientific basis.
You don't often get email from [email protected]. Learn why this is
https://aka.ms/LearnAboutSenderIdentification
Hi Joga,
Fully agree on this, unfortunately it is still often shown the other way around
which is at least confusing.
There is a publication on this very topic
https://www.sciencedirect.com/science/article/abs/pii/S0304380008002305
that arrives at the same conclusion and can be helpful.
Best,
Wilbert
Op do 17 aug 2023 om 19:47 schreef Gobburu, Joga
<[email protected]<mailto:[email protected]>>:
Dear James – how have you been?
Yes, you said it most eloquently. Its not about plotting per se but “the
problem is really that the loess line is fitting noise in the wrong direction
if the observed is actually on the x-axis”. Thank you…J
From: James G Wright <[email protected]<mailto:[email protected]>>
Date: Thursday, August 17, 2023 at 7:16 AM
To: Gobburu, Joga
<[email protected]<mailto:[email protected]>>,
[email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>>
Subject: Re: [NMusers] Observed (yaxis) vs Predicted (xaxis) Diagnostic Plot -
Scientific basis.
You don't often get email from
[email protected]<mailto:[email protected]>. Learn why this is
https://aka.ms/LearnAboutSenderIdentification
CAUTION: This message originated from a non-UMB email system. Hover over any
links before clicking and use caution opening attachments.
So whichever axis the observed data is plotted on is parallel to the direction
of noise (random residual error). When you fit the loess line, I think it will
generally assume noise is vertical i.e. parallel to the y-axis. So the problem
is really that the loess line is fitting noise in the wrong direction if the
observed is actually on the x-axis ... which means you are right, the observed
needs to go on the y-axis and deviations need to be interpreted parallel to the
y-axis.
Kind regards, James
https://product.popypkpd.com/
PS Of course, if you were to fit a loess line with horizontal noise and
observed data on the x-axis, you should reach identical conclusions to the
conventional vertical noise and observed data on the y-axis.
On 17/08/2023 11:35, Gobburu, Joga wrote:
Dear Friends – Observations versus population predicted is considered a
standard diagnostic plot in our field. I used to place observations on the
x-axis and predictions on the yaxis. Then I was pointed to a publication from
ISOP
( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/)
which recommended plotting predictions on the xaxis and observations on the
yaxis. To the best of my knowledge, there was no justification provided. It did
question my decades old practice, so I did some thinking and digging. Thought
to share it here so others might benefit from it. If this is obvious to you
all, then I can say I am caught up!
1. We write our models as observed = predicted + random error; which can be
interpreted to be in the form: y = f(x) + random error. It is technically not
though. Hence predicted goes on the xaxis, as it is free of random error. It is
considered a correlation plot, which makes plotting either way acceptable. This
is not so critical as the next one.
2. However, there is a statistical reason why it is important to keep
predictions on the xaxis. Invariably we always add a loess trend line for these
diagnostic plots. To demonstrate the impact, I took a simple iv bolus single
dose dataset and compared both approaches. The results are available at this
link: https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf. I
used Pumas software, but the scientific underpinning is agnostic to software.
See the two plots on Pages 5 and 6. The interpretation of the bias between the
two approaches is different. This is the statistical reason why it matters to
plot predictions on the xaxis.
Joga Gobburu
University of Maryland
--
James G Wright PhD,
Scientist, Wright Dose Ltd
Tel: UK (0)772 5636914
Hi Pyry + Martin + NMusers,
I wanted to make 2 comments on this discussion. Pyry wrote:
With reference to Martin saying that “A greater problem with these plots is the
commonly held expectation that for a "good model" a smooth or regression line
should align with the line of unity. Though this seems intuitive it is a flawed
assumption”, I would like to defend the assumption that in a relevant number of
cases it is reasonable to assume that the plot of observations (y-axis) versus
predictions (x-axis) is expected to have a regression line going through unity.
Comment 1
I see similarity here between the two positions:
DV v IPRED and DV v PRED plots are sometimes misleading - Martin
DV v IPRED and DV v PRED plots are often sound - Pyry
Both statements are true.
There are 4 “errors" we can encounter, as Martin rightly alluded to:
1) The DV v IPRED looks good, but the model is not (slides 19-23 and slide 47).
2) The DV v PRED looks good, but the model is not (slides 19-23 and slide 47).
3) The DV v IPRED looks poor, but the model is right (slide 46).
4) The DV v PRED looks poor, but the model is right.
The slide numbers refer to my PAGE 2017 presentation, slides available here
( https://www.page-meeting.org/pdf_assets/1203-Maloney%20-%20PAGE%202017%20-%208%20June%2017%20-%20final.pdf)
and talk available here ( https://www.youtube.com/watch?v=E3T2p6Mv0Xc).
The “silly” examples in the presentation simply serve to show how shrinkage
and/or model flexibility via random effects can make these graphs misleading,
even with linear mixed models. Thus, as always, we need to understand our data,
our models, and hence the usefulness of such plots; there is no “golden rule”
that they are (or are not) useful. In addition, I personally am “offended" ;-)
when an author chooses to present these graphs as “evidence" that their model
is sound; either provide/show an extensive set of model diagnostics, PPCs,
individual fits, residual analyses, LOO-CV results etc. that truly
assess/challenge the model, or do not bother! (rant over!)
To comment on DV v PRED, an old adage is that is can support the
appropriateness of the “structural” model. This is not true. Consider the
following PK data, where the prediction at 2 hr and 12 hr are the same (a flat
PK profile!).
Dose 2hr 12hr PRED
1 2 0.5 1
10 20 5 10
100 200 50 100
1000 2000 500 1000
Here a plot of DV v PRED would look great (on the log scale), even though the
"PK model” is complete nonsense. So again, these basic plots (and the “archaic”
value given to them) can be inappropriate. Indeed, simply by adding more and
more parameters to our model, we can always “fix” these plots, however foolish
these model additions are.
Comment 2
I would argue that residuals (like DV - IPRED) versus predicted (or other
covariates like time, dose etc.) are much more useful than DV v IPRED, since
now our y-axis quantifies the magnitude of the lack of fit; we can much more
easily determine the magnitude of the lack of fit for both individual
observations and any smooth through the data. A bias may be evident from the
smooth, but much more important is the magnitude of that bias. Similarly,
having a “flat" smooth is not great if there are many observations being
grossly under predicted, but are “balance” by as many observations being
grossly over-predicted (...many years ago I recall seeing a “final PK model”
WRES v Time plot with a smooth that looked perfectly flat…it looked great,
until I saw that the y-axis ran from +20 to -20!).
Best wishes,
Al
Alan Maloney PhD
Consultant Pharmacometrician
Free Book! - Drug Development For Patients - see www.alanmaloney.com
http://www.alanmaloney.com/ - please read, send comments and support patients!
Phone: +46 734 04 38 49
E-mail:[email protected]
Quoted reply history
> On 23 Aug 2023, at 09:35, Pyry Välitalo <[email protected]> wrote:
>
> Dear Martin and NMusers,
>
> With reference to Martin saying that “A greater problem with these plots is
> the commonly held expectation that for a "good model" a smooth or regression
> line should align with the line of unity. Though this seems intuitive it is a
> flawed assumption”, I would like to defend the assumption that in a relevant
> number of cases it is reasonable to assume that the plot of observations
> (y-axis) versus predictions (x-axis) is expected to have a regression line
> going through unity.
>
> First, to be clear, I do not disagree with anything said in the classic
> Karlsson and Savic 2007 paper. With any model where random effects enter into
> the model nonlinearly, the plot of observations (y-axis) versus PRED (x-axis)
> can have trends which look like model misspecification, even if the
> data-generating model for observations has exactly the same parameter values
> as the diagnostics-generating model. This is because the PRED signifies the
> prediction for the median individual with random effect values at zero, which
> is different from the mean prediction. And the local regression line, as the
> name implies, trends around the local mean of the observed data.
>
> So basically we need a PRED-like data item that reflects the expected mean
> population prediction, integrated over the possible individual random effects
> values. Lucky for us, this data item exists, and is called EPRED. The data
> item was not available at the time the Savic and Karlsson paper was
> published. It is available now. The EPRED solves the problems caused by model
> nonlinearity and high inter-individual variability by integrating over the
> random effects, given the parameter estimates. I do note that it does not
> solve the problems of censoring and dose adaptation.
>
> So, because the local regression line reflects the mean values and the EPRED
> data item reflects mean values, I contend that in the absence of censoring
> and dose adaptation, the plot of observations (y-axis) versus EPRED (x-axis)
> can be expected to have a regression line that mostly agrees with the line of
> unity, with some caveats (see below). This expectation holds even if the
> observations are not symmetrically distributed over the mean, because the
> local regression simply follows the mean. Moreover, if the model accounts for
> censoring and dose adaptation, then it would be possible to manually code and
> calculate the simulation-predicted population mean values (e.g. simulating
> 1000 datasets, and for each observation taking the mean simulated value that
> accounts for censoring and dose adaptation), and use those in x-axis. Also to
> note, in this NMusers message group I focused on the EPRED data item because
> it is NONMEM-specific, but the general concept is software-agnostic: Having
> Monte Carlo-generated population mean predictions on the x-axis should result
> in the plot of observations (y-axis) versus predictions (x-axis) trending
> through the line of unity.
>
> Caveat 1: Because of random variability, it cannot be expected that the
> regression line always goes perfectly through the line of unity. This should
> come as no surprise, e.g. it is also not expected that a VPC will have
> observed data percentiles always perfectly in the middle of the
> simulation-generated confidence intervals for prediction intervals.
>
> Caveat 2: For small datasets, it is possible that there be additional bias in
> the plot of observations (y-axis) versus EPRED (x-axis) if the
> data-generating model is exactly the same as the diagnostics-generating
> model, because the data-generating model is not necessarily the one that best
> agrees with the data. Illustrative example: Suppose we simulate a dataset of
> 10 individual concentration-profiles at steady-state with high drug
> accumulation, thus the concentrations will be highly dependent on the
> clearance parameter. It is entirely possible that the 10 simulated clearance
> random effects (eta) values will have a mean that is either above or below
> zero to some relevant extent, thus greatly affecting the steady-state
> predictions. Thus, as a result there could be an apparent, systematic
> disagreement between the simulated data (observations, y-axis) and the EPRED
> (x-axis) because of clearance random effects trending above or below zero due
> to random variability. This problem could be remedied by fitting a model to
> the simulated data, and using that model for generating the diagnostics. At
> larger dataset sizes, the problem disappears because it becomes less and less
> likely for the mean of the random effects to deviate from zero to a relevant
> extent. This same caveat also exists for the VPC diagnostic; if one simulates
> a small dataset as observations, and then produces a VPC from the same
> simulation model (without fitting the model to the previously simulated
> data), then there may be apparent misspecification in the resulting VPC
> figure.
>
> Supplemental remark 1: To illustrate how the loess follows mean even if the
> data are not symmetrically distributed, the following R code snippet may be
> relevant. It simulates 100 observations from lognormal distribution, and then
> compares the smoothing curves from "loess" and "mgcv::gam" functions to the
> theoretically expected mean value. There is a close agreement between the
> loess curves and the analytically calculated mean value.
> library(tidyverse)
> with(list(omega=0.6),
> map_dfr(1:100,~tibble(x=1:10,y=exp(rnorm(10,0,omega)))) %>%
> mutate(theoretical=exp(omega^2/2)) %>%
> ggplot(aes(x,y))+geom_point()+
> geom_smooth(method="loess",col=3)+geom_smooth(method=mgcv::gam),col=4)+
> geom_line(aes(y=theoretical),col=2)
>
> ps. The usual disclaimer, the opinions expressed in this message are mine
> alone, and not necessarily those of my employer.
>
> Best wishes,
> Pyry Välitalo
> PK Assessor at Finnish Medicines Agency
>
> On Fri, 18 Aug 2023 at 10:59, Martin Bergstrand
> <[email protected]
> <mailto:[email protected]>> wrote:
>> Dear Joga and all,
>>
>> Joga makes a valuable point that all pharmacometricians should be aware of.
>> Standard methodology for regression assumes that the x-variable is without
>> error (loess, linear regression etc.). Note that it is the same for NLME
>> models i.e. we generally assume that our independent variables e.g. time,
>> covariates etc. are without error.
>>
>> For DV vs. PRED plots it is common practice, even among those that do not
>> know why, to plot PRED on the x-axis and DV on the y-axis. A greater problem
>> with these plots is the commonly held expectation that for a "good model" a
>> smooth or regression line should align with the line of unity. Though this
>> seems intuitive it is a flawed assumption. This issue was clearly pointed
>> out by Mats Karlsson and Rada Savic in their 2007 paper titled "Diagnosing
>> Model Diagnostics''. For simple well-behaved examples you will see an
>> alignment around the line of unity for DV vs. PRED plots. However, there are
>> several factors that contribute to an expected deviation from this
>> expectation:
>> (1) Censoring (e.g. censoring of observations < LLOQ)
>> - In this case DVs are capped at LLOQ but PRED values are not. This makes
>> it perfectly expected that there will be a deviation from alignment around
>> the line of unity in the lower range.
>> (2) Strong non-linearities
>> - The more nonlinear the modelled system is, the greater the expected
>> deviation from the line of unity. Especially in combination with significant
>> ETA correlations.
>> (3) High variability
>> - With higher between/within subject variability (e.g. IIV and RUV) that
>> isn't normally distributed (e.g. exponential distributions) will result in
>> an expected deviation from the line of unity. Note: this is a form of
>> non-linearity so it may fall under the above category.
>> (4) Adaptive designs (e.g. TDM dosing)
>> - Listed in the original paper by Karlsson & Savic but I have not been able
>> to recreate an issue in this case.
>>
>> I am rather sure that many thousands of hours have been spent on modeling
>> trying to correct for perceived model misspecifications that are not really
>> there. This is why I recommend relying primarily on simulation-based model
>> diagnostics (e.g. VPCs) and as far as possible account for censoring that
>> affects the original dataset. As pointed out by Karlsson & Savic a
>> simulation/re-estimation based approach can also be used to investigate the
>> expected behavior for DV vs. PRED plots for a particular model and dataset
>> (e.g. mirror plots in Xpose). Note that to my knowledge there is yet no
>> automated way to handle censoring in this context (clearly doable if anyone
>> wants to develop a nifty implementation of that).
>>
>> If we leave the DV vs. PRED plot case, there are many other instances where
>> we use scatter plots where it is much less clear what can be considered the
>> independent variable and yet other cases where the assumption that the
>> x-variable is without error is violated in a way that makes the results hard
>> to interpret. One instance of the latter is when exposure-response is
>> studied by plotting observed PD response versus observed trough plasma
>> concentrations. This is already a way too long email so I will not deep dive
>> into that problem as well.
>>
>> Best regards,
>>
>>
>> Martin Bergstrand, Ph.D.
>>
>> Principal Consultant
>>
>> Pharmetheus AB
>>
>> [email protected] <mailto:[email protected]>
>> www.pharmetheus.com http://www.pharmetheus.com/
>>
>>
>>
>> On Thu, Aug 17, 2023 at 12:44 PM Gobburu, Joga <[email protected]
>> <mailto:[email protected]>> wrote:
>>> Dear Friends – Observations versus population predicted is considered a
>>> standard diagnostic plot in our field. I used to place observations on the
>>> x-axis and predictions on the yaxis. Then I was pointed to a publication
>>> from ISOP
>>> ( https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5321813/figure/psp412161-fig-0001/)
>>> which recommended plotting predictions on the xaxis and observations on
>>> the yaxis. To the best of my knowledge, there was no justification
>>> provided. It did question my decades old practice, so I did some thinking
>>> and digging. Thought to share it here so others might benefit from it. If
>>> this is obvious to you all, then I can say I am caught up!
>>>
>>> We write our models as observed = predicted + random error; which can be
>>> interpreted to be in the form: y = f(x) + random error. It is technically
>>> not though. Hence predicted goes on the xaxis, as it is free of random
>>> error. It is considered a correlation plot, which makes plotting either way
>>> acceptable. This is not so critical as the next one.
>>> However, there is a statistical reason why it is important to keep
>>> predictions on the xaxis. Invariably we always add a loess trend line for
>>> these diagnostic plots. To demonstrate the impact, I took a simple iv bolus
>>> single dose dataset and compared both approaches. The results are available
>>> at this link:
>>> https://github.com/jgobburu/public_didactic/blob/main/iv_sd.html.pdf. I
>>> used Pumas software, but the scientific underpinning is agnostic to
>>> software. See the two plots on Pages 5 and 6. The interpretation of the
>>> bias between the two approaches is different. This is the statistical
>>> reason why it matters to plot predictions on the xaxis.
>>>
>>> Joga Gobburu
>>> University of Maryland
>>
>> This communication is confidential and is only intended for the use of the
>> individual or entity to which it is directed. It may contain information
>> that is privileged and exempt from disclosure under applicable law. If you
>> are not the intended recipient please notify us immediately. Please do not
>> copy it or disclose its contents to any other person.
>> Any personal data will be processed in accordance with Pharmetheus' privacy
>> notice, available here https://pharmetheus.com/privacy-policy/.