From: chyihung.hsu@pharma.novartis.com
Subject: [NMusers] Significance?
Date: Tue, 2 Apr 2002 16:56:12 -0400
Dear NONMEM users,
I am fitting a 3-comp. population model to a data set. Based on the base
model (model without any covariate), significant differences in post-hoc
estimates of CL and their corresponding etas were suggested between 2
groups of subjects (Groups A and B). But adding the grouping covariate to
the TVCL part of the base model did not seem to help, and the coefficient
and the reduction in OBJ were far from statistical significance. On the
contrary, this agreed with previous diagnostic and non-compartmental
results. Should significant coefficient be expected? any suggestions?
Thanks in advance.
Best regards,
Chyi-Hung
Additional data and model information:
Data: intensive samples.
Model: random effects: CL,V1,Q2, and V2 were assumed to be log-normally
distributed, with
var-cov matrix=block(block(CL,V1), block(V2,Q2)),
error = additive + multiplicative, and
V2 > V3.
Estimation: CONDITIONAL INTERACTION.
Significance?
From: LSheiner
Subject: Re: [NMusers] Significance?
Date: Tue, 02 Apr 2002 13:24:46 -0800
Sounds like,a contradiction -- The post-hoc clearances are different between the
groups,
but including the group indcator in the clearance model does not help. I'd bet that
if both findings are clear (ie,., the post-hoc clearances really are quite different,
while inclusion of the group effect in the model really yields no appreciable
change in OBJ), then you've made some interesting mistake. The
question clearly is not which to believe, but why the contradiction?
LBS.
chyihung.hsu@pharma.novartis.com wrote:
--
_/ _/ _/_/ _/_/_/ _/_/_/ Professor Lewis B Sheiner, MD
_/ _/ _/ _/_ _/_/ mail: Box 0626, UCSF, SF, CA,94143
_/ _/ _/ _/ _/ courier: Rm C255, UCSF, SF, CA,94122
_/_/ _/_/ _/_/_/ _/ 415-476-1965 (v), 415-476-2796 (fax)
From: "Piotrovskij, Vladimir [PRDBE]"
Subject: RE: [NMusers] Significance?
Date: Wed, 3 Apr 2002 12:38:47 +0200
Chyi-Hung,
In general, posthoc estimates give you only some hints about potential fixed effects. If
you cannot prove these by the likelihood ratio test or other statistical tools it is not a
problem. Do not consider this as a contradiction.
Best regards,
Vladimir
From: LSheiner
Subject: Re: [NMusers] Significance?
Date: Wed, 03 Apr 2002 11:23:18 -0800
Quite the contrary. In my experience, the obj function is far more sensitive to
differences in goodness of fit than the graphics.
I will often find a marginal increment in goodness of fit
(say <=10 points) with some modeling change, but
be unable to find any picture that looks "better."
I do not accept such changes.
In contrast, I cannot recall any case
in which I could see a clear signal in a picture, which I could not then
confirm by making the indicated model change and finding that the obj
function changed markedly -- Except where I had made some kind of
modeling or coding error!
I would definitely not let siuch a thing pass without
a very thorough investigation.
LBS
--
_/ _/ _/_/ _/_/_/ _/_/_/ Professor Lewis B Sheiner, MD
_/ _/ _/ _/_ _/_/ mail: Box 0626, UCSF, SF, CA,94143
_/ _/ _/ _/ _/ courier: Rm C255, UCSF, SF, CA,94122
_/_/ _/_/ _/_/_/ _/ 415-476-1965 (v), 415-476-2796 (fax)
From: "Piotrovskij, Vladimir [PRDBE]"
Subject: RE: [NMusers] Significance?
Date: Thu, 4 Apr 2002 08:42:32 +0200
I agree: OBJ is more sensitive than the graphics. And it also tells you something about
the significance of an effect whereas the graphics doesn't. As to whether accept an
effect or not, I think the drop in OBJ by 10 is a good signal to accept (P<0.005) irrespective
to what the pictures show. You may eventually not generate the plots. Of course, you have
to be sure that there is no mistakes in the control streem...
Best regards,
Vladimir
From: "Bachman, William"
Subject: RE: [NMusers] Significance?
Date: Thu, 4 Apr 2002 09:42:04 -0500
Perhaps I am a bit naive, but, I would not recommend using just one metric (obj fcn versus plots)
for determining goodness of fit (or model discrimination) in any case. I consider the change in
objective function, change in plots AND change in variance estimates. Any one alone could
be misleading. Maybe I missed the point here.
From: "Piotrovskij, Vladimir [PRDBE]"
Subject: RE: [NMusers] Significance?
Date: Fri, 5 Apr 2002 12:24:31 +0200
Bill,
Let me be more concrete. Suppose we are exploring the sex differences in CL. Box plots show this
and also OMEGA(CL) is reduced when we run the model with the fixed effect of SEX factor, however,
the drop in OBJ is insufficient to accept the effect at the preselected level of significance (say, 0.005).
This may happen if we have too few females in the study group (not unfrequent case). What would you do?
From: "Bachman, William"
Subject: RE: [NMusers] Significance?
Date: Mon, 8 Apr 2002 09:21:00 -0400
Vladimir,
In your specific example of too few females [the story of my life ;)], I would be reluctant to accept the
sex difference. (Unless there is supporting evidence such as a known difference in metabolic activity for
males versus females from prior biochemical studies).
Bill
From: "Wang, Bing"
Subject: RE: [NMusers] Significance?
Date: Mon, 15 Apr 2002 11:59:21 -0700
Chyi-Hung,
Some questions/recommendations:
1. When you separate the two CL, (CL1 for Group A and CL2 for Group B), are
these two CL estimates greatly different? (Although you mentioned OBJ did
not change much, what about actual CL estimates?)
2. Try to separate the residual errors for Groups A and B in the $ERROR
block. If you do so, how different are those ERR estimates? Is it possible
that one group could be better described by a model other than the 3-c
model...? Do you expect the PK in Groups A and B to be the same?
3. If PK is linear, the modeling shall give results fairly close to the NC
analysis. Could the model be over parameterized somehow?
4. It may help to re-check the coding, and to confirm the comparison of OBJs
is fair: FOCE vs FOCE, but not FO vs FOCE...
5. If overall goodness-of-fit graphs look OK and the OBJ change says nothing
significant, and noncompartmental analysis (AUCs) did not show a trend
between Groups A and B, continue your modeling without considering such
group difference. Once you reach the final model, in your model validation
step, separate the CLs to see if it makes any difference there...
Regards.
Bing Wang
Amgen
*******