RE: Models that abort before convergence
Leonid et al, I'm a little confused by this discussion. To make an analogy, assume that drug company A has a wonderful theory that drug B will treat a disease. Theory makes sense by your favorite epistemology criteria etc. But of course, being good scientists, we know that theories must be verified, so we do an experiment, and the data suggest that the theory is wrong. Most of us would criticize as unscientific someone who who discarded the data (didn't point out flaws in the data, didn't provide opposing data, simply discounted it) in favor of continuing to believe the theory. Why do we not apply the same standards here? Theory says that models that do not converge (or fail covariance) are "bad". Data (that so far as I know no one has found to be flawed, nor provided opposing data) suggests that, by at least one criteria (same parameter estimates, same SD of parameter estimates) there are no important differences. I don't disagree that failing a covariance step, or failing to converge provide information about a model. But it doesn't seem to be informative about what we probably really care about -does the line go through the points, how confident are we WRT the precision of the parameters and is the model predictive. I'm not sure if the small number of published examples (of bootstrap with ~500 samples) are a small number of anecdotes or a small number of trials with N ~ 500, but I've run 5 or so myself and found the same to be consistently the case. That is, a successful covariance step is not informative WRT the parameter values or their precision. I suspect others have similar experience. If there are other "studies"/anecdotes with different conclusions, someone should publish them. Otherwise, it seems like we are obligated to abandon this theory in favor of the data. Mark Sale MD
Next Level Solutions, LLC
www.NextLevelSolns.com
919-846-9185
Quoted reply history
-------- Original Message --------
Subject: RE: [NMusers] Models that abort before convergence
From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Date: Tue, November 18, 2008 11:13 pm
To: [EMAIL PROTECTED], [email protected] ,
[EMAIL PROTECTED]
Dennis,
I do not support extreme views (from places where people walk upside down
:) ) that Nonmem error messages should be ignored: they serve the useful
purpose to alert when Nonmem is having some difficulties, and should always
be part of the picture. If the data looks good, model is simple, then we
need to look for the reason for the poor convergence. Sometimes it helps to
use SIGDIG= 5 or 6 to get 3 significant digits precision. But if you are
working on the limit of the algorithms (as implemented) abilities:
nonlinear model + stiff differential equations + large range of doses and
concentrations, etc., then you face the situation when you cannot force
convergence even if you try hard. On my recent project, none of the
intermediate model converged even though bootstrap provided pretty narrow
CI (so it does not look like over-parametrized model), all diagnostic plots
were good, and the visual predictive check was reasonable. Then you just
blame the algorithm and move on. You loose the ability to justify your
covariate selection based on the objective function drop (which is not a
good idea any way), and may need to provide a little bit more detailed
investigation to convince reviewers (regulatory and/or journal) that the
model is adequate for the intended purpose.
Thanks
Leonid
Original Message:
-----------------
From: Dennis Fisher [EMAIL PROTECTED]
Date: Tue, 18 Nov 2008 11:21:23 -0800
To: [email protected] , [EMAIL PROTECTED]
Subject: [NMusers] Models that abort before convergence
Colleagues,
I am curious as to your thoughts about a particular NONMEM issue. I
often find myself in a situation where a complex model does not
converge to 3 digits ("no of digits: unreportable") yet the objective
function is markedly better than a previous model and graphics suggest
that the model is quite good (and better than the previous one). Nick
Holford has advocated (and I agree) that NONMEM's SE's have minimal
utility and the inability to calculate them is not important.
However, I have not seen similar discussion about whether one can /
should accept a model that did not converge.
The particular situation that I dealing with at the moment is that a
dataset that I am analyzing yielded a series of results that did not
converge as I added parameters (despite an improving fit and a marked
decrease in the objective function), then yet a more complicated model
yielded 3.0 significant digits. In this case, there is no problem (I
can use this final model for bootstrap, VPC, etc.) but what if none of
these models had converged.
Dennis
Dennis Fisher MD
P < (The "P Less Than" Company)
Phone: 1-866-PLessThan (1-866-753-7784)
Fax: 1-415-564-2220
www.PLessThan.com
--------------------------------------------------------------------
mail2web.com - Microsoft® Exchange solutions from a leading provider -
http://link.mail2web.com/Business/Exchange