Leonid wrote privately to Mark but Mark posted to nmusers:
"We were discussing the usefulness of nonmem error messages: what to do if $COV failed, or even if estimation step has not converged successfully (e.e., infinite OF message).
Nick point is that we just ignore the error messages.
My point is that we erro messages prompt us to study the model because more often than not, error messages point out to real problem (although sometimes they need to be ignored if you are happy with the model)."
Nick replied:
My point is not "ignore the error messages". It is "Do not use the termination messages as a guide to whether the model is good or bad."
(Note that NONMEM does not list them as ERROR messages. They are simply messages about NONMEM's view of the world when it decided to finish the estimation step. Some messages can be ignored (ROUNDING ERRORS) while others (EXCEEDED NUMBER OF FUNCTION EVALUATIONS) probably mean you should restart the model where it finished and keep going. Other messages in NMVI about boundary conditions are usually just a nuisance but if you did happen to be asleep and do not look at your parameter estimates then this is a reminder to wake up.
Leonid suggests we use these messages to examine the model. But there is no clue in these messages as to which part of the model (or the data) should be examined. So they are worthless except to remind you that you should be thinking about your model and data.
But you MUST think about the model and the data ALWAYS! It makes no difference what termination message you get you must continue to think (the hard part <grin>) and remember the advice of Box:
"All models are wrong but some are useful".
NONMEM has no idea if your model is wrong. It is always wrong but it seems Leonid is mislead into thinking it is not wrong when NONMEM says MINIMIZATION SUCCESSFUL. NONMEM especially has no idea if your model is useful. Only you and your colleagues who want to use the results of the modelling can decide if its useful. Usefulness can be investigated by model evaluation procedures (e.g. VPC, NPDE, etc) but the final decision will rest with a human brain not NONMEM's randomly generated minimization messages.
Nick
Mark Sale - Next Level Solutions wrote:
> Leonid,
>
> I agree with your point that failure to converge/and or covariance is a message that the model is a prompt to study the model. I object to those who claim that model that fails covariance is not useful despite data to the contrary (just went around and around with a sponsor about this - actually their stats consultant who basically just kept insisting on the theory regardless of data that we presented to the contrary). But, I think that the messages are completely non-specific - they tell you something is less than ideal, but give no clue as to what. I suspect that graphics are likely to be much more consistently informative, telling you not only that something is less than ideal, but some clue what to do to fix it. As such, I'm not sure that convergence and covariance messages add anything to the process (anything that a good and thorough analyst would have known already, based on VPC, NPC, various post hoc plots etc).
>
> Mark
>
> Mark Sale MD
> Next Level Solutions, LLC
> www.NextLevelSolns.com http://www.nextlevelsolns.com/
> 919-846-9185
>
Quoted reply history
> -------- Original Message --------
> Subject: RE: [NMusers] Models that abort before convergence
> From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Wed, November 19, 2008 10:23 pm
> To: [EMAIL PROTECTED]
>
> Mark,
> I am sorry, I simply do not understand what you are saying. I do
> not want
> to bother the group, it could be that I am the only one who is
> missing your
> point, but could you repeat what exactly you are trying to say?
> We were discussing the usefulness of nonmem error messages: what
> to do if
> $COV failed, or even if estimation step has not converged successfully
> (e.e., infinite OF message).
>
> Nick point is that we just ignore the error messages.
> My point is that we erro messages prompt us to study the model
> because more
> often than not, error messages point out to real problem (although
> sometimes they need to be ignored if you are happy with the
> model). What is
> your opinion?
>
> Thanks
> Leonid
>
> Original Message:
> -----------------
> From: Mark Sale - Next Level Solutions [EMAIL PROTECTED]
> Date: Wed, 19 Nov 2008 06:48:36 -0700
> To: [email protected]
> Subject: RE: [NMusers] Models that abort before convergence
>
> <!-- wmLetter_head_start -->
> <table align="center" style="empty-cells: show;" bgcolor="#ffffff"
> border="0" cellpadding="0" cellspacing="0" width="100%"><tr><td
> valign="top" style="text-align: left;" align="left" width="108"><img
> src="cid:left@b8c2bfa43541d55928145888195cae14"></td><td
> style="vertical-align: top;"><div style="padding: 5px; overflow-x:
> auto;"><!-- wmLetter_head_end --><html><body><span
> style="font-family:Verdana; color:#000000; font-size:10pt;">Leonid et
> al,
I'm a little confused by this discussion. To make an
> analogy, assume that drug company A has a wonderful theory that
> drug B will
> treat a disease. Theory makes sense by your favorite epistemology
> criteria
> etc. But of course, being good scientists, we know that theories
> must be
> verified, so we do an experiment, and the data suggest that the
> theory is
> wrong. Most of us would criticize as unscientific someone who who
> discarded the data (didn't point out flaws in the data, didn't provide
> opposing data, simply discounted it) in favor of continuing to
> believe the
> theory.
Why do we not apply the same standards here? Theory says
> that models that do not converge (or fail covariance) are "bad". Data
> (that so far as I know no one has found to be flawed, nor provided
> opposing
> data) suggests that, by at least one criteria (same parameter
> estimates,
> same SD of parameter estimates) there are no important differences. I
> don't disagree that failing a covariance step, or failing to converge
> provide information about a model. But it doesn't seem to be
> informative
> about what we probably really care about -does the line go through the
> points, how confident are we WRT the precision of the parameters
> and is the
> model predictive.
I'm not sure if the small number of published
> examples (of bootstrap with ~500 samples) are a small number of
> anecdotes
> or a small number of trials with N ~ 500, but I've run 5 or so
> myself and
> found the same to be consistently the case. That is, a successful
> covariance step is not informative WRT the parameter values or their
> precision. I suspect others have similar experience. If there are
> other
> "studies"/anecdotes with different conclusions, someone should publish
> them. Otherwise, it seems like we are obligated to abandon this
> theory in
> favor of the data.
Mark Sale MD
> Next Level Solutions, LLC
> <a href=" http://www.NextLevelSolns.com
> http://www.nextlevelsolns.com/"
> mce_href=" http://www.NextLevelSolns.com
> http://www.nextlevelsolns.com/">www.NextLevelSolns.com</a
> http://www.nextlevelsolns.com%3c/a>
> 919-846-9185
> <blockquote webmail="1" style="border-left: 2px solid blue;
> margin-left:
> 8px; padding-left: 8px; font-size: 10pt; color: black; font-family:
> verdana;">
> <div >
> -------- Original Message --------
> Subject: RE: [NMusers] Models that abort before convergence
> From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Tue, November 18, 2008 11:13 pm
> To: [EMAIL PROTECTED], [email protected],
> [EMAIL PROTECTED]
>
> Dennis,
> I do not support extreme views (from places where people walk upside
> down
> :) ) that Nonmem error messages should be ignored: they serve the
> useful
> purpose to alert when Nonmem is having some difficulties, and should
> always
> be part of the picture. If the data looks good, model is simple,
> then we
> need to look for the reason for the poor convergence. Sometimes it
> helps
> to
> use SIGDIG= 5 or 6 to get 3 significant digits precision. But if
> you are
> working on the limit of the algorithms (as implemented) abilities:
> nonlinear model + stiff differential equations + large range of doses
> and
> concentrations, etc., then you face the situation when you cannot
> force
> convergence even if you try hard. On my recent project, none of
> the
> intermediate model converged even though bootstrap provided pretty
> narrow
> CI (so it does not look like over-parametrized model), all diagnostic
> plots
> were good, and the visual predictive check was reasonable. Then
> you just
> blame the algorithm and move on. You loose the ability to justify
> your
> covariate selection based on the objective function drop (which is
> not a
> good idea any way), and may need to provide a little bit more
> detailed
> investigation to convince reviewers (regulatory and/or journal)
> that the
> model is adequate for the intended purpose.
> Thanks
> Leonid
>
>
> Original Message:
> -----------------
> From: Dennis Fisher [EMAIL PROTECTED]
> Date: Tue, 18 Nov 2008 11:21:23 -0800
> To: [email protected], [EMAIL PROTECTED]
> Subject: [NMusers] Models that abort before convergence
>
>
> Colleagues,
>
> I am curious as to your thoughts about a particular NONMEM issue.
> I
> often find myself in a situation where a complex model does not
> converge to 3 digits ("no of digits: unreportable") yet the
> objective
> function is markedly better than a previous model and graphics
> suggest
> that the model is quite good (and better than the previous one).
> Nick
> Holford has advocated (and I agree) that NONMEM's SE's have
> minimal
> utility and the inability to calculate them is not important.
> However, I have not seen similar discussion about whether one can
> /
> should accept a model that did not converge.
>
> The particular situation that I dealing with at the moment is that
> a
> dataset that I am analyzing yielded a series of results that did
> not
> converge as I added parameters (despite an improving fit and a
> marked
> decrease in the objective function), then yet a more complicated
> model
> yielded 3.0 significant digits. In this case, there is no problem
> (I
> can use this final model for bootstrap, VPC, etc.) but what if
> none of
> these models had converged.
>
> Dennis
>
> Dennis Fisher MD
> P < (The "P Less Than" Company)
> Phone: 1-866-PLessThan (1-866-753-7784)
> Fax: 1-415-564-2220
> <a href=" http://www.PLessThan.com http://www.plessthan.com/"
> target="_blank"
> mce_href=" http://www.PLessThan.com
> http://www.plessthan.com/">www.PLessThan.com</a
> http://www.plessthan.com%3c/a>
>
>
> --------------------------------------------------------------------
> mail2web.com - Microsoft® Exchange solutions from a leading
> provider -
> <a href=" http://link.mail2web.com/Business/Exchange"; target="_blank"
> mce_href=" http://link.mail2web.com/Business/Exchange";> http://link.mail2web.c
> http://link.mail2web.c/
> om/Business/Exchange</a>
>
>
>
> </div>
> </blockquote></span></body></html><!-- wmLetter_tail_start
> --></div></td></tr></table><!-- wmLetter_tail_end -->
>
> --------------------------------------------------------------------
> mail2web.com - Microsoft® Exchange solutions from a leading provider -
> http://link.mail2web.com/Business/Exchange
--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
[EMAIL PROTECTED] tel:+64(9)923-6730 fax:+64(9)373-7090
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford