Dear All,
Here is the summary of the the replies for the questions that I sent out recently. I received a total of 35 replies. This e-mail consists of 4 parts, as follows:
Part 1: Each original questions is followed by the summary of replies
Part 2: All comments that I received are copy-pasted after the summary of replies.
Part 3: CSV file with the original data is copy-pasted after the comments
Part 4: R code that I used to summarize the results is provided
Thanks to all who participated.
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
############ RESULTS ###########################
> 1. Would you like Nonmem to stop producing all run-time (not syntax)
> error/warning messages (134, 137, number of significant digits, etc.)
> and "MINIMIZATION SUCCESSFUL" messages (YES/NO):
question YES No Missing
Q1 2 (5.7%) 33 (94.3%) 0 (0%)
> 2. Do you remember at least one example when the run-time error message helped you to find an error in your code (YES/NO):
question YES No Missing
Q2 31 (88.6%) 3 (8.6%) 1 (2.9%)
> 3. In your experience, run-time error messages allow you to detect model errors or problems quicker than it would be done without error messages: (agree/disagree)
question AGREE No Missing
Q3 27 (77.1%) 3 (8.6%) 5 (14.3%)
> 4. Have you ever used in your report/publication ANY model that did not have $COV step completed (YES/NO):
question YES No Missing
Q4 25 (71.4%) 9 (25.7%) 1 (2.9%)
> 5. Have you ever used in your report/publication ANY model that did not converge (YES/NO):
question YES No Missing
Q5 13 (37.1%) 21 (60%) 1 (2.9%)
> 6. Have you ever used in your report/publication FINAL model that did
> not have $COV step completed (YES/NO):
question YES No Missing
Q6 16 (45.7%) 18 (51.4%) 1 (2.9%)
> 7. Have you ever used in your report/publication FINAL model that did
> not converge (YES/NO):
question YES No Missing
Q7 3 (8.6%) 31 (88.6%) 1 (2.9%)
> 8. Define yourself as novice/intermediate/experienced Nonmem user:
Missing Novice Intermediate Experienced
1 3 15 16
############ Comments ###########################
Honestly, nobody is proposing to remove the minimization successful
statement or any non syntax error message. I agree with the comments
that stress it's important to take them with a pinch of salt as these
"errors" not always point you in the right directions, and not always
being picky about $COV step or number of significant digits help in
selecting the best model
-------------------------
Question 1 has at least 3 parts and cannot be answered YES or NO in any meaningful way. Please note the messages we have discussed are not ERROR or WARNING messages. They are a message about the minimization status.
I would be happy if NONMEM stuck to the facts. It can tell me if it achieved the requested sigdigs (CONVERGED) or ran out of function evals (PREMATURE TERMINATION). But it should keep its subjective judgements to itself.
If you made a list of run-time error messages and another of run-time warning messages then perhaps your survey could be more helpful in deciding which are meaningful?
----------------------
My 2 cents on the discussion, for what it’s worth… The error message is usually related to an error in the dataset / dataset programming issue / initial estimates / model parameterization. Generally, all need to be addressed. Just as important, I wouldn’t trust a MINIMIZATION SUCCESSFUL message either as you will often see flip-flop or 3-CMT identifiability issues that NONMEM doesn’t flag. NONMEM has its flaws, but I’ve seen many of NONMEM’s error messages disappear after correcting a programming error in the dataset, refining the initial estimates, or re-parameterizing the model. I think the practical pharmacometrician isn’t going to accept NONMEM outputs strictly at face value, but would challenge the findings to confirm the results. That’s just good science.
----------------------
I have never reported out as a final model a run that failed to converge or failed the COV step. My guess is that individuals who frequently do probably tend to be more mechanistic in their model building than I am and often push the complexity of their models beyond what can be supported by the data in hand. For those that do report out models that don't converge, I wonder if they have tried re-running their models with different starting values (15-20% different) and see if NONMEM fails to converge at the same set of parameter estimates. My guess is in many cases it won't although both sets of estimates may appear "reasonable" and give similar fits and VPC.
For individuals who have strong prior beliefs about their mechanistic models, my thinking is that rather than using approximate maximum likelihood methods and ignoring the diagnostics that might suggest their model is unstable or not fully supported by the data, I think they would be better served by using a Bayesian approach. That way they can be explicit about the strength of their priors and they don't have to worry about convergence and COV step failures. JMHO.
------------------
############## Original Data ##################
ID,Q1,Q2,Q3,Q4,Q5,Q6,Q7,Q8
1,0,0,-1,1,1,1,0,1
2,0,1,1,1,0,1,0,3
3,0,1,-1,1,0,1,0,2
4,0,1,-1,1,1,1,1,3
5,0,0,-1,1,0,1,0,2
6,0,1,-1,1,0,1,0,3
7,0,1,1,0,0,0,0,3
8,1,1,0,0,0,0,0,3
9,0,1,1,0,0,0,0,2
10,0,1,1,1,0,0,0,-1
11,0,1,1,1,0,1,0,3
12,0,0,1,0,0,0,0,2
13,0,1,1,1,1,1,0,2
14,0,1,1,0,0,0,0,3
15,0,1,1,1,1,1,0,3
16,1,-1,0,1,1,1,0,2
17,0,1,1,-1,-1,-1,-1,1
18,0,1,1,0,0,0,0,3
19,0,1,1,1,0,0,0,3
20,0,1,1,1,0,0,0,3
21,0,1,1,1,1,0,0,1
22,0,1,1,0,0,0,0,2
23,0,1,0,1,1,1,0,3
24,0,1,1,1,1,0,0,2
25,0,1,1,1,1,0,0,3
26,0,1,1,0,0,0,0,2
27,0,1,1,1,0,0,0,2
28,0,1,1,1,1,1,0,2
29,0,1,1,1,0,1,0,3
30,0,1,1,1,0,0,0,3
31,0,1,1,1,1,1,1,2
32,0,1,1,1,1,1,1,3
33,0,1,1,1,1,1,0,2
34,0,1,1,1,0,0,0,2
35,0,1,1,0,0,0,0,2
############## R code ############################
raw.data <- read.table("C:/poll.csv",sep=",",header=T)
res <- NULL
for(Qname in paste("Q",1:7,sep="") ){
x <- raw.data[,Qname]
n.yes <- sum(x == 1)
n.no <- sum(x == 0)
n.na <- sum(x == -1)
n <- length(x)
temp <- data.frame(n=n,question=Qname,
YES=paste(n.yes," (",round(100*n.yes/n,1),"%)",sep=""),
No=paste(n.no," (",round(100*n.no/n,1),"%)",sep=""),
Missing=paste(n.na," (",round(100*n.na/n,1),"%)",sep=""))
res <- rbind(res,temp)
}
res
table(raw.data$Q8)
##########################################################
unscientific poll
Thanks Leonid for this service to our community!
I am not surprised by the results but rather by the number of replies.
There are supposedly 1000 NONMEM users on this beautiful planet, and 35
participated in the survey. My guess is that these are about 50% of the
active participants of the usernet. Therefore we may have, let's say, 100 regular readers of the usernet. That is 10% of the users. That number
troubles me as it tells me that 90% of the NONMEM users have established
their routine and see little need (have little time) to read the postings
of the usernet. Yet, the discussions on our usernet are extremely relevant
to our daily modeling work.
Cheers to our usernet!
Joachim
__________________________________________
Joachim GREVEL, Ph.D.
MERCK SERONO International S.A.
Exploratory Medicine
1202 Geneva
Tel: +41.22.414.4751
Fax: +41.22.414.3059
Email: joachim.grevel
erono.net
Leonid Gibiansky <LGibiansky
/2008 07:00 PM
To
nmusers <nmusers
c
Subject
[NMusers] unscientific poll
for the questions that I sent out
recently. I received a total of 35 replies. This e-mail consists of 4
parts, as follows:
Part 1: Each original questions is followed by the summary of replies
Part 2: All comments that I received are copy-pasted after the summary
of replies.
Part 3: CSV file with the original data is copy-pasted after the comments
Part 4: R code that I used to summarize the results is provided
Thanks to all who participated.
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
############ RESULTS ###########################
syntax)
> error/warning messages (134, 137, number of significant digits, etc.)
> and "MINIMIZATION SUCCESSFUL" messages (YES/NO):
question YES No Missing
Q1 2 (5.7%) 33 (94.3%) 0 (0%)
> 2. Do you remember at least one example when the run-time error
message helped you to find an error in your code (YES/NO):
question YES No MissingQ2 31 (88.6%) 3 (8.6%) 1 (2.9%)
> 3. In your experience, run-time error messages allow you to detect
model errors or problems quicker than it would be done without error
messages: (agree/disagree)
question AGREE No Missing
Q3 27 (77.1%) 3 (8.6%) 5 (14.3%)
> 4. Have you ever used in your report/publication ANY model that did
not have $COV step completed (YES/NO):
question YES No Missing
Q4 25 (71.4%) 9 (25.7%) 1 (2.9%)> 5. Have you ever used in your report/publication ANY model that did
not converge (YES/NO):
question YES No Missing
Q5 13 (37.1%) 21 (60%) 1 (2.9%)
that did
> not have $COV step completed (YES/NO):
question YES No Missing
Q6 16 (45.7%) 18 (51.4%) 1 (2.9%)
> 7. Have you ever used in your report/publication FINAL model that did
> not converge (YES/NO):
question YES No Missing
Q7 3 (8.6%) 31 (88.6%) 1 (2.9%)
> 8. Define yourself as novice/intermediate/experienced Nonmem user:
Missing Novice Intermediate Experienced
1 3 15 16
############ Comments ###########################
Honestly, nobody is proposing to remove the minimization successful
statement or any non syntax error message. I agree with the comments
that stress it's important to take them with a pinch of salt as these
"errors" not always point you in the right directions, and not always
being picky about $COV step or number of significant digits help in
selecting the best model-------------------------
Question 1 has at least 3 parts and cannot be answered YES or NO in any
meaningful way. Please note the messages we have discussed are not ERROR
or WARNING messages. They are a message about the minimization status.
I would be happy if NONMEM stuck to the facts. It can tell me if it
achieved the requested sigdigs (CONVERGED) or ran out of function evals
(PREMATURE TERMINATION). But it should keep its subjective judgements to
itself.
If you made a list of run-time error messages and another of run-time
warning messages then perhaps your survey could be more helpful in
deciding which are meaningful?
----------------------
My 2 cents on the discussion, for what it?s worth? The error message is
usually related to an error in the dataset / dataset programming issue /
initial estimates / model parameterization. Generally, all need to be
addressed. Just as important, I wouldn?t trust a MINIMIZATION
SUCCESSFUL message either as you will often see flip-flop or 3-CMT
identifiability issues that NONMEM doesn?t flag. NONMEM has its flaws,
but I?ve seen many of NONMEM?s error messages disappear after correcting
a programming error in the dataset, refining the initial estimates, or
re-parameterizing the model. I think the practical pharmacometrician
isn?t going to accept NONMEM outputs strictly at face value, but would
challenge the findings to confirm the results. That?s just good science.
----------------------
nverge
or failed the COV step. My guess is that individuals who frequently do
probably tend to be more mechanistic in their model building than I am
and often push the complexity of their models beyond what can be
supported by the data in hand. For those that do report out models that
don't converge, I wonder if they have tried re-running their models with
different starting values (15-20% different) and see if NONMEM fails to
converge at the same set of parameter estimates. My guess is in many
cases it won't although both sets of estimates may appear "reasonable"
and give similar fits and VPC.
models, my thinking is that rather than using approximate maximum
model is unstable or not fully supported by the data, I think they would
be better served by using a Bayesian approach. That way they can be
explicit about the strength of their priors and they don't have to worry
about convergence and COV step failures. JMHO.
------------------
############## Original Data ##################
ID,Q1,Q2,Q3,Q4,Q5,Q6,Q7,Q8
1,0,0,-1,1,1,1,0,1
,0,0,-1,1,0,1,0,2
6,0,1,-1,1,0,1,0,3
7,0,1,1,0,0,0,0,3
8,1,1,0,0,0,0,0,3
9,0,1,1,0,0,0,0,2
10,0,1,1,1,0,0,0,-1
11,0,1,1,1,0,1,0,3
12,0,0,1,0,0,0,0,2
13,0,1,1,1,1,1,0,2
14,0,1,1,0,0,0,0,3
15,0,1,1,1,1,1,0,3
16,1,-1,0,1,1,1,0,2
17,0,1,1,-1,-1,-1,-1,1
18,0,1,1,0,0,0,0,3
19,0,1,1,1,0,0,0,3
20,0,1,1,1,0,0,0,3
21,0,1,1,1,1,0,0,1
22,0,1,1,0,0,0,0,2
23,0,1,0,1,1,1,0,3
24,0,1,1,1,1,0,0,2
25,0,1,1,1,1,0,0,3
26,0,1,1,0,0,0,0,227,0,1,1,1,0,0,0,2
28,0,1,1,1,1,1,0,2
29,0,1,1,1,0,1,0,3
33,0,1,1,1,1,1,0,2
34,0,1,1,1,0,0,0,2
35,0,1,1,0,0,0,0,2
.data <- read.table("C:/poll.csv",sep=",",header=T)
res <- NULL
for(Qname in paste("Q",1:7,sep="") ){
x <- raw.data[,Qname]
n.yes <- sum(x = 1)
n.no <- sum(x =
= 0)
n.na <- sum(x = -1)
n <- length(x)
temp <- data.frame(n=n,question=Qname,
YES=paste(n.yes," (",round(100*n.yes/n,1),"%)",sep=""),
No=paste(n.no," (",round(100*n.no/n,1),"%)",sep=""),
1),"%)",sep=""))
res <- rbind(res,temp)
}
res
table(raw.data$Q8)
##########################################################
-----------------------------------------
This message and any attachment are confidential, may be privileged
or otherwise protected from disclosure and are intended only for
use by the addressee(s) named herein. If you are not the intended
recipient, you must not copy this message or attachment or disclose
sion in error, please notify the sender immediately and
delete the message and any attachment from your system. Merck
Serono does not accept liability for any omissions or errors in
this message which may arise as a result of E-Mail-transmission or
for damages resulting from any unauthorized changes of the content
of this message and any attachment thereto. If verification is
required, please request a hard-copy version. Merck Serono does not
guarantee that this message is free of viruses and does not accept
liability for any damages caused by any virus transmitted
therewith.
--=_alternative 00273275C1257515_=--
Thanks Leonid for this service to our community!
I am not surprised by the results but rather by the number of replies.
There are supposedly 1000 NONMEM users on this beautiful planet, and 35
participated in the survey. My guess is that these are about 50% of the
active participants of the usernet. Therefore we may have, let's say, 100
regular readers of the usernet. That is 10% of the users. That number
troubles me as it tells me that 90% of the NONMEM users have established
their routine and see little need (have little time) to read the postings
of the usernet. Yet, the discussions on our usernet are extremely relevant
to our daily modeling work.
Cheers to our usernet!
Joachim
__________________________________________
Joachim GREVEL, Ph.D.
MERCK SERONO International S.A.
Exploratory Medicine
1202 Geneva
Tel: +41.22.414.4751
Fax: +41.22.414.3059
Email: [EMAIL PROTECTED]
Leonid Gibiansky <[EMAIL PROTECTED]>
Sent by: [EMAIL PROTECTED]
12/03/2008 07:00 PM
To
nmusers <[email protected]>
cc
Subject
[NMusers] unscientific poll
Dear All,
Here is the summary of the the replies for the questions that I sent out
recently. I received a total of 35 replies. This e-mail consists of 4
parts, as follows:
Part 1: Each original questions is followed by the summary of replies
Part 2: All comments that I received are copy-pasted after the summary
of replies.
Part 3: CSV file with the original data is copy-pasted after the comments
Part 4: R code that I used to summarize the results is provided
Thanks to all who participated.
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
############ RESULTS ###########################
> 1. Would you like Nonmem to stop producing all run-time (not syntax)
> error/warning messages (134, 137, number of significant digits, etc.)
> and "MINIMIZATION SUCCESSFUL" messages (YES/NO):
question YES No Missing
Q1 2 (5.7%) 33 (94.3%) 0 (0%)
> 2. Do you remember at least one example when the run-time error
message helped you to find an error in your code (YES/NO):
question YES No Missing
Q2 31 (88.6%) 3 (8.6%) 1 (2.9%)
> 3. In your experience, run-time error messages allow you to detect
model errors or problems quicker than it would be done without error
messages: (agree/disagree)
question AGREE No Missing
Q3 27 (77.1%) 3 (8.6%) 5 (14.3%)
> 4. Have you ever used in your report/publication ANY model that did
not have $COV step completed (YES/NO):
question YES No Missing
Q4 25 (71.4%) 9 (25.7%) 1 (2.9%)
> 5. Have you ever used in your report/publication ANY model that did
not converge (YES/NO):
question YES No Missing
Q5 13 (37.1%) 21 (60%) 1 (2.9%)
> 6. Have you ever used in your report/publication FINAL model that did
> not have $COV step completed (YES/NO):
question YES No Missing
Q6 16 (45.7%) 18 (51.4%) 1 (2.9%)
> 7. Have you ever used in your report/publication FINAL model that did
> not converge (YES/NO):
question YES No Missing
Q7 3 (8.6%) 31 (88.6%) 1 (2.9%)
> 8. Define yourself as novice/intermediate/experienced Nonmem user:
Missing Novice Intermediate Experienced
1 3 15 16
############ Comments ###########################
Honestly, nobody is proposing to remove the minimization successful
statement or any non syntax error message. I agree with the comments
that stress it's important to take them with a pinch of salt as these
"errors" not always point you in the right directions, and not always
being picky about $COV step or number of significant digits help in
selecting the best model
-------------------------
Question 1 has at least 3 parts and cannot be answered YES or NO in any
meaningful way. Please note the messages we have discussed are not ERROR
or WARNING messages. They are a message about the minimization status.
I would be happy if NONMEM stuck to the facts. It can tell me if it
achieved the requested sigdigs (CONVERGED) or ran out of function evals
(PREMATURE TERMINATION). But it should keep its subjective judgements to
itself.
If you made a list of run-time error messages and another of run-time
warning messages then perhaps your survey could be more helpful in
deciding which are meaningful?
----------------------
My 2 cents on the discussion, for what it?s worth? The error message is
usually related to an error in the dataset / dataset programming issue /
initial estimates / model parameterization. Generally, all need to be
addressed. Just as important, I wouldn?t trust a MINIMIZATION
SUCCESSFUL message either as you will often see flip-flop or 3-CMT
identifiability issues that NONMEM doesn?t flag. NONMEM has its flaws,
but I?ve seen many of NONMEM?s error messages disappear after correcting
a programming error in the dataset, refining the initial estimates, or
re-parameterizing the model. I think the practical pharmacometrician
isn?t going to accept NONMEM outputs strictly at face value, but would
challenge the findings to confirm the results. That?s just good science.
----------------------
I have never reported out as a final model a run that failed to converge
or failed the COV step. My guess is that individuals who frequently do
probably tend to be more mechanistic in their model building than I am
and often push the complexity of their models beyond what can be
supported by the data in hand. For those that do report out models that
don't converge, I wonder if they have tried re-running their models with
different starting values (15-20% different) and see if NONMEM fails to
converge at the same set of parameter estimates. My guess is in many
cases it won't although both sets of estimates may appear "reasonable"
and give similar fits and VPC.
For individuals who have strong prior beliefs about their mechanistic
models, my thinking is that rather than using approximate maximum
likelihood methods and ignoring the diagnostics that might suggest their
model is unstable or not fully supported by the data, I think they would
be better served by using a Bayesian approach. That way they can be
explicit about the strength of their priors and they don't have to worry
about convergence and COV step failures. JMHO.
------------------
############## Original Data ##################
ID,Q1,Q2,Q3,Q4,Q5,Q6,Q7,Q8
1,0,0,-1,1,1,1,0,1
2,0,1,1,1,0,1,0,3
3,0,1,-1,1,0,1,0,2
4,0,1,-1,1,1,1,1,3
5,0,0,-1,1,0,1,0,2
6,0,1,-1,1,0,1,0,3
7,0,1,1,0,0,0,0,3
8,1,1,0,0,0,0,0,3
9,0,1,1,0,0,0,0,2
10,0,1,1,1,0,0,0,-1
11,0,1,1,1,0,1,0,3
12,0,0,1,0,0,0,0,2
13,0,1,1,1,1,1,0,2
14,0,1,1,0,0,0,0,3
15,0,1,1,1,1,1,0,3
16,1,-1,0,1,1,1,0,2
17,0,1,1,-1,-1,-1,-1,1
18,0,1,1,0,0,0,0,3
19,0,1,1,1,0,0,0,3
20,0,1,1,1,0,0,0,3
21,0,1,1,1,1,0,0,1
22,0,1,1,0,0,0,0,2
23,0,1,0,1,1,1,0,3
24,0,1,1,1,1,0,0,2
25,0,1,1,1,1,0,0,3
26,0,1,1,0,0,0,0,2
27,0,1,1,1,0,0,0,2
28,0,1,1,1,1,1,0,2
29,0,1,1,1,0,1,0,3
30,0,1,1,1,0,0,0,3
31,0,1,1,1,1,1,1,2
32,0,1,1,1,1,1,1,3
33,0,1,1,1,1,1,0,2
34,0,1,1,1,0,0,0,2
35,0,1,1,0,0,0,0,2
############## R code ############################
raw.data <- read.table("C:/poll.csv",sep=",",header=T)
res <- NULL
for(Qname in paste("Q",1:7,sep="") ){
x <- raw.data[,Qname]
n.yes <- sum(x == 1)
n.no <- sum(x == 0)
n.na <- sum(x == -1)
n <- length(x)
temp <- data.frame(n=n,question=Qname,
YES=paste(n.yes," (",round(100*n.yes/n,1),"%)",sep=""),
No=paste(n.no," (",round(100*n.no/n,1),"%)",sep=""),
Missing=paste(n.na,"
(",round(100*n.na/n,1),"%)",sep=""))
res <- rbind(res,temp)
}
res
table(raw.data$Q8)
##########################################################
-----------------------------------------
This message and any attachment are confidential, may be privileged
or otherwise protected from disclosure and are intended only for
use by the addressee(s) named herein. If you are not the intended
recipient, you must not copy this message or attachment or disclose
the contents to any other person. If you have received this
transmission in error, please notify the sender immediately and
delete the message and any attachment from your system. Merck
Serono does not accept liability for any omissions or errors in
this message which may arise as a result of E-Mail-transmission or
for damages resulting from any unauthorized changes of the content
of this message and any attachment thereto. If verification is
required, please request a hard-copy version. Merck Serono does not
guarantee that this message is free of viruses and does not accept
liability for any damages caused by any virus transmitted
therewith.
My guess is the low response is likely an effect of inexperienced and
intermediate users remaining silent to allow the most informed users
decide the outcome of the poll. This was a consideration for me, even
though I eventually decided to participate as an intermediate user.
Regular reading of a forum without contributing is a common internet
phenomenon: http://en.wikipedia.org/wiki/Lurker. I dont expect nmusers
to be any different.
My thanks to Leonid for doing the poll. It helped me put the
discussions/debates/exchanges/advice in context.
Best regards to all, I will go back to lurking now ;)
Douglas Eleveld