unscientific poll

Dear All, Here is the summary of the the replies for the questions that I sent out recently. I received a total of 35 replies. This e-mail consists of 4 parts, as follows: Part 1: Each original questions is followed by the summary of replies Part 2: All comments that I received are copy-pasted after the summary of replies. Part 3: CSV file with the original data is copy-pasted after the comments Part 4: R code that I used to summarize the results is provided Thanks to all who participated. Leonid -------------------------------------- Leonid Gibiansky, Ph.D. President, QuantPharm LLC web: www.quantpharm.com e-mail: LGibiansky at quantpharm.com tel: (301) 767 5566 ############ RESULTS ########################### > 1. Would you like Nonmem to stop producing all run-time (not syntax) > error/warning messages (134, 137, number of significant digits, etc.) > and "MINIMIZATION SUCCESSFUL" messages (YES/NO): question YES No Missing Q1 2 (5.7%) 33 (94.3%) 0 (0%) > 2. Do you remember at least one example when the run-time error message helped you to find an error in your code (YES/NO): question YES No Missing Q2 31 (88.6%) 3 (8.6%) 1 (2.9%) > 3. In your experience, run-time error messages allow you to detect model errors or problems quicker than it would be done without error messages: (agree/disagree) question AGREE No Missing Q3 27 (77.1%) 3 (8.6%) 5 (14.3%) > 4. Have you ever used in your report/publication ANY model that did not have $COV step completed (YES/NO): question YES No Missing Q4 25 (71.4%) 9 (25.7%) 1 (2.9%) > 5. Have you ever used in your report/publication ANY model that did not converge (YES/NO): question YES No Missing Q5 13 (37.1%) 21 (60%) 1 (2.9%) > 6. Have you ever used in your report/publication FINAL model that did > not have $COV step completed (YES/NO): question YES No Missing Q6 16 (45.7%) 18 (51.4%) 1 (2.9%) > 7. Have you ever used in your report/publication FINAL model that did > not converge (YES/NO): question YES No Missing Q7 3 (8.6%) 31 (88.6%) 1 (2.9%) > 8. Define yourself as novice/intermediate/experienced Nonmem user: Missing Novice Intermediate Experienced 1 3 15 16 ############ Comments ########################### Honestly, nobody is proposing to remove the minimization successful statement or any non syntax error message. I agree with the comments that stress it's important to take them with a pinch of salt as these "errors" not always point you in the right directions, and not always being picky about $COV step or number of significant digits help in selecting the best model ------------------------- Question 1 has at least 3 parts and cannot be answered YES or NO in any meaningful way. Please note the messages we have discussed are not ERROR or WARNING messages. They are a message about the minimization status. I would be happy if NONMEM stuck to the facts. It can tell me if it achieved the requested sigdigs (CONVERGED) or ran out of function evals (PREMATURE TERMINATION). But it should keep its subjective judgements to itself. If you made a list of run-time error messages and another of run-time warning messages then perhaps your survey could be more helpful in deciding which are meaningful? ---------------------- My 2 cents on the discussion, for what it’s worth… The error message is usually related to an error in the dataset / dataset programming issue / initial estimates / model parameterization. Generally, all need to be addressed. Just as important, I wouldn’t trust a MINIMIZATION SUCCESSFUL message either as you will often see flip-flop or 3-CMT identifiability issues that NONMEM doesn’t flag. NONMEM has its flaws, but I’ve seen many of NONMEM’s error messages disappear after correcting a programming error in the dataset, refining the initial estimates, or re-parameterizing the model. I think the practical pharmacometrician isn’t going to accept NONMEM outputs strictly at face value, but would challenge the findings to confirm the results. That’s just good science. ---------------------- I have never reported out as a final model a run that failed to converge or failed the COV step. My guess is that individuals who frequently do probably tend to be more mechanistic in their model building than I am and often push the complexity of their models beyond what can be supported by the data in hand. For those that do report out models that don't converge, I wonder if they have tried re-running their models with different starting values (15-20% different) and see if NONMEM fails to converge at the same set of parameter estimates. My guess is in many cases it won't although both sets of estimates may appear "reasonable" and give similar fits and VPC. For individuals who have strong prior beliefs about their mechanistic models, my thinking is that rather than using approximate maximum likelihood methods and ignoring the diagnostics that might suggest their model is unstable or not fully supported by the data, I think they would be better served by using a Bayesian approach. That way they can be explicit about the strength of their priors and they don't have to worry about convergence and COV step failures. JMHO. ------------------ ############## Original Data ################## ID,Q1,Q2,Q3,Q4,Q5,Q6,Q7,Q8 1,0,0,-1,1,1,1,0,1 2,0,1,1,1,0,1,0,3 3,0,1,-1,1,0,1,0,2 4,0,1,-1,1,1,1,1,3 5,0,0,-1,1,0,1,0,2 6,0,1,-1,1,0,1,0,3 7,0,1,1,0,0,0,0,3 8,1,1,0,0,0,0,0,3 9,0,1,1,0,0,0,0,2 10,0,1,1,1,0,0,0,-1 11,0,1,1,1,0,1,0,3 12,0,0,1,0,0,0,0,2 13,0,1,1,1,1,1,0,2 14,0,1,1,0,0,0,0,3 15,0,1,1,1,1,1,0,3 16,1,-1,0,1,1,1,0,2 17,0,1,1,-1,-1,-1,-1,1 18,0,1,1,0,0,0,0,3 19,0,1,1,1,0,0,0,3 20,0,1,1,1,0,0,0,3 21,0,1,1,1,1,0,0,1 22,0,1,1,0,0,0,0,2 23,0,1,0,1,1,1,0,3 24,0,1,1,1,1,0,0,2 25,0,1,1,1,1,0,0,3 26,0,1,1,0,0,0,0,2 27,0,1,1,1,0,0,0,2 28,0,1,1,1,1,1,0,2 29,0,1,1,1,0,1,0,3 30,0,1,1,1,0,0,0,3 31,0,1,1,1,1,1,1,2 32,0,1,1,1,1,1,1,3 33,0,1,1,1,1,1,0,2 34,0,1,1,1,0,0,0,2 35,0,1,1,0,0,0,0,2 ############## R code ############################ raw.data <- read.table("C:/poll.csv",sep=",",header=T) res <- NULL for(Qname in paste("Q",1:7,sep="") ){ x <- raw.data[,Qname] n.yes <- sum(x == 1) n.no <- sum(x == 0) n.na <- sum(x == -1) n <- length(x) temp <- data.frame(n=n,question=Qname, YES=paste(n.yes," (",round(100*n.yes/n,1),"%)",sep=""), No=paste(n.no," (",round(100*n.no/n,1),"%)",sep=""), Missing=paste(n.na," (",round(100*n.na/n,1),"%)",sep="")) res <- rbind(res,temp) } res table(raw.data$Q8) ##########################################################