bug in NONMEM VI?

6 messages 5 people Latest: Jan 11, 2008

RE: re: bug in NONMEM VI?

From: John Lukas Date: January 02, 2008 technical
________________________________
Quoted reply history
From: John Lukas Sent: Mon 12/17/2007 11:51 PM To: [email protected] Subject: re: bug in NONMEM VI? Dear Nmusers, A simple LME model run using NM version VI, FO method, with 4 covariates (5 THETA's) as, . . $PRED SUM1=THETA(2)*COV1+THETA(3)*COV2 SUM2=THETA(4)*COV4+THETA(5)*COV3 TLAU=THETA(1)+SUM1+SUM2 F=TLAU+ETA(1) Y=F+EPS(1) $THETA 4 .1 .1 .1 .1 $OMEGA .1 $SIGMA .1 $ESTIMATION MAXEVALS=1900 PRINT=5 METHOD=0 (let's call this FIT A) gave a different fit from another (FIT B) where only the order of THETA( 4) and 5 was reversed, all else kept the same, as, . . SUM2=THETA(5)*COV4+THETA(4)*COV3 . . (even more strange, FIT B, the reversed order run, proved to be the good run!) NONMEM version V had the same result for both FITS A and B, as expected (and same as FIT B from version 6). FOCE gave good answers always in both NONMEM version V and version VI. But, the question remains about that dependence on ordering of the THETAs with the FO method for version VI. Any comments? Has there been discussion on this earlier that I missed? Thanks in advance. John PS I am aware of other *important* incosistencies between version VI and V from university colleagues working on a PK problem with FOCE... John C Lukas Strategic Consulting Services Pharsight Corp. line: + 33 492 726 495 cell: + 33 626 496 777

RE: RE: re: bug in NONMEM VI?

From: William Bachman Date: January 02, 2008 technical
Have you tried both of these runs using the SLOW option on the estimation record? _____
Quoted reply history
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Lukas Sent: Wednesday, January 02, 2008 2:36 PM To: [email protected] Subject: [NMusers] RE: re: bug in NONMEM VI? _____ From: John Lukas Sent: Mon 12/17/2007 11:51 PM To: [email protected] Subject: re: bug in NONMEM VI? Dear Nmusers, A simple LME model run using NM version VI, FO method, with 4 covariates (5 THETA's) as, . . $PRED SUM1=THETA(2)*COV1+THETA(3)*COV2 SUM2=THETA(4)*COV4+THETA(5)*COV3 TLAU=THETA(1)+SUM1+SUM2 F=TLAU+ETA(1) Y=F+EPS(1) $THETA 4 .1 .1 .1 .1 $OMEGA .1 $SIGMA .1 $ESTIMATION MAXEVALS=1900 PRINT=5 METHOD=0 (let's call this FIT A) gave a different fit from another (FIT B) where only the order of THETA( 4) and 5 was reversed, all else kept the same, as, . . SUM2=THETA(5)*COV4+THETA(4)*COV3 . . (even more strange, FIT B, the reversed order run, proved to be the good run!) NONMEM version V had the same result for both FITS A and B, as expected (and same as FIT B from version 6). FOCE gave good answers always in both NONMEM version V and version VI. But, the question remains about that dependence on ordering of the THETAs with the FO method for version VI. Any comments? Has there been discussion on this earlier that I missed? Thanks in advance. John PS I am aware of other *important* incosistencies between version VI and V from university colleagues working on a PK problem with FOCE. John C Lukas Strategic Consulting Services Pharsight Corp. line: + 33 492 726 495 cell: + 33 626 496 777

RE: RE: re: bug in NONMEM VI?

From: Peter Bonate Date: January 02, 2008 technical
I'd also like to see how correlated COV3 and COV4 are. This may be a collinearity issue. pete bonate Peter L. Bonate, PhD, FCP Genzyme Corporation Senior Director Clinical Pharmacology and Pharmacokinetics 4545 Horizon Hill Blvd San Antonio, TX 78229 USA [EMAIL PROTECTED] phone: 210-949-8662 fax: 210-949-8219 crackberry: 210-315-2713
Quoted reply history
________________________________ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Bill Bachman Sent: Wednesday, January 02, 2008 2:38 PM To: 'John Lukas'; [email protected] Subject: RE: [NMusers] RE: re: bug in NONMEM VI? Have you tried both of these runs using the SLOW option on the estimation record? ________________________________ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Lukas Sent: Wednesday, January 02, 2008 2:36 PM To: [email protected] Subject: [NMusers] RE: re: bug in NONMEM VI? ________________________________ From: John Lukas Sent: Mon 12/17/2007 11:51 PM To: [email protected] Subject: re: bug in NONMEM VI? Dear Nmusers, A simple LME model run using NM version VI, FO method, with 4 covariates (5 THETA's) as, . . $PRED SUM1=THETA(2)*COV1+THETA(3)*COV2 SUM2=THETA(4)*COV4+THETA(5)*COV3 TLAU=THETA(1)+SUM1+SUM2 F=TLAU+ETA(1) Y=F+EPS(1) $THETA 4 .1 .1 .1 .1 $OMEGA .1 $SIGMA .1 $ESTIMATION MAXEVALS=1900 PRINT=5 METHOD=0 (let's call this FIT A) gave a different fit from another (FIT B) where only the order of THETA( 4) and 5 was reversed, all else kept the same, as, . . SUM2=THETA(5)*COV4+THETA(4)*COV3 . . (even more strange, FIT B, the reversed order run, proved to be the good run!) NONMEM version V had the same result for both FITS A and B, as expected (and same as FIT B from version 6). FOCE gave good answers always in both NONMEM version V and version VI. But, the question remains about that dependence on ordering of the THETAs with the FO method for version VI. Any comments? Has there been discussion on this earlier that I missed? Thanks in advance. John PS I am aware of other *important* incosistencies between version VI and V from university colleagues working on a PK problem with FOCE... John C Lukas Strategic Consulting Services Pharsight Corp. line: + 33 492 726 495 cell: + 33 626 496 777

RE: RE: re: bug in NONMEM VI?

From: John Lukas Date: January 03, 2008 technical
Hi all, Bill is right. SLOW in NONMEM version 6 works fine giving the better answer in the minimization problem (FO method). Peter is also right in that COV3 and COV4 are more correlated than with the others! I suppose NONMEM version 5 was inherently SLOW (!)? Thanks and have a great new year! John John C Lukas Strategic Consulting Services Pharsight Corp. line: + 33 492 726 495 cell: + 33 626 496 777
Quoted reply history
________________________________ From: Bonate, Peter [mailto:[EMAIL PROTECTED] Sent: Wed 1/2/2008 10:03 PM To: Bill Bachman; John Lukas; [email protected] Subject: RE: [NMusers] RE: re: bug in NONMEM VI? I'd also like to see how correlated COV3 and COV4 are. This may be a collinearity issue. pete bonate Peter L. Bonate, PhD, FCP Genzyme Corporation Senior Director Clinical Pharmacology and Pharmacokinetics 4545 Horizon Hill Blvd San Antonio, TX 78229 USA [EMAIL PROTECTED] phone: 210-949-8662 fax: 210-949-8219 crackberry: 210-315-2713 ________________________________ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Bill Bachman Sent: Wednesday, January 02, 2008 2:38 PM To: 'John Lukas'; [email protected] Subject: RE: [NMusers] RE: re: bug in NONMEM VI? Have you tried both of these runs using the SLOW option on the estimation record? ________________________________ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Lukas Sent: Wednesday, January 02, 2008 2:36 PM To: [email protected] Subject: [NMusers] RE: re: bug in NONMEM VI? ________________________________ From: John Lukas Sent: Mon 12/17/2007 11:51 PM To: [email protected] Subject: re: bug in NONMEM VI? Dear Nmusers, A simple LME model run using NM version VI, FO method, with 4 covariates (5 THETA's) as, . . $PRED SUM1=THETA(2)*COV1+THETA(3)*COV2 SUM2=THETA(4)*COV4+THETA(5)*COV3 TLAU=THETA(1)+SUM1+SUM2 F=TLAU+ETA(1) Y=F+EPS(1) $THETA 4 .1 .1 .1 .1 $OMEGA .1 $SIGMA .1 $ESTIMATION MAXEVALS=1900 PRINT=5 METHOD=0 (let's call this FIT A) gave a different fit from another (FIT B) where only the order of THETA( 4) and 5 was reversed, all else kept the same, as, . . SUM2=THETA(5)*COV4+THETA(4)*COV3 . . (even more strange, FIT B, the reversed order run, proved to be the good run!) NONMEM version V had the same result for both FITS A and B, as expected (and same as FIT B from version 6). FOCE gave good answers always in both NONMEM version V and version VI. But, the question remains about that dependence on ordering of the THETAs with the FO method for version VI. Any comments? Has there been discussion on this earlier that I missed? Thanks in advance. John PS I am aware of other *important* incosistencies between version VI and V from university colleagues working on a PK problem with FOCE... John C Lukas Strategic Consulting Services Pharsight Corp. line: + 33 492 726 495 cell: + 33 626 496 777

RE: RE: re: bug in NONMEM VI?

From: Piet van der Graaf Date: January 04, 2008 technical
John/All, Although your problem has been solved, I am not sure this should be the end of this topic or the beginning of a more general discussion on how much we can 'trust' NONMEM and what we can do to standardise at least the basic algorithms we use to analyse data in our scientific community to get comparable and consistent results between individuals and groups. I am very concerned about findings like this: it seems to me that you only found out by chance that there was something wrong with your first fit and you only spotted the error because you used a simple model. How often does something like this happen without us noticing when we use more complex models? I had a similar issue recently when I found out that with another simple model NMV always gave wrong predictions for the first ID in the dataset, whereas NMVI (using exactly the same code) didn't. I still don't know why this happened, but I don't think I would have spotted it had I not compared the 2 NM versions which is obviously not something I would like to do for every analysis I perform. So, this brings me to my more specific point/comment/question: I am intrigued by your PS that you are 'aware of other *important* inconsistencies between version VI and V from university colleagues working on a PK problem with FOCE...', and was wondering if anyone has collated a list of these inconsistencies and also recommendations how to spot and solve them. For example, should we now always use the SLOW option in NMVI (to be honest I had never come across this before) or at least compare SLOW with NOSLOW for every analysis? Thanks, Piet Piet H. van der Graaf Preclinical M&S Pfizer IPC 654 Sandwich CT13 9NJ ++-44-1304-648330
Quoted reply history
________________________________ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Lukas Sent: 03 January 2008 07:52 To: [email protected] Subject: RE: [NMusers] RE: re: bug in NONMEM VI? Hi all, Bill is right. SLOW in NONMEM version 6 works fine giving the better answer in the minimization problem (FO method). Peter is also right in that COV3 and COV4 are more correlated than with the others! I suppose NONMEM version 5 was inherently SLOW (!)? Thanks and have a great new year! John John C Lukas Strategic Consulting Services Pharsight Corp. line: + 33 492 726 495 cell: + 33 626 496 777 ________________________________ From: Bonate, Peter [mailto:[EMAIL PROTECTED] Sent: Wed 1/2/2008 10:03 PM To: Bill Bachman; John Lukas; [email protected] Subject: RE: [NMusers] RE: re: bug in NONMEM VI? I'd also like to see how correlated COV3 and COV4 are. This may be a collinearity issue. pete bonate Peter L. Bonate, PhD, FCP Genzyme Corporation Senior Director Clinical Pharmacology and Pharmacokinetics 4545 Horizon Hill Blvd San Antonio, TX 78229 USA [EMAIL PROTECTED] phone: 210-949-8662 fax: 210-949-8219 crackberry: 210-315-2713 ________________________________ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Bill Bachman Sent: Wednesday, January 02, 2008 2:38 PM To: 'John Lukas'; [email protected] Subject: RE: [NMusers] RE: re: bug in NONMEM VI? Have you tried both of these runs using the SLOW option on the estimation record? ________________________________ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Lukas Sent: Wednesday, January 02, 2008 2:36 PM To: [email protected] Subject: [NMusers] RE: re: bug in NONMEM VI? ________________________________ From: John Lukas Sent: Mon 12/17/2007 11:51 PM To: [email protected] Subject: re: bug in NONMEM VI? Dear Nmusers, A simple LME model run using NM version VI, FO method, with 4 covariates (5 THETA's) as, . . $PRED SUM1=THETA(2)*COV1+THETA(3)*COV2 SUM2=THETA(4)*COV4+THETA(5)*COV3 TLAU=THETA(1)+SUM1+SUM2 F=TLAU+ETA(1) Y=F+EPS(1) $THETA 4 .1 .1 .1 .1 $OMEGA .1 $SIGMA .1 $ESTIMATION MAXEVALS=1900 PRINT=5 METHOD=0 (let's call this FIT A) gave a different fit from another (FIT B) where only the order of THETA( 4) and 5 was reversed, all else kept the same, as, . . SUM2=THETA(5)*COV4+THETA(4)*COV3 . . (even more strange, FIT B, the reversed order run, proved to be the good run!) NONMEM version V had the same result for both FITS A and B, as expected (and same as FIT B from version 6). FOCE gave good answers always in both NONMEM version V and version VI. But, the question remains about that dependence on ordering of the THETAs with the FO method for version VI. Any comments? Has there been discussion on this earlier that I missed? Thanks in advance. John PS I am aware of other *important* incosistencies between version VI and V from university colleagues working on a PK problem with FOCE... John C Lukas Strategic Consulting Services Pharsight Corp. line: + 33 492 726 495 cell: + 33 626 496 777

RE: RE: re: bug in NONMEM VI?

From: Alison Boeckmann Date: January 11, 2008 technical
See attached message. On Wed, 2 Jan 2008 23:51:51 -0800, "John Lukas" <[EMAIL PROTECTED]> said: > Hi all, > > Bill is right. SLOW in NONMEM version 6 works fine giving the better > answer in the minimization problem (FO method). Peter is also right in > that COV3 and COV4 are more correlated than with the others! I suppose > NONMEM version 5 was inherently SLOW (!)? > > Thanks and have a great new year! > > John > > > John C Lukas > > Strategic Consulting Services > > Pharsight Corp. > > > > line: + 33 492 726 495 > > cell: + 33 626 496 777 > > > ________________________________ >
Quoted reply history
> From: Bonate, Peter [mailto:[EMAIL PROTECTED] > Sent: Wed 1/2/2008 10:03 PM > To: Bill Bachman; John Lukas; [email protected] > Subject: RE: [NMusers] RE: re: bug in NONMEM VI? > > > I'd also like to see how correlated COV3 and COV4 are. This may be a > collinearity issue. > > pete bonate > > Peter L. Bonate, PhD, FCP > Genzyme Corporation > Senior Director > Clinical Pharmacology and Pharmacokinetics > 4545 Horizon Hill Blvd > San Antonio, TX 78229 USA > [EMAIL PROTECTED] > phone: 210-949-8662 > fax: 210-949-8219 > crackberry: 210-315-2713 > > > ________________________________ > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > On Behalf Of Bill Bachman > Sent: Wednesday, January 02, 2008 2:38 PM > To: 'John Lukas'; [email protected] > Subject: RE: [NMusers] RE: re: bug in NONMEM VI? > > > > Have you tried both of these runs using the SLOW option on the estimation > record? > > > > ________________________________ > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > On Behalf Of John Lukas > Sent: Wednesday, January 02, 2008 2:36 PM > To: [email protected] > Subject: [NMusers] RE: re: bug in NONMEM VI? > > > > > > > > ________________________________ > > From: John Lukas > Sent: Mon 12/17/2007 11:51 PM > To: [email protected] > Subject: re: bug in NONMEM VI? > > Dear Nmusers, > > > > A simple LME model run using NM version VI, FO method, with 4 covariates > (5 THETA's) as, > > . > > . > > $PRED > > > > SUM1=THETA(2)*COV1+THETA(3)*COV2 > > SUM2=THETA(4)*COV4+THETA(5)*COV3 > > > > TLAU=THETA(1)+SUM1+SUM2 > > F=TLAU+ETA(1) > > > > Y=F+EPS(1) > > > > $THETA 4 .1 .1 .1 .1 > > $OMEGA .1 > > $SIGMA .1 > > > > $ESTIMATION MAXEVALS=1900 PRINT=5 METHOD=0 > > > > (let's call this FIT A) gave a different fit from another (FIT B) where > only the order of THETA( 4) and 5 was reversed, all else kept the same, > as, > > . > > . > > SUM2=THETA(5)*COV4+THETA(4)*COV3 > > . > > . > > (even more strange, FIT B, the reversed order run, proved to be the good > run!) > > > > NONMEM version V had the same result for both FITS A and B, as expected > (and same as FIT B from version 6). FOCE gave good answers always in > both NONMEM version V and version VI. But, the question remains about > that dependence on ordering of the THETAs with the FO method for version > VI. > > > > Any comments? Has there been discussion on this earlier that I missed? > Thanks in advance. > > > > John > > PS I am aware of other *important* incosistencies between version VI and > V from university colleagues working on a PK problem with FOCE... > > > > > > John C Lukas > > Strategic Consulting Services > > Pharsight Corp. > > > > line: + 33 492 726 495 > > cell: + 33 626 496 777 > > > > > -- Alison Boeckmann [EMAIL PROTECTED] I was concerned about this issue, and John was kind enough to send me a control stream and data set. I cannot reproduce his results on my Sun workstation, but I do see a difference between NONMEM V and NONMEM VI minimum as NONMEM V. Surprisingly, the minimum value of the objective function is higher with VI than with V. According to Tom Ludden, "it is more common, when results differ between V and VI using a conditional estimation method, that the NONMEM VI results (without SLOW) are associated with a lower objective function value rather than a higher value. At least that has been our experience." My NONMEM V runs are consistent (model A vs. model B). My NONMEM VI runs are consistent (model A vs. model B). NONMEM VI reaches a different minimum than does NONMEM V unless the SLOW option of $ESTIM is used, in which case it reaches the same minimum as NONMEM V. There are 130 subjects with ages from <1 to 96. Of these, 31 are under the age of 21. Age and WT are clearly correlated till roughly that age, after that there is no strong correlation. (I looked a simple NONMEM scatter of AGE vs. WT). I cannot explain why NONMEM VI results differs from NONMEM V, but in my runs it has nothing to do with the order of thetas (model A vs. model B.) It is always disturbing when NONMEM VI gets a different result than NONMEM V, but this is a known problem and I cannot see that this particular case is evidence for a bug in VI. Legend for results: "VI" is NONMEM VI 1.0 "V" is NONMEM V 1.0 "A" uses the model COV2=THETA(4)*(AGE-36)+THETA(5)*(WT-65) "B" uses the model COV2=THETA(5)*(AGE-36)+THETA(4)*(WT-65) "s" is SLOW "n" is SLOW option omitted (default). Minimum value of the objective function: VInA: mm 45.414 VInB: mm 45.413 VIsA: mm 33.771 VIsB: mm 33.771 V__A: mm 33.771 V__B: mm 33.771 Final values of THETA: VInA: tf -3.27E-01 -2.85E-02 2.88E-04 -1.04E-03 1.44E-02 VInB: tf -3.27E-01 -2.85E-02 2.88E-04 1.44E-02 -1.05E-03 VIsA: tf -3.13E-01 -2.65E-02 2.87E-04 -1.40E-03 1.45E-02 VIsB: tf -3.13E-01 -2.65E-02 2.87E-04 1.45E-02 -1.40E-03 V__A: tf -3.13E-01 -2.65E-02 2.86E-04 -1.40E-03 1.45E-02 V__B: tf -3.14E-01 -2.65E-02 2.87E-04 1.45E-02 -1.38E-03 Final value of OMEGA: VInA: of 1.57E-02 VInB: of 1.57E-02 VIsA: of 2.49E-01 VIsB: of 2.49E-01 V__A: of 2.49E-01 V__B: of 2.50E-01 Final value of SIGMA: VInA: sf 4.51E-01 VInB: sf 4.51E-01 VIsA: sf 2.42E-01 VIsB: sf 2.42E-01 V__A: sf 2.43E-01 V__B: sf 2.42E-01 I use a Sun Solaris 2 workstation, using f90 (Forte Developer 7 Fortran 95 7.0 Patch 111714-04 2002/12/17)