DV Simulation

From:"Luciane Velasque" Subject:[NMusers] DV Simulation Date:Mon, 20 May 2003 16:19:02 -0300 Dear NM users: I have fit a model with some covariables and random effects. Now I would like to validate my model againt another group of individuals. Can I simulate the DV this new group of individuals using my model? How can I ask NONMEM to do this? Thanks Luciane V

From: Nick Holford Subject: Re: [NMusers] DV Simulation Date: Wed, 21 May 2003 08:55:58 +1200 Luciane, You can certainly simulate the DV using the model you have developed. A good reason for doing this is to perform a predictive check i.e. when you simulate from your model do the simulated observations resemble the original observations? There are many ways of answering this question. A relatively simple one is to simulate 1000 DV values at typical time points (e.g. the protocol observation times for your original data set) then find the 95% prediction interval from these 1000 DVs and compare it with a scatterplot of your original data. This is a form of internal validation that the model and parameter estimates are adequate for your data. Because of the approximations NONMEM has to make in its estimation you may be disappointed to find that the predictive check fails. Simply looking at the scatterplot may show you that 95% of your observations do not fall within the prediction intervals. Note that even if your model passes this predictive check it does not say that your model is more generally applicable e.g. to describe a new data set. The only way to test this kind of external validity is to collect new data. Simulation is not enough. Put the final estimates from NONMEM as initial estimates in your NM-TRAN control stream then add this record: $SIMULATION (20030521) ONLYSIM SUBPROBLEMS=1000 It is helpful to put this record somewhere e.g. in $ERROR REPI=IREP ; simulation replication index You will also need to add a suitable $TABLE record e.g. $TABLE ID TIME DV REPI NOAPPEND ONEHEADER FILE=mdl1000.sim The $SIMULATION record uses 20030521 as a random number seed. You should replace this with another big number if you want to create a different simulation data set. The simulated DV values will appear in the table file. You will have to work out your own method for extracting the DV values at each time point and working out the 95% prediction interval. The REPI item in the table file can be helpful in distinguishing each of the 1000 replications. Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

From:"Hutmacher, Matthew [Non-Employee/1820]" Subject: RE: [NMusers] DV Simulation Date: Tue, 20 May 2003 16:58:08 -0500 Luciane, To add to Nick's suggestion, as Nick states below: 'Put the final estimates from NONMEM as initial estimates in your NM-TRAN control stream then add this record: $SIMULATION (20030521) ONLYSIM SUBPROBLEMS=1000' If you don't want to imput the final estimates from your previous NONMEM into you simulation run by hand, you can use the $MSFI file. For example $SIMULATION (20030521) ONLYSIMULATION SUBPROBLEMS=10 $MSFI prevmodel.msf $TABLE ... model.sim Matt

From:Toufigh Gordi Subject: Re: [NMusers] DV Simulation Date: Thu, 29 May 2003 14:03:07 -0400 Hi! I am trying to simulate 100 subjects using the follwing coding: $PROB ALL DOSES ; 2-COMP NLCL $INPUT ID AMT RATE DUR DATE=DROP TIME DV EVID SEXF PKPD $DATA simbase2.csv IGNORE=# $SUBROUTINE ADVAN6 TRANS1 TOL=5 $MODEL NCOMP=2 COMP=CENT COMP=PERIPH $PK "FIRST " COMMON /PRCOMG/ IDUM1,IDUM2,IMAX,IDUM4,IDUM5 " INTEGER IDUM1,IDUM2,IMAX,IDUM4,IDUM5 " IMAX=1000000000 VM=THETA(1)*EXP(ETA(1)) KM=THETA(2) V1=THETA(3)*EXP(ETA(2)) Q=THETA(4) V2=THETA(5) D1=DUR S1=V1 $DES CONC=A(1)/V1 DADT(1)=-A(1)*(VM/(KM+CONC))/V1-A(1)*Q/V1+A(2)*Q/V2 DADT(2)=A(1)*Q/V1-A(2)*Q/V2 CLN=VM/(KM+CONC) K12=Q/V1 K21=Q/V2 K10=CLN/V1 BETA=0.5*((K12+K21+K10)-SQRT((K12+K21+K10)**2-4*K21*K10)) BTH=LOG(2)/BETA VSS=V1+V2 $ERROR Y=F*(1+EPS(1)) REPI=IREP $THETA (2.92 FIX) ;1 VM $THETA (2.87 FIX) ;2 KM $THETA (42.6 FIX) ;3 V1 $THETA (0.421 FIX) ;4 Q $THETA (48.1 FIX) ;5 V2 $OMEGA 0.105 FIX ;1 IIV VM $OMEGA 0.0176 FIX ;2 IIV V1 $SIGMA 0.00987 FIX $SIMULATION (20030521) ONLYSIM SUBPROBLEMS=100 $TABLE ID TIME DV REPI NOPRINT ONEHEADER FILE=SDTAB101 $TABLE ID VSS CLN K12 K21 K10 BETA BTH NOPRINT ONEHEADER FILE=MYTAB101 The simulation works and the tables are generated. However, in the SDTAB101 file I get the results for only 1 subject and not 100, as written in the model. Could anyone comment on what mistake I am making? Thank you. Toufigh Gordi

From: Nick Holford Subject: Re: [NMusers] DV Simulation Date: Fri, 30 May 2003 06:49:09 +1200 Toufigh, The only place you have specified 100 in the "model" (NM-TRAN abbreviated code) is in the SUBPROBLEMS option of trhe $SIMULATION record. This asks NONMEM to do 100 simulation runs. Each simulation run will use the subjects in the file specified in $DATA. If there is only 1 subject in this file then only 1 subject will appear in each of the 100 output simulation tables. If you want 100 subjects then you must provide 100 subjects in the file you specify in $DATA. Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

From:"Liu, Jing" Subject: RE: [NMusers] DV Simulation Date: Thu, 29 May 2003 15:14:39 -0400 Hi, Nick: Shall NOPREDICTION be specified in $SIMULATION? Jing

From: "Ludden, Thomas" Subject:RE: [NMusers] DV Simulation Date:Thu, 29 May 2003 15:17:44 -0400 Taufigh, I believe you can obtain the table output for all 100 subproblems in a single table by including the FORWARD option on the $TABLE record. This causes the output for a subproblem to be appended to the end of previous output to that table. In fact, this option may be necessary only on the first $TABLE record. It may be the default behavior for $TABLE records after the first one. Tom Ludden

From: Nick Holford Subject:Re: [NMusers] DV Simulation Date: Fri, 30 May 2003 07:58:31 +1200 Jing, I have never noticed this option before. I have also done simulations of odd-type data in collaboration with Stuart Beal and he did not suggest using it. NMHELP says this: PREDICTION | Permitted with ONLYSIM, but is not required. Indicates that Y | (with NM-TRAN abbreviated code) or F (with a user-supplied PRED | or ERROR routine) will be set to a simulated value. Also, etas | (if any) are population etas only if epsilons also appear. This | is the default. | NOPREDICTION | Permitted with ONLYSIM, but is not required. Indicates that the | DV data item will be set directly to the simulated value in | abbreviated or user-supplied code (as when simulating odd-type | data). Also, etas (if any) are always population etas, even if | epsilons do not appear. This "help" is a mystery to me. PREDICTION refers to what happens to Y and F while NOPREDICTION refers to what happens to DV. Can anyone help translate this into something of practical use? NMHELP says this about FORWARD: NOFORWARD | When a table file is opened during a given problem, it is posi- | tioned at the start of the file. This is the default. | FORWARD | When a table file is opened during a given problem, it is for- | warded to the end of the file. | This is another option I have never used. It refers to the behaviour if you have more than one $PROBLEM in your control stream and you use the same FILE in $TABLE in different problems. This means that output from each problem will be added to the end of the output from previous problems in the same table file. Note that a problem is not the same as a $SIMULATION SUBPROBLEM. In $SIMULATION the output from each subproblem is automatically appended to the table file. FORWARD (on the first $TABLE record) caused output to be appended to the first table file I had created with a previous run of the control stream (note only one $PROBLEM in the control stream). The second table file in my control stream was overwritten when I re-ran the problem. To have output appended to the second table file I had to specify the FORWARD option explicitly for that file. So based on my testing I would have to disagree with Tom's suggestion that FORWARD applied to the first $TABLE record becomes the default for subsequent tables. The observed behaviour is more flexible and therefore IMHO preferable. Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

From:Toufigh Gordi Subject:RE: [NMusers] DV Simulation Date:Thu, 29 May 2003 16:07:31 -0400 Thanks for all the suggestions! I tried the simplest one, i.e. use of FORWARD in the $TABLE and it gave me the 100 simulations, based on a data file containing 1 subject. TG

From:"Ekaterina Gibiansky" Subject:Re: [NMusers] DV Simulation Date: Thu, 29 May 2003 16:30:42 -0400 Nick, Jing, I had to use NOPREDICTION option when Y was probability (proportional odds model), so no EPS was involved. Katya

From: "Liu, Jing" Subject:RE: [NMusers] DV Simulation Date:Thu, 29 May 2003 17:03:56 -0400 Hi, All: Thank you all for your kind response. I had the same situation when I had to use NOPREDICTION in the proportional odds model. Otherwise I got different ETA at different time points for the same subject. I guess we need to use the NOPREDICTION option when there is only one degree of random effect. Thanks again, Jing

From:Nick Holford Subject:Re: [NMusers] DV Simulation Date: Fri, 30 May 2003 09:18:39 +1200 I have to say that I am now finding the way $SIMULATION deals with $TABLE output more complex than I thought originally. It seems that when using $SIMULATION the $TABLE output from each SUBPROBLEM is concatenated to the $TABLE FILE only if there is one $TABLE record. This is similar to using FORWARD but if the problem is re-run a new $TABLE FILE is created rather than concatenating output to an existing file. If there is a second $TABLE record then output for each SUBPROBLEM overwrites the output from previous subproblems for both $TABLE FILEs. One then needs to use the FORWARD option to concatenate output from each SUBPROBLEM to each $TABLE FILE. If you use $SIMULATION and only one $TABLE record then you will probably get what you expect (which is the way it was for me until today!). But if you use the FORWARD option you must be careful because if you re-run the problem (as you might well do while debugging) be aware that the output from each run will be concatenated to the table file from earlier runs. -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

From: "Ludden, Thomas" Subject: RE: [NMusers] DV Simulation Date:Thu, 29 May 2003 17:27:32 -0400 In NONMEM V the need for the FORWARD option is a bit complex. That is why I purposely chose to use "may be" instead of the more definite "will be" since I was relying upon distant memory. :-) As Nick pointed out, when there is more than one table file the default is NOFORWARD and the FORWARD option is required for each $TABLE record. However, when only one table record is specified and there are SUBPROBLEMS the tabled items for all subproblems are placed in the single table file without using the FORWARD option. Marc Gastonguay noted this behavior a little over a year ago. The inconsistency will be fixed and the help file updated in version VI. For those who might be interested, there is a bit of history to the way that tables are produced. NONMEM originally produced only one table. Somewhere about version 4 or 5, I believe, it became possible to output up to 10 tables. It appears that there are different sections of Fortran code used for one table (old code) versus more than one table (new code). An inquiry to Professor Stuart Beal yielded the following explanation. ...there are three errors. First, the HELP item for $TABLE talks about FORWARD and NOFORWARD in terms of separate problems. However, the same rules that govern separate problems also govern separate subproblems, and this isn't mentioned. So we'll fix this with HELP Ver VI. Second, the control stream with two $TABLE records runs correctly. This is because, as may be seen in the help item, NOFORWARD is the default. But the error is with the control stream with one $TABLE record. The table should have only the output from the last subproblem. So we'll fix this with Ver VI. Third, the HELP item for $TABLE is incomplete also in that it doesn't mention the following. Within the same (sub)problem, with any sequence of contiguous $TABLE records using the same name with the FILE option, all but the first of the sequence will be treated as though they contain the FORWARD option, even though some may contain NOFORWARD (explicitly), or some may contain neither FORWARD nor NOFORWARD. For to do otherwise, makes no sense. Tom Ludden

From: Nick Holford Subject: Re: [NMusers] DV Simulation Date: Fri, 30 May 2003 09:32:05 +1200 Katya, Can you explain what happens when you don't use NOPREDICTION with your odd type data? Your reply does not explain why it was needed. What would happen if you were not simulating odd type data and used this option? Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

From:Nick Holford Subject: Re: [NMusers] DV Simulation Date:Fri, 30 May 2003 19:08:04 +1200 Tom, Thanks for the confirmation of more strange quirks of NONMEM. But one of the quirks is actually really useful and I am shocked to see Stuart's assertion that this useful feature is an error that needs to be fixed. > But the error is with the control stream with one $TABLE record. The > table should have only the output from the last subproblem. > So we'll fix this with Ver VI. The current behaviour with one $TABLE FILE and multiple $SIMULATION SUBPROBLEMS works just fine for me because if I ask for 1000 subproblems and also want the $TABLE FILE output then I get output from every SUBPROBLEM by default. But if this "error" is fixed I will only get the $TABLE FILE output from the last subproblem -- throwing away the output from all the previous problems. Why would this be preferable default behaviour? If I really don't want any $TABLE FILE output then I simply dont include $TABLE records. What use would it be to have just the last $TABLE FILE out of 1000 SUBPROBLEMS? The FORWARD option is a horrible solution. As I pointed out earlier this option concatenates output to any existing file even when control streams are executed as separate runs. How often does anyone want to do this? What really needs to be fixed is the utter unreliability of being able to do $SIMULATION and $ESTIMATION in the same problem. If the simulation step generates a strange parameter or DV (as it necessarily will because it can sample anywhere in the distributions) then NONMEM finds it cannot start the $ESTIMATION step for some reason and abandons not only that SUBPROBLEM but all subsequent SUBPROBLEMS! This usually happens around SUBPROBLEM 500 out of 1000 -- hours if not days later. This NONMEM bug is the reason I am forced to use ONLYSIM to create all the simulated data sets I need (1000 or more sometimes) then devise some ad hoc way to work my way through the 1000 simulated data sets one at a time to do the ESTIMATION. Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/

From: "Ludden, Thomas" Subject: RE: [NMusers] DV Simulation Date: Fri, 30 May 2003 10:05:27 -0400 Nick, Thank you for your comments. I will pass them along to Stuart Beal. The fact that NONMEM "crashes" during repetition of subproblems or superproblems is a difficulty that most users have experienced at one time or another. Stuart and I have discussed the need for correcting this in some way. The current structure of the NONMEM code may make it difficult to eliminate all crashes but a number of them may be preventable. Stuart is currently working on the numerics and search algorithms. I can report that the current version of NONMEM VI beta behaves much more uniformly across Compaq Visual Fortran compiler 6.6b options and in comparison to the Sun Workshop Fortran compiler 4.0. Thanks again for your comments. We hope we can address at least some of the problems you describe. Tom Ludden

From: "Ekaterina Gibiansky" Subject: Re: [NMusers] DV Simulation Date: Fri, 30 May 2003 10:31:55 -0400 Nick, I get an error message from NMTRAN: 122 USE OF ETA IMPLIES POPULATION DATA, BUT OTHER INFO. IS NOT CONSISTENT And that other info, as I understand (from having different wording in some runs or from the manual, I do not remember), is absense of EPS in the ERROR block. The ERROR block was something of this kind: Y = P5 ;P5 = P(Y=5) IF (DV.EQ.4) Y = P4 ;P4 = P(Y=4) IF (DV.EQ.3) Y = P3 ; etc IF (DV.EQ.2) Y = P2 IF (DV.EQ.1) Y = P1 P1 - P5 were defined by PKPD through thetas and etas. The data file contained ID variable and population data. Katya

From: Nick Holford Subject: Re: [NMusers] DV Simulation Date: Thu, 05 Jun 2003 16:16:20 +1200 Katya, Thank you for the error message and example code. But I am still mystified by the NOPREDICTION option. I have done some experimentation with a simple binary DV model. If I use $SIM and $EST in the same $PROB the estimation results are identical to those obtained from running $SIM ONLYSIM NOPREDICTION in one problem then running $EST on the output of the first problem. So why is NOPREDICTION needed with ONLYSIM? The DV values generated by both methods are the same. Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/ _______________________________________________________