From: "Mouksassi Mohamad-Samer" mohamad-samer.mouksassi@umontreal.ca
Subject: [NMusers] Permutation test with small number of possible permutations
Date: Thu, November 25, 2004 10:37 am
Dear NONMEM users,
I am a new NONMEM user. I am working on modeling the Pharmacokinetics
of two drugs. I have only eight patients per drug.
I was screening for any potential significant covariates, but I know that
the results would be suspect and the p values are bigger than the nominal
when we have a small number of individuals 1. I am using FOCE INTER method.
So I decided to do a permutation randomization test 1 for the significant
covariates at the theoretical level of 0.05 to have the actual p values.
(Although this approach was subsequently criticised 2 and I am aware of
the results and conclusions of Ribbing et al.)
To perform a permutation test, ideally we must do all the possible permutations
but in practice this number is astronomical so we sample say 1000 from the
possible permutations and do the test.
My question is if we have a small number of permutations must we only do
all possible permutations?
In my case I have a dichotomous covariate taking values 0 and 1, 4 patients
(0) and 4 with (1) so the number of possible permutations is:
8! / (4! * 4!) = 70.
I am using S-plus to do the test, is there a function that do the permutation
that guarantee that the permutation at each time is different from all the
previous ones? sample(,replace=F) do random permutations only.
Samer MOUKSASSI
1: Wahlby U, Jonsson EN, Karlsson MO.
Assessment of actual significance levels for covariate effects in NONMEM.
J Pharmacokinet Pharmacodyn. 2001 Jun;28(3):231-52.
2: Beal SL.
Commentary on significance levels for covariate effects in NONMEM.
J Pharmacokinet Pharmacodyn. 2002 Aug;29(4):403-10; discussion 411-2. No
abstract available.
2: Ribbing J, Jonsson EN
Power, selection bias and predictive performance of the Population
Pharmacokinetic Covariate Model.
J Pharmacokinet Pharmacodyn. 2004 Apr;31(2):109-34.
Permutation test with small number of possible permutations
From: "Nick Holford" n.holford@auckland.ac.nz
Subject: RE: [NMusers] Permutation test with small number of possible permutations
Date: Fri, November 26, 2004 5:05 pm
Mouksassi,
This a good question. I am afraid I cannot help you with an algorithm to generate
all possible permutations but I believe that in principle this is the correct
approach. It is the underlying method of Fisher's Exact Test i.e. to compute P from
all possible permutations of the data. The empirical methods of sampling at random
from randomly generated permuations are easy to implement and usually the only
practical method for anything more than small samples.
I don't think that Stu Beal's commentary contained any substantive criticism of the
basic method of employing a randomization test procedure to define a null
distribution and estimate the P value. Indeed he has been co-author of a paper which
have used exactly this method e.g. Zhang L, Beal SL, Sheiner LB. Simultaneous vs.
sequential analysis for population PK/PD data I: best-case performance. J
Pharmacokinet Pharmacodyn 2003;30(6):387-404.
Nick
--
Nick Holford, Dept Pharmacology &anp Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556
http://www.health.auckland.ac.nz/pharmacology/staff/nholford/
From: Leonid Gibiansky"
Subject: RE: [NMusers] Permutation test with small number of possible permutations
Date: Fri, November 26, 2004 8:47 pm
Samer,
Actually, there will be only 35 different permutations (permutation
1234 versus 5678 is the same as 5678 versus 1234). The code below gives them all:
N <- 0
i <-1
for(j in (i+1):6)
for(k in (j+1):7)
for(n in (k+1):8) {
if(i < j & j < k & k < n) {
N <- N+1
print(paste("Permutation", N,", class A subjects:", i,j,k,n))
}
}
P-value based on many (say, 1000) random permutations should be approximately the
same as a p-value based on all (35 in this case) permutations.
Leonid
From: "Mouksassi Mohamad-Samer" mohamad-samer.mouksassi@umontreal.ca
Subject: RE: [NMusers] Permutation test with small number of possible permutations
Date: Fri, November 26, 2004 11:05 pm
Leonid,
First thanks for the script.
Is there a general formula for the number of permutations?
If class A is assigned to 5678 or 1234 it would be different because we would have then :
A A A A B B B B
B B B B A A A A these are not equivalent, am I wrong?
ID 1 2 3 4 5 6 7 8
The p value should be the same but the variance of it would be much bigger as it is (p*(1-p))/n for a computed 0.05
sd(p)= 0.00689 (1000)
sd(p)= 0.036 (35) Too large ....to be useful
sd(p)= 0.026 (70)
Nick,
maybe I used the wrong word criticized, Stu Beal said that this may
not hold in all the settings and he added it is instructive to consider
another idea for calibrating the critical value as the posterior
predictive check.
Samer
From: "Leonid Gibiansky" leonidg@metrumrg.com
Subject: RE: [NMusers] Permutation test with small number of possible permutations
Date: Sat, November 27, 2004 1:48 am
Samer
A A A A B B B B
B B B B A A A A
are equivalent, solutions will differ by the sign of the effect or,
equivalently, by the class label only (for example, CL_A > CL_B versus CL_B > CL_A).
A general formula for the number of permutations (for two classes A and B) is N! /( K!*(N-K)!),
where N is the number of subjects, K is the number of elements in class A. But in this
particular case, the number of elements in classes A and B are the same. This creates an
additional symmetry. This symmetry can be used to reduce the number of permutations. You can
use 70 permutations, if you like, but the result will be the same. If you like to use 70, just
replace the first line of the script (i <- 1) by
for(i in 1:5)
A more general script (for more than two classes) is more difficult to write, I did it at
some point in FORTRAN but I do not have it handy now.
I am not sure that your interpretation of the variance is correct. These 35 (or 70) permutations
are not random: they are carefully chosen to represent all possible permutations. Thus, they
give you an exact distribution, equivalent to the infinite number of random permutations. You
cannot use binomial sd(p) formula in this situation.
Leonid
From: "Mouksassi Mohamad-Samer" mohamad-samer.mouksassi@umontreal.ca
Subject: RE: [NMusers] Permutation test with small number of possible permutations
Date: Sat, November 27, 2004 1:10 pm
Leonid,
Thanks for the comments.
I do have actually a covariate with 3 classes and I see that we must
consider the symmetry. ( As we usually there is no interaction between
the covariates in the model )
I agree with you that this is an exact distribution so the quantiles describe
it and the sd is not applicable here as it is not a random sample
Samer
From: andreas.krause@pharma.novartis.com
Subject: RE: [NMusers] Permutation test with small number of possible permutations
Date: Thu, December 2, 2004 5:12 am
The approach looks reasonable. Using all possible permutations or sampling
with a sample size similar to the number of possible permutations does not
make much difference in practice going by my experience.
If you want to go down that road and as you say you use S-Plus anyway, you
can generate all permutations of a given set of values there. I don't
think there is a canned function for it, but this one will do:
# Create all possible subsets of a given vector for a given sample size.
# This function generates a matrix with each row containing one of the
possible combinations.
# Andreas Krause, May 2002
all.combinations <- function(values=c("one", "two", "three"), samples=2)
{
nvalues <- length(values)
out <- matrix(NA, nvalues^samples, samples)
for(i in 1:ncol(out))
out[, i] <- rep(values, rep(nvalues^(i-1),
nvalues))
return(out)
}
You could eliminate from the result the redundant combinations as has been
suggested. If computation time is small, you might as well ignore it
though.
If you want to go for it, develop a code to characterize each combination
of values. That code must be identical for redundant combinations and
non-identical otherwise. Derive that code for each row of the matrix
obtained from the above and eliminate the rows corresponding to non-unique
codes. S-Plus functions to use are apply and duplicated.
Hope this helps.
Andreas
Dr. Andreas Krause
Novartis Pharma AG
Biostatistics
Modeling and Simulation
WSJ-27.1.073
4002 Basel
Switzerland
From: "Kevin Dykstra" kdykstra@pharsight.com
Subject: RE: [NMusers] Permutation test with small number of possible permutations
Date: Thu, December 2, 2004 9:52 am
Andreas and Mouksassi--There is, in fact, a canned function in SPlus to create all
possible combinations of several vectors. This would be a fairly natural way to do
the exersize you outline. It is expand.grid:
> a = c("A", "B", "C")
> b = c(1, 2)
> expand.grid(a, b)
Var1 Var2
1 A 1
2 B 1
3 C 1
4 A 2
5 B 2
6 C 2
Kind regards,
Kevin Dykstra
Pharsight
_______________________________________________________