Permutation test with small number of possible permutations

From: "Mouksassi Mohamad-Samer" mohamad-samer.mouksassi@umontreal.ca Subject: [NMusers] Permutation test with small number of possible permutations Date: Thu, November 25, 2004 10:37 am Dear NONMEM users, I am a new NONMEM user. I am working on modeling the Pharmacokinetics of two drugs. I have only eight patients per drug. I was screening for any potential significant covariates, but I know that the results would be suspect and the p values are bigger than the nominal when we have a small number of individuals 1. I am using FOCE INTER method. So I decided to do a permutation randomization test 1 for the significant covariates at the theoretical level of 0.05 to have the actual p values. (Although this approach was subsequently criticised 2 and I am aware of the results and conclusions of Ribbing et al.) To perform a permutation test, ideally we must do all the possible permutations but in practice this number is astronomical so we sample say 1000 from the possible permutations and do the test. My question is if we have a small number of permutations must we only do all possible permutations? In my case I have a dichotomous covariate taking values 0 and 1, 4 patients (0) and 4 with (1) so the number of possible permutations is: 8! / (4! * 4!) = 70. I am using S-plus to do the test, is there a function that do the permutation that guarantee that the permutation at each time is different from all the previous ones? sample(,replace=F) do random permutations only. Samer MOUKSASSI 1: Wahlby U, Jonsson EN, Karlsson MO. Assessment of actual significance levels for covariate effects in NONMEM. J Pharmacokinet Pharmacodyn. 2001 Jun;28(3):231-52. 2: Beal SL. Commentary on significance levels for covariate effects in NONMEM. J Pharmacokinet Pharmacodyn. 2002 Aug;29(4):403-10; discussion 411-2. No abstract available. 2: Ribbing J, Jonsson EN Power, selection bias and predictive performance of the Population Pharmacokinetic Covariate Model. J Pharmacokinet Pharmacodyn. 2004 Apr;31(2):109-34.