Dear Eduard,
Have you tried SAEM or IMP methods to estimate the variability ?
Also did you check how good are your population predictions when compared to individual observations? I'm curious to see whether your model with the Markovian and time components would not be capturing all the variability, conditional on the data.
Kind regards,
Pascal Girard
Le mardi 5 février 2019 à 11:13:36 UTC+1, Eduard Schmulenson <e.schmulenson_at_uni-bonn.de> a écrit :
Dear all,
I am currently trying to model the transitions between four adverse event grades (0-3) using a continuous-time Markov modeling approach. I have included a dose effect as well as a time effect on the transition constants. Overall, the parameters are well estimated and the VPC looks also quite good.
However, the model does not have any IIV or other variability incorporated, so no individual predictions can be made. I have tried different approaches to include variability:
- Six different etas: a) One eta per transition constant without a block structure (this has resulted in rounding errors), b) with a full block structure (see Lacroix BD et al. CPT PSP 2014), also with rounding errors and c) with two OMEGA BLOCK(3) structures which solely include “forward” and “backward” transition constants, respectively (also with rounding errors).
- Two different etas: One mutual eta on “forward” and “backward” transition constants (shrinkage values of ~ 40 and 60%, respectively, which do not lower after including the dose effect. The impact of time cannot be estimated anymore)
- Just one eta on every transition constant (shrinkage value of 46% which slightly increases after including the dose and time effect.
The etas were added as exponential variables.
Other tested covariates were not significant or resulted in run errors when a bootstrap was performed.
Are there any other possibilities to incorporate variability in this type of model? Or is it solely a data-dependent issue? You can find the control stream (without any IIV) below.
My second question is about the assessment of predictive performance in the same model. One can compare the observed proportions of an adverse event grade vs. the simulated probability or the observed vs. simulated grade. Is there a meaningful error which I can calculate in order to assess bias and precision? Would be a median prediction error and a median absolute prediction error appropriate for this type of data? And what kind of error would you suggest when one has to calculate a relative error which would include a division by 0?
Thank you very much in advance.
Best regards,
Eduard
##########################################
$ABB COMRES = 1
$SUBROUTINES ADVAN6 TOL = 4
$MODEL
NCOMP = 4
COMP = (G0) ; No AE
COMP = (G1) ; Mild AE
COMP = (G2) ; Moderate AE
COMP = (G3) ; Severe AE
$PK
IF(NEWIND.NE.2) THEN
PSDV = 0
COM(1) = 0
ENDIF
PRSP = PSDV ; Previous DV
IF(PRSP.EQ.1) COM(1) = 0
IF(PRSP.EQ.2) COM(1) = 1
IF(PRSP.EQ.3) COM(1) = 2
IF(PRSP.EQ.4) COM(1) = 3
F1 = 0
F2 = 0
F3 = 0
F4 = 0
IF(COM(1).EQ.0) F1 = 1
IF(COM(1).EQ.1) F2 = 1
IF(COM(1).EQ.2) F3 = 1
IF(COM(1).EQ.3) F4 = 1
TVK01 = THETA(1)
K01 = TVK01*EXP(ETA(1))
TVK12 = THETA(2)
K12 = TVK12
TVK23 = THETA(3)
K23 = TVK23
TVK10 = THETA(4)
K10 = TVK10
TVK21 = THETA(5)
K21 = TVK21
TVK32 = THETA(6)
K32 = TVK32
TVKT = THETA(8)
KT = TVKT
$DES
K01F = K01*EXP(KT*T) ; Time effect
K12F = K12*EXP(KT*T)
K23F = K23*EXP(KT*T)
K10B = K10*EXP(THETA(7)*(DOSEDAY-3000)) ; Dose effect
K21B = K21*EXP(THETA(7)*(DOSEDAY-3000))
K32B = K32*EXP(THETA(7)*(DOSEDAY-3000))
DADT(1) = K10B*A(2) - K01F*A(1) ; Grade 0
DADT(2) = K01F*A(1) + K21B*A(3) - A(2)*(K10B + K12F) ; Grade 1
DADT(3) = K12F*A(2) + K32B*A(4) - A(3)*(K21B + K23F) ; Grade 2
DADT(4) = K23F*A(3) - K32B*A(4) ; Grade 3
$ERROR
Y = 1
IF(DV.EQ.1.AND.CMT.EQ.0) Y = A(1)
IF(DV.EQ.2.AND.CMT.EQ.0) Y = A(2)
IF(DV.EQ.3.AND.CMT.EQ.0) Y = A(3)
IF(DV.EQ.4.AND.CMT.EQ.0) Y = A(4)
P0 = A(1)
P1 = A(2)
P2 = A(3)
P3 = A(4)
; Cumulative probabilities
CUP0 = P0
CUP1 = P0 + P1
CUP2 = P0 + P1 + P2
CUP3 = P0 + P1 + P2 + P3
; Start of simulation block
IF(ICALL.EQ.4) THEN
IF(CMT.EQ.0) THEN
CALL RANDOM (2,R)
IF(R.LE.CUP0) DV = 1
IF(R.GT.CUP0.AND.R.LE.CUP1) DV = 2
IF(R.GT.CUP1.AND.R.LE.CUP2) DV = 3
IF(R.GT.CUP2) DV = 4
ENDIF
ENDIF
; End of simulation block
PSDV=DV
$THETA
…
$OMEGA
0 FIX
$COV PRINT=E
;$SIM (7776) (8877 UNIFORM) ONLYSIM NOPREDICTION
$EST METHOD=1 LAPLACIAN LIKE SIG=2 PRINT=1 MAX99 NOABORT
_____________________
Eduard Schmulenson, M.Sc.
Apotheker/Pharmacist
Klinische Pharmazie
Pharmazeutisches Institut
Universität Bonn
An der Immenburg 4
D-53121 Bonn
Tel.: +49 228 73-5242
e.schmulenson_at_uni-bonn.de
(image/gif attachment: image002.gif)
(image/png attachment: image001.png)
Ordinal data model: Incorporation of variability & assessment of predictive performance
Dear Eduard,
Did you check the distribution of the number of transitions per individual?
If this number is low, IIV is difficult to estimate and high shrinkage is
expected, as well as a non-normal distribution of the post-hoc estimates.
With respect to your second question, you can obtain a rough idea about
accuracy by calculating some kind of expected value for the outcome at each
observation (i.e. multiply the probability for each observation with the
corresponding order of the observation category (1, 2, 3 and 4 in your
case) for the different possible observations and sum the products) and
subtract the actual observation (i.e. the order of the observed category).
For a rough idea about precision, you can just look at the predicted
probability for the observed outcome.
To compensate for the lack of weighting/normalisation of these "residuals"
you can calculate them for all your VPC points and compare the
distributions of these "residuals" in your simulated datasets with the
distribution in your observed dataset.
I have done this only once, but it was useful to evaluate the IIV structure
and to avoid overestimation of IIV.
Kind regards,
Wilbert
Op di 5 feb. 2019 om 12:16 schreef Girard Pascal <pascal.girard7_at_yahoo.fr>:
> Dear Eduard,
>
> Have you tried SAEM or IMP methods to estimate the variability ?
>
> Also did you check how good are your population predictions when compared
> to individual observations? I'm curious to see whether your model with th
e
> Markovian and time components would not be capturing all the variability,
> conditional on the data.
>
> Kind regards,
>
> Pascal Girard
>
Quoted reply history
> Le mardi 5 février 2019 à 11:13:36 UTC+1, Eduard Schmulenson <
> e.schmulenson_at_uni-bonn.de> a écrit :
>
>
> Dear all,
>
>
>
> I am currently trying to model the transitions between four adverse event
> grades (0-3) using a continuous-time Markov modeling approach. I have
> included a dose effect as well as a time effect on the transition
> constants. Overall, the parameters are well estimated and the VPC looks
> also quite good.
>
> However, the model does not have any IIV or other variability
> incorporated, so no individual predictions can be made. I have tried
> different approaches to include variability:
>
> - Six different etas: a) One eta per transition constant without
> a block structure (this has resulted in rounding errors), b) with a full
> block structure (see Lacroix BD et al. CPT PSP 2014), also with rounding
> errors and c) with two OMEGA BLOCK(3) structures which solely include
> “forward” and “backward” transition const
ants, respectively (also with
> rounding errors).
>
> - Two different etas: One mutual eta on “forward
and “backward”
> transition constants (shrinkage values of ~ 40 and 60%, respectively, whi
ch
> do not lower after including the dose effect. The impact of time cannot b
e
> estimated anymore)
>
> - Just one eta on every transition constant (shrinkage value of
> 46% which slightly increases after including the dose and time effect.
>
>
>
> The etas were added as exponential variables.
>
> Other tested covariates were not significant or resulted in run errors
> when a bootstrap was performed.
>
>
>
> Are there any other possibilities to incorporate variability in this type
> of model? Or is it solely a data-dependent issue? You can find the contro
l
> stream (without any IIV) below.
>
>
>
>
>
> My second question is about the assessment of predictive performance in
> the same model. One can compare the observed proportions of an adverse
> event grade vs. the simulated probability or the observed vs. simulated
> grade. Is there a meaningful error which I can calculate in order to asse
ss
> bias and precision? Would be a median prediction error and a median
> absolute prediction error appropriate for this type of data? And what kin
d
> of error would you suggest when one has to calculate a relative error whi
ch
> would include a division by 0?
>
>
>
> Thank you very much in advance.
>
>
>
> Best regards,
>
> Eduard
>
>
>
> ##########################################
>
> $ABB COMRES = 1
>
> $SUBROUTINES ADVAN6 TOL = 4
>
> $MODEL
>
> NCOMP = 4
>
> COMP = (G0) ; No AE
>
> COMP = (G1) ; Mild AE
>
> COMP = (G2) ; Moderate AE
>
> COMP = (G3) ; Severe AE
>
> $PK
>
> IF(NEWIND.NE.2) THEN
>
> PSDV = 0
>
> COM(1) = 0
>
> ENDIF
>
> PRSP = PSDV ; Previous DV
>
>
>
> IF(PRSP.EQ.1) COM(1) = 0
>
> IF(PRSP.EQ.2) COM(1) = 1
>
> IF(PRSP.EQ.3) COM(1) = 2
>
> IF(PRSP.EQ.4) COM(1) = 3
>
>
>
> F1 = 0
>
> F2 = 0
>
> F3 = 0
>
> F4 = 0
>
>
>
> IF(COM(1).EQ.0) F1 = 1
>
> IF(COM(1).EQ.1) F2 = 1
>
> IF(COM(1).EQ.2) F3 = 1
>
> IF(COM(1).EQ.3) F4 = 1
>
>
>
> TVK01 = THETA(1)
>
> K01 = TVK01*EXP(ETA(1))
>
>
>
> TVK12 = THETA(2)
>
> K12 = TVK12
>
>
>
> TVK23 = THETA(3)
>
> K23 = TVK23
>
>
>
> TVK10 = THETA(4)
>
> K10 = TVK10
>
>
>
> TVK21 = THETA(5)
>
> K21 = TVK21
>
>
>
> TVK32 = THETA(6)
>
> K32 = TVK32
>
>
>
> TVKT = THETA(8)
>
> KT = TVKT
>
>
>
> $DES
>
> K01F = K01*EXP(KT*T) ; Time effect
>
> K12F = K12*EXP(KT*T)
>
> K23F = K23*EXP(KT*T)
>
>
>
> K10B = K10*EXP(THETA(7)*(DOSEDAY-3000)) ; Dose effect
>
> K21B = K21*EXP(THETA(7)*(DOSEDAY-3000))
>
> K32B = K32*EXP(THETA(7)*(DOSEDAY-3000))
>
>
>
> DADT(1) = K10B*A(2) - K01F*A(1) ; Grade 0
>
> DADT(2) = K01F*A(1) + K21B*A(3) - A(2)*(K10B + K12F) ; Grade 1
>
> DADT(3) = K12F*A(2) + K32B*A(4) - A(3)*(K21B + K23F) ; Grade 2
>
> DADT(4) = K23F*A(3) - K32B*A(4)
> ; Grade 3
>
>
>
> $ERROR
>
> Y = 1
>
> IF(DV.EQ.1.AND.CMT.EQ.0) Y = A(1)
>
> IF(DV.EQ.2.AND.CMT.EQ.0) Y = A(2)
>
> IF(DV.EQ.3.AND.CMT.EQ.0) Y = A(3)
>
> IF(DV.EQ.4.AND.CMT.EQ.0) Y = A(4)
>
>
>
> P0 = A(1)
>
> P1 = A(2)
>
> P2 = A(3)
>
> P3 = A(4)
>
>
>
> ; Cumulative probabilities
>
>
>
> CUP0 = P0
>
> CUP1 = P0 + P1
>
> CUP2 = P0 + P1 + P2
>
> CUP3 = P0 + P1 + P2 + P3
>
>
>
> ; Start of simulation block
>
> IF(ICALL.EQ.4) THEN
>
> IF(CMT.EQ.0) THEN
>
> CALL RANDOM (2,R)
>
> IF(R.LE.CUP0) DV = 1
>
> IF(R.GT.CUP0.AND.R.LE.CUP1) DV = 2
>
> IF(R.GT.CUP1.AND.R.LE.CUP2) DV = 3
>
> IF(R.GT.CUP2) DV = 4
>
> ENDIF
>
> ENDIF
>
> ; End of simulation block
>
>
>
> PSDV=DV
>
>
>
> $THETA
>
> …
>
> $OMEGA
>
> 0 FIX
>
>
>
> $COV PRINT=E
>
> ;$SIM (7776) (8877 UNIFORM) ONLYSIM NOPREDICTION
>
> $EST METHOD=1 LAPLACIAN LIKE SIG=2 PRINT=1 MAX99 NOABORT
>
>
>
> [image: cid:image004.png_at_01D3092E.080FB8B0][image: unnamed]
>
> _____________________
>
> *Eduard Schmulenson, M.Sc.*
>
> Apotheker/Pharmacist
>
>
>
> Klinische Pharmazie
>
> Pharmazeutisches Institut
>
> Universität Bonn
>
> An der Immenburg 4
>
> D-53121 Bonn
>
>
>
> Tel.: +49 228 73-5242
>
> e.schmulenson_at_uni-bonn.de
>
>
>
(image/gif attachment: image002.gif)
(image/png attachment: image001.png)
Hi Eduard,
If you want to have a minimalistic model, with respect to both fixed effects and random effects, but still incorporate all 4 categories as well as the Markov element, you can try the minimal Continuous-Time Markov Model (mCTMM) described in "A Minimal Continuous-Time Markov Pharmacometric Model. Schindler E, Karlsson MO. AAPS J. 2017 Sep;19(5):1424-1435"
Best regards,
Mats
Quoted reply history
From: owner-nmusers_at_globomaxnm.com <owner-nmusers_at_globomaxnm.com> On Behalf Of Eduard Schmulenson
Sent: den 5 februari 2019 11:04
To: nmusers_at_globomaxnm.com
Subject: [NMusers] Ordinal data model: Incorporation of variability & assessment of predictive performance
Dear all,
I am currently trying to model the transitions between four adverse event grades (0-3) using a continuous-time Markov modeling approach. I have included a dose effect as well as a time effect on the transition constants. Overall, the parameters are well estimated and the VPC looks also quite good.
However, the model does not have any IIV or other variability incorporated, so no individual predictions can be made. I have tried different approaches to include variability:
- Six different etas: a) One eta per transition constant without a block structure (this has resulted in rounding errors), b) with a full block structure (see Lacroix BD et al. CPT PSP 2014), also with rounding errors and c) with two OMEGA BLOCK(3) structures which solely include "forward" and "backward" transition constants, respectively (also with rounding errors).
- Two different etas: One mutual eta on "forward" and "backward" transition constants (shrinkage values of ~ 40 and 60%, respectively, which do not lower after including the dose effect. The impact of time cannot be estimated anymore)
- Just one eta on every transition constant (shrinkage value of 46% which slightly increases after including the dose and time effect.
The etas were added as exponential variables.
Other tested covariates were not significant or resulted in run errors when a bootstrap was performed.
Are there any other possibilities to incorporate variability in this type of model? Or is it solely a data-dependent issue? You can find the control stream (without any IIV) below.
My second question is about the assessment of predictive performance in the same model. One can compare the observed proportions of an adverse event grade vs. the simulated probability or the observed vs. simulated grade. Is there a meaningful error which I can calculate in order to assess bias and precision? Would be a median prediction error and a median absolute prediction error appropriate for this type of data? And what kind of error would you suggest when one has to calculate a relative error which would include a division by 0?
Thank you very much in advance.
Best regards,
Eduard
##########################################
$ABB COMRES = 1
$SUBROUTINES ADVAN6 TOL = 4
$MODEL
NCOMP = 4
COMP = (G0) ; No AE
COMP = (G1) ; Mild AE
COMP = (G2) ; Moderate AE
COMP = (G3) ; Severe AE
$PK
IF(NEWIND.NE.2) THEN
PSDV = 0
COM(1) = 0
ENDIF
PRSP = PSDV ; Previous DV
IF(PRSP.EQ.1) COM(1) = 0
IF(PRSP.EQ.2) COM(1) = 1
IF(PRSP.EQ.3) COM(1) = 2
IF(PRSP.EQ.4) COM(1) = 3
F1 = 0
F2 = 0
F3 = 0
F4 = 0
IF(COM(1).EQ.0) F1 = 1
IF(COM(1).EQ.1) F2 = 1
IF(COM(1).EQ.2) F3 = 1
IF(COM(1).EQ.3) F4 = 1
TVK01 = THETA(1)
K01 = TVK01*EXP(ETA(1))
TVK12 = THETA(2)
K12 = TVK12
TVK23 = THETA(3)
K23 = TVK23
TVK10 = THETA(4)
K10 = TVK10
TVK21 = THETA(5)
K21 = TVK21
TVK32 = THETA(6)
K32 = TVK32
TVKT = THETA(8)
KT = TVKT
$DES
K01F = K01*EXP(KT*T) ; Time effect
K12F = K12*EXP(KT*T)
K23F = K23*EXP(KT*T)
K10B = K10*EXP(THETA(7)*(DOSEDAY-3000)) ; Dose effect
K21B = K21*EXP(THETA(7)*(DOSEDAY-3000))
K32B = K32*EXP(THETA(7)*(DOSEDAY-3000))
DADT(1) = K10B*A(2) - K01F*A(1) ; Grade 0
DADT(2) = K01F*A(1) + K21B*A(3) - A(2)*(K10B + K12F) ; Grade 1
DADT(3) = K12F*A(2) + K32B*A(4) - A(3)*(K21B + K23F) ; Grade 2
DADT(4) = K23F*A(3) - K32B*A(4) ; Grade 3
$ERROR
Y = 1
IF(DV.EQ.1.AND.CMT.EQ.0) Y = A(1)
IF(DV.EQ.2.AND.CMT.EQ.0) Y = A(2)
IF(DV.EQ.3.AND.CMT.EQ.0) Y = A(3)
IF(DV.EQ.4.AND.CMT.EQ.0) Y = A(4)
P0 = A(1)
P1 = A(2)
P2 = A(3)
P3 = A(4)
; Cumulative probabilities
CUP0 = P0
CUP1 = P0 + P1
CUP2 = P0 + P1 + P2
CUP3 = P0 + P1 + P2 + P3
; Start of simulation block
IF(ICALL.EQ.4) THEN
IF(CMT.EQ.0) THEN
CALL RANDOM (2,R)
IF(R.LE.CUP0) DV = 1
IF(R.GT.CUP0.AND.R.LE.CUP1) DV = 2
IF(R.GT.CUP1.AND.R.LE.CUP2) DV = 3
IF(R.GT.CUP2) DV = 4
ENDIF
ENDIF
; End of simulation block
PSDV=DV
$THETA
...
$OMEGA
0 FIX
$COV PRINT=E
;$SIM (7776) (8877 UNIFORM) ONLYSIM NOPREDICTION
$EST METHOD=1 LAPLACIAN LIKE SIG=2 PRINT=1 MAX99 NOABORT
[cid:image004.png_at_01D3092E.080FB8B0][unnamed]
_____________________
Eduard Schmulenson, M.Sc.
Apotheker/Pharmacist
Klinische Pharmazie
Pharmazeutisches Institut
Universitt Bonn
An der Immenburg 4
D-53121 Bonn
Tel.: +49 228 73-5242
e.schmulenson_at_uni-bonn.de<mailto:e.schmulenson_at_uni-bonn.de>
Nr du har kontakt med oss p Uppsala universitet med e-post s innebr det att vi behandlar dina personuppgifter. Fr att lsa mer om hur vi gr det kan du lsa hr: http://www.uu.se/om-uu/dataskydd-personuppgifter/
E-mailing Uppsala University means that we will process your personal data. For more information on how this is performed, please read here: http://www.uu.se/en/about-uu/data-protection-policy
(image/png attachment: image001.png)
(image/gif attachment: image002.gif)