Hi all,
I have a question related to the objective function value when multiple
endpoints are modelled jointly. Specifically I would like to know if a
change in in OFV between models is driven primarily by one of the endpoints
or if both contributes to the change, or maybe they are even driving the
OFV in oposite directions.
Is there a way to get some form of partial OFV by endpoint?
Best regards,
Matts
OFV by endpoint of joint models?
Hi Matts,
You can use the table item OBJI and sum over each ID to get any subset you
want.
Matt
Quoted reply history
On Mon, Oct 10, 2022, 9:12 AM Matts Kågedal <[email protected]> wrote:
> Hi all,
> I have a question related to the objective function value when multiple
> endpoints are modelled jointly. Specifically I would like to know if a
> change in in OFV between models is driven primarily by one of the endpoints
> or if both contributes to the change, or maybe they are even driving the
> OFV in oposite directions.
>
> Is there a way to get some form of partial OFV by endpoint?
> Best regards,
> Matts
>
Hi Matts,
The easiest way to assess is when one of two endpoints is modeled directly (TTE, logistic regression) as often is the case, than look at the Y value for those endpoints, as reported in the PRED variable. The sum of those values is the ofv, or proportional to it, for that particular endpoint - the other endpoint is than affected in the inverse way.
If you have multiple continuous endpoints it becomes more complicated. You could either look at the sum of absolute CWRES to get an idea, but not exact in terms of ofv comparison. Another approximate comparison would be to run the model without evaluation (e.g. MAXEVAL=0) with the original msfofile as $MSFI for the separate endpoints (by e.g. IGN(DVID.NE.x) where x is your endpoint). It is not exact, again, as it ignores the correlation between endpoints but should get you in the neighborhood. As an improvement to this method you could force evaluation at the original posthocs by reading them in in your datafile - this would still ignore correlation but the effect would be largely diminished because the posthocs are fixed to those estimated with correlation.
Hope this helps,
Jeroen
http://pd-value.com
[email protected]
@PD_value
+31 6 23118438
-- More value out of your data!
Quoted reply history
On 10-10-2022 16:03, Matts Kågedal wrote:
> Hi all,
>
> I have a question related to the objective function value when multiple endpoints are modelled jointly. Specifically I would like to know if a change in in OFV between models is driven primarily by one of the endpoints or if both contributes to the change, or maybe they are even driving the OFV in oposite directions.
>
> Is there a way to get some form of partial OFV by endpoint?
> Best regards,
> Matts
Hi Matt,
That would work in case you had one subset of subjects with observations of one
endpoint, and another subset of subjects with observations of a different
endpoint - e.g. one study with only PK samples and another study with only PD
samples.
But in such a setting there would not be much value in joint modelling
(simultaneous estimation of the two models), since no subject has observations
of both endpoints.
Or maybe I missed your point?
Best regards
Jakob
Jakob Ribbing, Ph.D.
Senior Consultant, Pharmetheus AB
Cell/Mobile: +46 (0)70 514 33 77
[email protected]
www.pharmetheus.com
Phone, Office: +46 (0)18 513 328
Uppsala Science Park, Dag Hammarskjölds väg 36B
SE-752 37 Uppsala, Sweden
Quoted reply history
> On 10 Oct 2022, at 17:02, Matthew Fidler <[email protected]> wrote:
>
> Hi Matts,
>
> You can use the table item OBJI and sum over each ID to get any subset you
> want.
>
> Matt
>
>
>
> On Mon, Oct 10, 2022, 9:12 AM Matts Kågedal <[email protected]
> <mailto:[email protected]>> wrote:
> Hi all,
> I have a question related to the objective function value when multiple
> endpoints are modelled jointly. Specifically I would like to know if a change
> in in OFV between models is driven primarily by one of the endpoints or if
> both contributes to the change, or maybe they are even driving the OFV in
> oposite directions.
>
> Is there a way to get some form of partial OFV by endpoint?
> Best regards,
> Matts
--
*This communication is confidential and is only intended for the use of the
individual or entity to which it is directed. It may contain information
that is privileged and exempt from disclosure under applicable law. If you
are not the intended recipient please notify us immediately. Please do not
copy it or disclose its contents to any other person.*
*Any personal data
will be processed in accordance with Pharmetheus' privacy notice, available
here https://pharmetheus.com/privacy-policy/.**
*
Hi Jeroen
I note your thought about CWRES and OFV. In some exploratory work, we did not
find that the rank order of abs(CIWRES) or CIWRES^2 and PHI() was preserved
(with FOCEI) for continuous data. I had anticipated some rank similarity.
Cheers
Steve
________________________________________
Stephen Duffull | Professor
Otago Pharmacometrics Group
School of Pharmacy | He Rau Kawakawa
University of Otago | Te Whare Wānanga o Otāgo
Dunedin | Ōtepoti
Aotearoa New Zealand
Ph: 64 3 479 5099
Quoted reply history
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of
Jeroen Elassaiss-Schaap (PD-value B.V.)
Sent: Tuesday, 11 October 2022 4:07 am
To: Matts Kågedal <[email protected]>; [email protected]
Subject: Re: [NMusers] OFV by endpoint of joint models?
Hi Matts,
The easiest way to assess is when one of two endpoints is modeled directly
(TTE, logistic regression) as often is the case, than look at the Y value for
those endpoints, as reported in the PRED variable. The sum of those values is
the ofv, or proportional to it, for that particular endpoint - the other
endpoint is than affected in the inverse way.
If you have multiple continuous endpoints it becomes more complicated.
You could either look at the sum of absolute CWRES to get an idea, but not
exact in terms of ofv comparison. Another approximate comparison would be to
run the model without evaluation (e.g. MAXEVAL=0) with the original msfofile as
$MSFI for the separate endpoints (by e.g.
IGN(DVID.NE.x) where x is your endpoint). It is not exact, again, as it
ignores the correlation between endpoints but should get you in the
neighborhood. As an improvement to this method you could force evaluation at
the original posthocs by reading them in in your datafile
- this would still ignore correlation but the effect would be largely
diminished because the posthocs are fixed to those estimated with correlation.
Hope this helps,
Jeroen
https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1ebbd4a5cbce449913f108daaad22217%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010116756566698%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PEGSjbNvoRzu%2B6AbaXCXTHlNGkGc28Eb7QgKu1tE9sM%3D&reserved=0
[email protected]
@PD_value
+31 6 23118438
-- More value out of your data!
On 10-10-2022 16:03, Matts Kågedal wrote:
> Hi all,
> I have a question related to the objective function value when
> multiple endpoints are modelled jointly. Specifically I would like to
> know if a change in in OFV between models is driven primarily by one
> of the endpoints or if both contributes to the change, or maybe they
> are even driving the OFV in oposite directions.
>
> Is there a way to get some form of partial OFV by endpoint?
> Best regards,
> Matts
Hi Steven,
Thanks for sharing! CWRES is “polluted” by the ETA gradients more directly
compared to OFV. One would however hope for rank order consistency. Did you
also test this without interaction? Might also be interesting to test the other
residuals that nonmem offers in that respect.
Cheers
Jeroen
http://pd-value.com
[email protected]
@PD_value
+31 6 23118438
-- More value out of your data!
Quoted reply history
> Op 10 okt. 2022 om 18:51 heeft Stephen Duffull <[email protected]>
> het volgende geschreven:
>
> Hi Jeroen
>
> I note your thought about CWRES and OFV. In some exploratory work, we did
> not find that the rank order of abs(CIWRES) or CIWRES^2 and PHI() was
> preserved (with FOCEI) for continuous data. I had anticipated some rank
> similarity.
>
> Cheers
>
> Steve
> ________________________________________
> Stephen Duffull | Professor
> Otago Pharmacometrics Group
> School of Pharmacy | He Rau Kawakawa
> University of Otago | Te Whare Wānanga o Otāgo
> Dunedin | Ōtepoti
> Aotearoa New Zealand
> Ph: 64 3 479 5099
>
>
>
>
>
> -----Original Message-----
> From: [email protected] <[email protected]> On Behalf
> Of Jeroen Elassaiss-Schaap (PD-value B.V.)
> Sent: Tuesday, 11 October 2022 4:07 am
> To: Matts Kågedal <[email protected]>; [email protected]
> Subject: Re: [NMusers] OFV by endpoint of joint models?
>
> Hi Matts,
>
> The easiest way to assess is when one of two endpoints is modeled directly
> (TTE, logistic regression) as often is the case, than look at the Y value for
> those endpoints, as reported in the PRED variable. The sum of those values is
> the ofv, or proportional to it, for that particular endpoint - the other
> endpoint is than affected in the inverse way.
>
> If you have multiple continuous endpoints it becomes more complicated.
> You could either look at the sum of absolute CWRES to get an idea, but not
> exact in terms of ofv comparison. Another approximate comparison would be to
> run the model without evaluation (e.g. MAXEVAL=0) with the original msfofile
> as $MSFI for the separate endpoints (by e.g.
> IGN(DVID.NE.x) where x is your endpoint). It is not exact, again, as it
> ignores the correlation between endpoints but should get you in the
> neighborhood. As an improvement to this method you could force evaluation at
> the original posthocs by reading them in in your datafile
> - this would still ignore correlation but the effect would be largely
> diminished because the posthocs are fixed to those estimated with correlation.
>
> Hope this helps,
>
> Jeroen
>
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1ebbd4a5cbce449913f108daaad22217%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010116756566698%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PEGSjbNvoRzu%2B6AbaXCXTHlNGkGc28Eb7QgKu1tE9sM%3D&reserved=0
> [email protected]
> @PD_value
> +31 6 23118438
> -- More value out of your data!
>
>> On 10-10-2022 16:03, Matts Kågedal wrote:
>> Hi all,
>> I have a question related to the objective function value when
>> multiple endpoints are modelled jointly. Specifically I would like to
>> know if a change in in OFV between models is driven primarily by one
>> of the endpoints or if both contributes to the change, or maybe they
>> are even driving the OFV in oposite directions.
>>
>> Is there a way to get some form of partial OFV by endpoint?
>> Best regards,
>> Matts
>
>
Thanks Mats,
Sounds great and like a lot of work, let me know when you have implemented
the ability to do this in simeval :)
Best,
Matts
Quoted reply history
On Mon, Oct 10, 2022 at 10:45 PM Mats Karlsson <[email protected]>
wrote:
> Hi Matts,
>
>
>
> One opportunity to learn about the expected fit of a model to data in
> relation to the actual fit is to use the PsN functionality “simeval”. In
> this functionality multiple data sets are simulated from the final model
> and the realized design. The OFV per subject (and the overall OFV) can be
> assessed after evaluation (i.e., MAXEVAL=0) or estimation of each of the
> simulated data sets. This will provide reference OFV distributions with
> which the real data OFV (subject or total study population) can be compared
> in a PPC like manner. This is developed from Largajolli et al. (
> https://www.page-meeting.org/default.asp?abstract=3208).
>
>
>
> In your case, you are interested in learning about the relative
> contribution of the different variables of a joint model. While not having
> tried it, I imagine that you can, based on your final joint model(s),
> obtain the expected OFV distribution one variable at a time as well as them
> jointly. From this it ought to be possible to learn some about the quality
> of the model with respect to variable A, variable B and their joint
> distribution in describing the real data.
>
>
>
> Best regards,
>
> Mats
>
> *From:* [email protected] <[email protected]> *On
> Behalf Of *Stephen Duffull
> *Sent:* den 10 oktober 2022 21:56
> *To:* Jeroen Elassaiss-Schaap (PD-value) <[email protected]>
> *Cc:* Matts Kågedal <[email protected]>; [email protected]
> *Subject:* RE: [NMusers] OFV by endpoint of joint models?
>
>
>
> HI Jeroen
>
>
>
> I tested this with additive error (i.e. interaction has no influence) and
> combined. Rank order was not preserved.
>
>
>
> To be clearer, this was a PK only example and I compared sum(CIWRES^2) for
> each individual vs PHI(). I was trying to see if I could get the PHI() per
> analyte for a multiple response model and thought that a quick way of doing
> this was to grab the relative contribution from CIWRES.
>
>
>
> Cheers
>
>
>
> Steve
>
>
>
> *From:* Jeroen Elassaiss-Schaap (PD-value) <[email protected]>
> *Sent:* Tuesday, 11 October 2022 8:36 am
> *To:* Stephen Duffull <[email protected]>
> *Cc:* Matts Kågedal <[email protected]>; [email protected]
> *Subject:* Re: [NMusers] OFV by endpoint of joint models?
>
>
>
> Hi Steven,
>
>
>
> Thanks for sharing! CWRES is “polluted” by the ETA gradients more directly
> compared to OFV. One would however hope for rank order consistency. Did you
> also test this without interaction? Might also be interesting to test the
> other residuals that nonmem offers in that respect.
>
>
>
> Cheers
>
> Jeroen
>
> http://pd-value.com
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1017274f1209442cc1b408daaaf6aab3%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010273680373817%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VG6Wo1oVz8Ti2jcYV0hY%2BdqZTETwRUgxbARS4eRhoY4%3D&reserved=0
> [email protected]
> @PD_value
> +31 6 23118438 <+31%206%2023118438>
> -- More value out of your data!
>
>
>
> Op 10 okt. 2022 om 18:51 heeft Stephen Duffull <
> [email protected]> het volgende geschreven:
>
> Hi Jeroen
>
> I note your thought about CWRES and OFV. In some exploratory work, we did
> not find that the rank order of abs(CIWRES) or CIWRES^2 and PHI() was
> preserved (with FOCEI) for continuous data. I had anticipated some rank
> similarity.
>
> Cheers
>
> Steve
> ________________________________________
> Stephen Duffull | Professor
> Otago Pharmacometrics Group
> School of Pharmacy | He Rau Kawakawa
> University of Otago | Te Whare Wānanga o Otāgo
> Dunedin | Ōtepoti
> Aotearoa New Zealand
> Ph: 64 3 479 5099
>
>
>
>
>
> -----Original Message-----
> From: [email protected] <[email protected]> On
> Behalf Of Jeroen Elassaiss-Schaap (PD-value B.V.)
> Sent: Tuesday, 11 October 2022 4:07 am
> To: Matts Kågedal <[email protected]>; [email protected]
> Subject: Re: [NMusers] OFV by endpoint of joint models?
>
> Hi Matts,
>
> The easiest way to assess is when one of two endpoints is modeled directly
> (TTE, logistic regression) as often is the case, than look at the Y value
> for those endpoints, as reported in the PRED variable. The sum of those
> values is the ofv, or proportional to it, for that particular endpoint -
> the other endpoint is than affected in the inverse way.
>
> If you have multiple continuous endpoints it becomes more complicated.
> You could either look at the sum of absolute CWRES to get an idea, but not
> exact in terms of ofv comparison. Another approximate comparison would be
> to run the model without evaluation (e.g. MAXEVAL=0) with the original
> msfofile as $MSFI for the separate endpoints (by e.g.
> IGN(DVID.NE.x) where x is your endpoint). It is not exact, again, as it
> ignores the correlation between endpoints but should get you in the
> neighborhood. As an improvement to this method you could force evaluation
> at the original posthocs by reading them in in your datafile
> - this would still ignore correlation but the effect would be largely
> diminished because the posthocs are fixed to those estimated with
> correlation.
>
> Hope this helps,
>
> Jeroen
>
>
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1ebbd4a5cbce449913f108daaad22217%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010116756566698%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PEGSjbNvoRzu%2B6AbaXCXTHlNGkGc28Eb7QgKu1tE9sM%3D&reserved=0
> [email protected]
> @PD_value
> +31 6 23118438
> -- More value out of your data!
>
> On 10-10-2022 16:03, Matts Kågedal wrote:
>
> Hi all,
>
> I have a question related to the objective function value when
>
> multiple endpoints are modelled jointly. Specifically I would like to
>
> know if a change in in OFV between models is driven primarily by one
>
> of the endpoints or if both contributes to the change, or maybe they
>
> are even driving the OFV in oposite directions.
>
>
>
> Is there a way to get some form of partial OFV by endpoint?
>
> Best regards,
>
> Matts
>
>
>
>
>
>
>
>
>
>
>
> När du har kontakt med oss på Uppsala universitet med e-post så innebär
> det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör
> det kan du läsa här: http://www.uu.se/om-uu/dataskydd-personuppgifter/
>
> E-mailing Uppsala University means that we will process your personal
> data. For more information on how this is performed, please read here:
> http://www.uu.se/en/about-uu/data-protection-policy
>
Dear Matts Kådagel, Jacob Ribbing and Mats Karlsson,
Since you all have touched on joint / simultaneous / sequential modelling,
I'd like to remind you of some classical papers that originally addressed
this type of question -- please find the two links at the very end of this
email.
Admittedly, apart from becoming lost in a series of names such as Matts,
Matt and Mats, I am also at a loss as to the lack of high-level
discussion that could have been implied in the original question.
In this era of data and ever more data, a clinical setting is evolving and
we are likely to have multiple endpoints such as biomarkers and clinical
endpoints altogether and hence joint modelling is necessary. I was
expecting some discussion on how we can reuse the classical examples in
order to update the methodology for clinical settings as of today. In other
words, if we don't know where we've come from, we may not know where we are
going.
I'm aware that I'm making a different voice here, in the hope of provoking
advancement no matter how little for the future.
Best regards,
Shan
https://link.springer.com/article/10.1023/B:JOPA.0000012998.04442.1f
https://pubmed.ncbi.nlm.nih.gov/15000422/
Quoted reply history
On Tue, Oct 11, 2022 at 10:58 AM Matts Kågedal <[email protected]>
wrote:
> Thanks Mats,
> Sounds great and like a lot of work, let me know when you have implemented
> the ability to do this in simeval :)
>
> Best,
> Matts
>
> On Mon, Oct 10, 2022 at 10:45 PM Mats Karlsson <
> [email protected]> wrote:
>
>> Hi Matts,
>>
>>
>>
>> One opportunity to learn about the expected fit of a model to data in
>> relation to the actual fit is to use the PsN functionality “simeval”. In
>> this functionality multiple data sets are simulated from the final model
>> and the realized design. The OFV per subject (and the overall OFV) can be
>> assessed after evaluation (i.e., MAXEVAL=0) or estimation of each of the
>> simulated data sets. This will provide reference OFV distributions with
>> which the real data OFV (subject or total study population) can be compared
>> in a PPC like manner. This is developed from Largajolli et al. (
>> https://www.page-meeting.org/default.asp?abstract=3208).
>>
>>
>>
>> In your case, you are interested in learning about the relative
>> contribution of the different variables of a joint model. While not having
>> tried it, I imagine that you can, based on your final joint model(s),
>> obtain the expected OFV distribution one variable at a time as well as them
>> jointly. From this it ought to be possible to learn some about the quality
>> of the model with respect to variable A, variable B and their joint
>> distribution in describing the real data.
>>
>>
>>
>> Best regards,
>>
>> Mats
>>
>> *From:* [email protected] <[email protected]> *On
>> Behalf Of *Stephen Duffull
>> *Sent:* den 10 oktober 2022 21:56
>> *To:* Jeroen Elassaiss-Schaap (PD-value) <[email protected]>
>> *Cc:* Matts Kågedal <[email protected]>; [email protected]
>> *Subject:* RE: [NMusers] OFV by endpoint of joint models?
>>
>>
>>
>> HI Jeroen
>>
>>
>>
>> I tested this with additive error (i.e. interaction has no influence) and
>> combined. Rank order was not preserved.
>>
>>
>>
>> To be clearer, this was a PK only example and I compared sum(CIWRES^2)
>> for each individual vs PHI(). I was trying to see if I could get the PHI()
>> per analyte for a multiple response model and thought that a quick way of
>> doing this was to grab the relative contribution from CIWRES.
>>
>>
>>
>> Cheers
>>
>>
>>
>> Steve
>>
>>
>>
>> *From:* Jeroen Elassaiss-Schaap (PD-value) <[email protected]>
>> *Sent:* Tuesday, 11 October 2022 8:36 am
>> *To:* Stephen Duffull <[email protected]>
>> *Cc:* Matts Kågedal <[email protected]>; [email protected]
>> *Subject:* Re: [NMusers] OFV by endpoint of joint models?
>>
>>
>>
>> Hi Steven,
>>
>>
>>
>> Thanks for sharing! CWRES is “polluted” by the ETA gradients more
>> directly compared to OFV. One would however hope for rank order
>> consistency. Did you also test this without interaction? Might also be
>> interesting to test the other residuals that nonmem offers in that respect.
>>
>>
>>
>> Cheers
>>
>> Jeroen
>>
>> http://pd-value.com
>> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1017274f1209442cc1b408daaaf6aab3%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010273680373817%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VG6Wo1oVz8Ti2jcYV0hY%2BdqZTETwRUgxbARS4eRhoY4%3D&reserved=0
>> [email protected]
>> @PD_value
>> +31 6 23118438 <+31%206%2023118438>
>> -- More value out of your data!
>>
>>
>>
>> Op 10 okt. 2022 om 18:51 heeft Stephen Duffull <
>> [email protected]> het volgende geschreven:
>>
>> Hi Jeroen
>>
>> I note your thought about CWRES and OFV. In some exploratory work, we
>> did not find that the rank order of abs(CIWRES) or CIWRES^2 and PHI() was
>> preserved (with FOCEI) for continuous data. I had anticipated some rank
>> similarity.
>>
>> Cheers
>>
>> Steve
>> ________________________________________
>> Stephen Duffull | Professor
>> Otago Pharmacometrics Group
>> School of Pharmacy | He Rau Kawakawa
>> University of Otago | Te Whare Wānanga o Otāgo
>> Dunedin | Ōtepoti
>> Aotearoa New Zealand
>> Ph: 64 3 479 5099
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: [email protected] <[email protected]> On
>> Behalf Of Jeroen Elassaiss-Schaap (PD-value B.V.)
>> Sent: Tuesday, 11 October 2022 4:07 am
>> To: Matts Kågedal <[email protected]>; [email protected]
>> Subject: Re: [NMusers] OFV by endpoint of joint models?
>>
>> Hi Matts,
>>
>> The easiest way to assess is when one of two endpoints is modeled
>> directly (TTE, logistic regression) as often is the case, than look at the
>> Y value for those endpoints, as reported in the PRED variable. The sum of
>> those values is the ofv, or proportional to it, for that particular
>> endpoint - the other endpoint is than affected in the inverse way.
>>
>> If you have multiple continuous endpoints it becomes more complicated.
>> You could either look at the sum of absolute CWRES to get an idea, but
>> not exact in terms of ofv comparison. Another approximate comparison would
>> be to run the model without evaluation (e.g. MAXEVAL=0) with the original
>> msfofile as $MSFI for the separate endpoints (by e.g.
>> IGN(DVID.NE.x) where x is your endpoint). It is not exact, again, as it
>> ignores the correlation between endpoints but should get you in the
>> neighborhood. As an improvement to this method you could force evaluation
>> at the original posthocs by reading them in in your datafile
>> - this would still ignore correlation but the effect would be largely
>> diminished because the posthocs are fixed to those estimated with
>> correlation.
>>
>> Hope this helps,
>>
>> Jeroen
>>
>>
>> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1ebbd4a5cbce449913f108daaad22217%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010116756566698%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PEGSjbNvoRzu%2B6AbaXCXTHlNGkGc28Eb7QgKu1tE9sM%3D&reserved=0
>> [email protected]
>> @PD_value
>> +31 6 23118438
>> -- More value out of your data!
>>
>> On 10-10-2022 16:03, Matts Kågedal wrote:
>>
>> Hi all,
>>
>> I have a question related to the objective function value when
>>
>> multiple endpoints are modelled jointly. Specifically I would like to
>>
>> know if a change in in OFV between models is driven primarily by one
>>
>> of the endpoints or if both contributes to the change, or maybe they
>>
>> are even driving the OFV in oposite directions.
>>
>>
>>
>> Is there a way to get some form of partial OFV by endpoint?
>>
>> Best regards,
>>
>> Matts
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> När du har kontakt med oss på Uppsala universitet med e-post så innebär
>> det att vi behandlar dina personuppgifter. För att läsa mer om hur vi gör
>> det kan du läsa här: http://www.uu.se/om-uu/dataskydd-personuppgifter/
>>
>> E-mailing Uppsala University means that we will process your personal
>> data. For more information on how this is performed, please read here:
>> http://www.uu.se/en/about-uu/data-protection-policy
>>
>
Dear All,
Is it possible to fix all of the THETAs, OMEGAs and SIGMAs to their estimated values, and then re-run the model with all of one type of data in, /but the other type of data retained for just a single individual/? This would give an objective function contribution per individual, for each type of data I think, which could be repeated across all individuals (and data types). Essentially, you would leave all of one type of data in the dataset, but have the other type of data only for the individual in question and then we wouldn't have the within-individual correlation problem highlighted by Jeroen. And we would end up with a set of individual contributions for each data type, which account for the "driving" effect of the other data type.
The reason I think this is the best we can do is that all of the information across individuals is in the global parameters, and it's the within individual correlation across the two endpoints that is really tricky to account for. And with one type of data retained for all subjects, we can then see a difference for the other type as we iterate across subjects, while retaining the within subject information provided by the first data type. Fundamentally, this still isn't quite separate as one type of data will alter the objective function contribution (i.e. likelihood) of the other i.e. adding PD for one individual can alter the objective function contribution from that individual's PK (and move the posthocs) - but that is because at a fundamental level, you cannot separate the objective function contributions of two observations that depend on each other!
Note that when these individual contributions are summed, they certainly won't add up to the original objective function, partly as there will be some padding dropped from the NONMEM objective function. However the padding should solely be a function of the number of data points (not their actual values), so the difference between the individual totals summed and the all data objective function could be divided by the number of observations, and a compensation applied to each individual objective function value based on the percentage of all observations in that individual.
I think this is as close as we can get to a contribution per data type (with the driving effect of the other data type retained). Now it is possible that this approach will turn out to be inferior to Jeroen's suggestion of running the model on one type of data with the other models post-hocs, which would be much quicker. By breaking it down among individuals, when you compare across models, you will sometimes find that some individuals benefit a great deal from a model change, and dominate the objective function.
Kind regards, James
PS It is also possible to access these objective function contributions by calculating individual contributions one at a time for each observation type and then both observation types (because all of the global population information is already in the THETAs, OMEGAs and SIGMAs), which uses much less computational power, however I couldn't write that down in a way that was easy to follow.
https://www.popypkpd.com
Quoted reply history
On 11/10/2022 10:44, Matts Kågedal wrote:
> Thanks Mats,
>
> Sounds great and like a lot of work, let me know when you have implemented the ability to do this in simeval :)
>
> Best,
> Matts
>
> On Mon, Oct 10, 2022 at 10:45 PM Mats Karlsson < [email protected] > wrote:
>
> Hi Matts,
>
> One opportunity to learn about the expected fit of a model to data
> in relation to the actual fit is to use the PsN functionality
> “simeval”. In this functionality multiple data sets are simulated
> from the final model and the realized design. The OFV per subject
> (and the overall OFV) can be assessed after evaluation (i.e.,
> MAXEVAL=0) or estimation of each of the simulated data sets. This
> will provide reference OFV distributions with which the real data
> OFV (subject or total study population) can be compared in a PPC
> like manner. This is developed from Largajolli et al.
> ( https://www.page-meeting.org/default.asp?abstract=3208).
>
> In your case, you are interested in learning about the relative
> contribution of the different variables of a joint model. While
> not having tried it, I imagine that you can, based on your final
> joint model(s), obtain the expected OFV distribution one variable
> at a time as well as them jointly. From this it ought to be
> possible to learn some about the quality of the model with respect
> to variable A, variable B and their joint distribution in
> describing the real data.
>
> Best regards,
>
> Mats
>
> *From:*[email protected] <[email protected]>
> *On Behalf Of *Stephen Duffull
> *Sent:* den 10 oktober 2022 21:56
> *To:* Jeroen Elassaiss-Schaap (PD-value) <[email protected]>
> *Cc:* Matts Kågedal <[email protected]>; [email protected]
> *Subject:* RE: [NMusers] OFV by endpoint of joint models?
>
> HI Jeroen
>
> I tested this with additive error (i.e. interaction has no
> influence) and combined. Rank order was not preserved.
>
> To be clearer, this was a PK only example and I compared
> sum(CIWRES^2) for each individual vs PHI(). I was trying to see
> if I could get the PHI() per analyte for a multiple response model
> and thought that a quick way of doing this was to grab the
> relative contribution from CIWRES.
>
> Cheers
>
> Steve
>
> *From:*Jeroen Elassaiss-Schaap (PD-value) <[email protected]
> <mailto:[email protected]>>
> *Sent:* Tuesday, 11 October 2022 8:36 am
> *To:* Stephen Duffull <[email protected]
> <mailto:[email protected]>>
> *Cc:* Matts Kågedal <[email protected]
> <mailto:[email protected]>>; [email protected]
> <mailto:[email protected]>
> *Subject:* Re: [NMusers] OFV by endpoint of joint models?
>
> Hi Steven,
>
> Thanks for sharing! CWRES is “polluted” by the ETA gradients more
> directly compared to OFV. One would however hope for rank order
> consistency. Did you also test this without interaction? Might
> also be interesting to test the other residuals that nonmem offers
> in that respect.
>
> Cheers
>
> Jeroen
>
> http://pd-value.com
>
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1017274f1209442cc1b408daaaf6aab3%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010273680373817%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VG6Wo1oVz8Ti2jcYV0hY%2BdqZTETwRUgxbARS4eRhoY4%3D&reserved=0
> [email protected] <mailto:[email protected]>
> @PD_value
> +31 6 23118438 <tel:+31%206%2023118438>
> -- More value out of your data!
>
> Op 10 okt. 2022 om 18:51 heeft Stephen Duffull
> <[email protected]> het volgende geschreven:
>
> Hi Jeroen
>
> I note your thought about CWRES and OFV. In some exploratory
> work, we did not find that the rank order of abs(CIWRES) or
> CIWRES^2 and PHI() was preserved (with FOCEI) for continuous
> data. I had anticipated some rank similarity.
>
> Cheers
>
> Steve
> ________________________________________
> Stephen Duffull | Professor
> Otago Pharmacometrics Group
> School of Pharmacy | He Rau Kawakawa
> University of Otago | Te Whare Wānanga o Otāgo
> Dunedin | Ōtepoti
> Aotearoa New Zealand
> Ph: 64 3 479 5099
>
> -----Original Message-----
> From: [email protected]
> <mailto:[email protected]><[email protected]
> <mailto:[email protected]>> On Behalf Of Jeroen
> Elassaiss-Schaap (PD-value B.V.)
> Sent: Tuesday, 11 October 2022 4:07 am
> To: Matts Kågedal <[email protected]
> <mailto:[email protected]>>; [email protected]
> <mailto:[email protected]>
> Subject: Re: [NMusers] OFV by endpoint of joint models?
>
> Hi Matts,
>
> The easiest way to assess is when one of two endpoints is
> modeled directly (TTE, logistic regression) as often is the
> case, than look at the Y value for those endpoints, as
> reported in the PRED variable. The sum of those values is the
> ofv, or proportional to it, for that particular endpoint - the
> other endpoint is than affected in the inverse way.
>
> If you have multiple continuous endpoints it becomes more
> complicated.
> You could either look at the sum of absolute CWRES to get an
> idea, but not exact in terms of ofv comparison. Another
> approximate comparison would be to run the model without
> evaluation (e.g. MAXEVAL=0) with the original msfofile as
> $MSFI for the separate endpoints (by e.g.
> IGN(DVID.NE.x) where x is your endpoint). It is not exact,
> again, as it ignores the correlation between endpoints but
> should get you in the neighborhood. As an improvement to this
> method you could force evaluation at the original posthocs by
> reading them in in your datafile
> - this would still ignore correlation but the effect would be
> largely diminished because the posthocs are fixed to those
> estimated with correlation.
>
> Hope this helps,
>
> Jeroen
>
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1ebbd4a5cbce449913f108daaad22217%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010116756566698%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PEGSjbNvoRzu%2B6AbaXCXTHlNGkGc28Eb7QgKu1tE9sM%3D&reserved=0
>
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1ebbd4a5cbce449913f108daaad22217%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010116756566698%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PEGSjbNvoRzu%2B6AbaXCXTHlNGkGc28Eb7QgKu1tE9sM%3D&reserved=0
> [email protected] <mailto:[email protected]>
> @PD_value
> +31 6 23118438
> -- More value out of your data!
>
> On 10-10-2022 16:03, Matts Kågedal wrote:
>
> Hi all,
>
> I have a question related to the objective function value
> when
>
> multiple endpoints are modelled jointly. Specifically I
> would like to
>
> know if a change in in OFV between models is driven
> primarily by one
>
> of the endpoints or if both contributes to the change, or
> maybe they
>
> are even driving the OFV in oposite directions.
>
> Is there a way to get some form of partial OFV by endpoint?
>
> Best regards,
>
> Matts
>
> När du har kontakt med oss på Uppsala universitet med e-post så
> innebär det att vi behandlar dina personuppgifter. För att läsa
> mer om hur vi gör det kan du läsa här:
> http://www.uu.se/om-uu/dataskydd-personuppgifter/
>
> E-mailing Uppsala University means that we will process your
> personal data. For more information on how this is performed,
> please read here: http://www.uu.se/en/about-uu/data-protection-policy
--
James G Wright PhD,
Scientist, Wright Dose Ltd
Tel: UK (0)772 5636914