Re: OFV by endpoint of joint models?
Dear All,
Is it possible to fix all of the THETAs, OMEGAs and SIGMAs to their estimated values, and then re-run the model with all of one type of data in, /but the other type of data retained for just a single individual/? This would give an objective function contribution per individual, for each type of data I think, which could be repeated across all individuals (and data types). Essentially, you would leave all of one type of data in the dataset, but have the other type of data only for the individual in question and then we wouldn't have the within-individual correlation problem highlighted by Jeroen. And we would end up with a set of individual contributions for each data type, which account for the "driving" effect of the other data type.
The reason I think this is the best we can do is that all of the information across individuals is in the global parameters, and it's the within individual correlation across the two endpoints that is really tricky to account for. And with one type of data retained for all subjects, we can then see a difference for the other type as we iterate across subjects, while retaining the within subject information provided by the first data type. Fundamentally, this still isn't quite separate as one type of data will alter the objective function contribution (i.e. likelihood) of the other i.e. adding PD for one individual can alter the objective function contribution from that individual's PK (and move the posthocs) - but that is because at a fundamental level, you cannot separate the objective function contributions of two observations that depend on each other!
Note that when these individual contributions are summed, they certainly won't add up to the original objective function, partly as there will be some padding dropped from the NONMEM objective function. However the padding should solely be a function of the number of data points (not their actual values), so the difference between the individual totals summed and the all data objective function could be divided by the number of observations, and a compensation applied to each individual objective function value based on the percentage of all observations in that individual.
I think this is as close as we can get to a contribution per data type (with the driving effect of the other data type retained). Now it is possible that this approach will turn out to be inferior to Jeroen's suggestion of running the model on one type of data with the other models post-hocs, which would be much quicker. By breaking it down among individuals, when you compare across models, you will sometimes find that some individuals benefit a great deal from a model change, and dominate the objective function.
Kind regards, James
PS It is also possible to access these objective function contributions by calculating individual contributions one at a time for each observation type and then both observation types (because all of the global population information is already in the THETAs, OMEGAs and SIGMAs), which uses much less computational power, however I couldn't write that down in a way that was easy to follow.
https://www.popypkpd.com
Quoted reply history
On 11/10/2022 10:44, Matts Kågedal wrote:
> Thanks Mats,
>
> Sounds great and like a lot of work, let me know when you have implemented the ability to do this in simeval :)
>
> Best,
> Matts
>
> On Mon, Oct 10, 2022 at 10:45 PM Mats Karlsson < [email protected] > wrote:
>
> Hi Matts,
>
> One opportunity to learn about the expected fit of a model to data
> in relation to the actual fit is to use the PsN functionality
> “simeval”. In this functionality multiple data sets are simulated
> from the final model and the realized design. The OFV per subject
> (and the overall OFV) can be assessed after evaluation (i.e.,
> MAXEVAL=0) or estimation of each of the simulated data sets. This
> will provide reference OFV distributions with which the real data
> OFV (subject or total study population) can be compared in a PPC
> like manner. This is developed from Largajolli et al.
> ( https://www.page-meeting.org/default.asp?abstract=3208).
>
> In your case, you are interested in learning about the relative
> contribution of the different variables of a joint model. While
> not having tried it, I imagine that you can, based on your final
> joint model(s), obtain the expected OFV distribution one variable
> at a time as well as them jointly. From this it ought to be
> possible to learn some about the quality of the model with respect
> to variable A, variable B and their joint distribution in
> describing the real data.
>
> Best regards,
>
> Mats
>
> *From:*[email protected] <[email protected]>
> *On Behalf Of *Stephen Duffull
> *Sent:* den 10 oktober 2022 21:56
> *To:* Jeroen Elassaiss-Schaap (PD-value) <[email protected]>
> *Cc:* Matts Kågedal <[email protected]>; [email protected]
> *Subject:* RE: [NMusers] OFV by endpoint of joint models?
>
> HI Jeroen
>
> I tested this with additive error (i.e. interaction has no
> influence) and combined. Rank order was not preserved.
>
> To be clearer, this was a PK only example and I compared
> sum(CIWRES^2) for each individual vs PHI(). I was trying to see
> if I could get the PHI() per analyte for a multiple response model
> and thought that a quick way of doing this was to grab the
> relative contribution from CIWRES.
>
> Cheers
>
> Steve
>
> *From:*Jeroen Elassaiss-Schaap (PD-value) <[email protected]
> <mailto:[email protected]>>
> *Sent:* Tuesday, 11 October 2022 8:36 am
> *To:* Stephen Duffull <[email protected]
> <mailto:[email protected]>>
> *Cc:* Matts Kågedal <[email protected]
> <mailto:[email protected]>>; [email protected]
> <mailto:[email protected]>
> *Subject:* Re: [NMusers] OFV by endpoint of joint models?
>
> Hi Steven,
>
> Thanks for sharing! CWRES is “polluted” by the ETA gradients more
> directly compared to OFV. One would however hope for rank order
> consistency. Did you also test this without interaction? Might
> also be interesting to test the other residuals that nonmem offers
> in that respect.
>
> Cheers
>
> Jeroen
>
> http://pd-value.com
>
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1017274f1209442cc1b408daaaf6aab3%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010273680373817%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VG6Wo1oVz8Ti2jcYV0hY%2BdqZTETwRUgxbARS4eRhoY4%3D&reserved=0
> [email protected] <mailto:[email protected]>
> @PD_value
> +31 6 23118438 <tel:+31%206%2023118438>
> -- More value out of your data!
>
> Op 10 okt. 2022 om 18:51 heeft Stephen Duffull
> <[email protected]> het volgende geschreven:
>
> Hi Jeroen
>
> I note your thought about CWRES and OFV. In some exploratory
> work, we did not find that the rank order of abs(CIWRES) or
> CIWRES^2 and PHI() was preserved (with FOCEI) for continuous
> data. I had anticipated some rank similarity.
>
> Cheers
>
> Steve
> ________________________________________
> Stephen Duffull | Professor
> Otago Pharmacometrics Group
> School of Pharmacy | He Rau Kawakawa
> University of Otago | Te Whare Wānanga o Otāgo
> Dunedin | Ōtepoti
> Aotearoa New Zealand
> Ph: 64 3 479 5099
>
> -----Original Message-----
> From: [email protected]
> <mailto:[email protected]><[email protected]
> <mailto:[email protected]>> On Behalf Of Jeroen
> Elassaiss-Schaap (PD-value B.V.)
> Sent: Tuesday, 11 October 2022 4:07 am
> To: Matts Kågedal <[email protected]
> <mailto:[email protected]>>; [email protected]
> <mailto:[email protected]>
> Subject: Re: [NMusers] OFV by endpoint of joint models?
>
> Hi Matts,
>
> The easiest way to assess is when one of two endpoints is
> modeled directly (TTE, logistic regression) as often is the
> case, than look at the Y value for those endpoints, as
> reported in the PRED variable. The sum of those values is the
> ofv, or proportional to it, for that particular endpoint - the
> other endpoint is than affected in the inverse way.
>
> If you have multiple continuous endpoints it becomes more
> complicated.
> You could either look at the sum of absolute CWRES to get an
> idea, but not exact in terms of ofv comparison. Another
> approximate comparison would be to run the model without
> evaluation (e.g. MAXEVAL=0) with the original msfofile as
> $MSFI for the separate endpoints (by e.g.
> IGN(DVID.NE.x) where x is your endpoint). It is not exact,
> again, as it ignores the correlation between endpoints but
> should get you in the neighborhood. As an improvement to this
> method you could force evaluation at the original posthocs by
> reading them in in your datafile
> - this would still ignore correlation but the effect would be
> largely diminished because the posthocs are fixed to those
> estimated with correlation.
>
> Hope this helps,
>
> Jeroen
>
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1ebbd4a5cbce449913f108daaad22217%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010116756566698%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PEGSjbNvoRzu%2B6AbaXCXTHlNGkGc28Eb7QgKu1tE9sM%3D&reserved=0
>
> https://apc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpd-value.com%2F&data=05%7C01%7Cstephen.duffull%40otago.ac.nz%7C1ebbd4a5cbce449913f108daaad22217%7C0225efc578fe4928b1579ef24809e9ba%7C0%7C0%7C638010116756566698%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PEGSjbNvoRzu%2B6AbaXCXTHlNGkGc28Eb7QgKu1tE9sM%3D&reserved=0
> [email protected] <mailto:[email protected]>
> @PD_value
> +31 6 23118438
> -- More value out of your data!
>
> On 10-10-2022 16:03, Matts Kågedal wrote:
>
> Hi all,
>
> I have a question related to the objective function value
> when
>
> multiple endpoints are modelled jointly. Specifically I
> would like to
>
> know if a change in in OFV between models is driven
> primarily by one
>
> of the endpoints or if both contributes to the change, or
> maybe they
>
> are even driving the OFV in oposite directions.
>
> Is there a way to get some form of partial OFV by endpoint?
>
> Best regards,
>
> Matts
>
> När du har kontakt med oss på Uppsala universitet med e-post så
> innebär det att vi behandlar dina personuppgifter. För att läsa
> mer om hur vi gör det kan du läsa här:
> http://www.uu.se/om-uu/dataskydd-personuppgifter/
>
> E-mailing Uppsala University means that we will process your
> personal data. For more information on how this is performed,
> please read here: http://www.uu.se/en/about-uu/data-protection-policy
--
James G Wright PhD,
Scientist, Wright Dose Ltd
Tel: UK (0)772 5636914