Re: Humongous log file with parallel NONMEM
Ron and Erik,
The problem with those solutions is they don't actually address the I/O
concern. While you will save some disk space at any given time (which I
assume shouldn't be an issue for Bill on AWS) , you are actually *increasing
*the I/O burden (eg now you have to deal with creation and deletion).
IF disk I/O is a bottleneck, a better solution would be to delete the file only
when the file reaches a critical size, or wait to some other time when
there is lower I/O burden, perhaps at the end of the run with a post-run
hook.
Bill, a little bit of an out there solution (maybe) could be to create a
dummy file with the same name as the log file, then write/append protect
it, so even if nonmem attempts to overwrite/append it will not be able to.
I don't know how much nonmem would complain trying to write to that file
but being unable to. It'd be pretty hacky but could be worth a shot
depending on how much I/O is a bottleneck.
Devin
Quoted reply history
On Mon, Jun 22, 2015 at 2:42 PM <[email protected]> wrote:
> Dear Bill and Ron,
>
> I was thinking about:
>
> $ ln -s /dev/null <control_stream>.log
>
> (After deleting the log if one already exists)
>
> Data written to the log file is actually written to the null device, which
> discards the data - two tests seem to indicate that it could work.
>
> Best regards,
>
> Erik
> ------------------------------
> *From:* [email protected] [[email protected]] on
> behalf of Ron Keizer [[email protected]]
> *Sent:* Monday, June 22, 2015 7:32 PM
> *To:* Bill Gillespie
> *Cc:* [email protected]
> *Subject:* Re: [NMusers] Humongous log file with parallel NONMEM
>
> hi Bill,
>
> a simple hack is to delete the file continually, by running the
> following command before you start NONMEM:
>
> watch -n 60 'find . -name *.log -delete' &>/dev/null &
>
> Notes:
> - will delete all log files every minute
> - I'm using 'find ...' instead of just 'rm *.log' here to ensure that
> log-files in subfolders will also be deleted, e.g. useful when using via PsN
> - run the command only once, the watch process will stay active
> - only works on linux
>
> best regards,
> Ron
>
>
> ----------------------------------------------
> Ron Keizer, PharmD PhD
> Pirana Software & Consulting BV
> California / the Netherlands
> www.pirana-software.com
> ----------------------------------------------
>
> On Mon, Jun 22, 2015 at 6:22 AM, Bill Gillespie <[email protected]>
> wrote:
>
>> Hi all,
>>
>> I'm running NONMEM (METHOD = BAYES) in parallel on 32 cores and it
>> generates a humongous log file with repeated entries like the following:
>>
>> ITERATION -577
>> STARTING SUBJECTS 1 TO 4 ON MANAGER: OK
>> STARTING SUBJECTS 5 TO 8 ON WORKER1: OK
>> STARTING SUBJECTS 9 TO 11 ON WORKER2: OK
>> STARTING SUBJECTS 12 TO 15 ON WORKER3: OK
>> STARTING SUBJECTS 16 TO 18 ON WORKER4: OK
>> STARTING SUBJECTS 19 TO 20 ON WORKER5: OK
>> STARTING SUBJECTS 21 TO 24 ON WORKER6: OK
>> STARTING SUBJECTS 25 TO 27 ON WORKER7: OK
>> STARTING SUBJECTS 28 TO 29 ON WORKER8: OK
>> STARTING SUBJECTS 30 TO 32 ON WORKER9: OK
>> STARTING SUBJECTS 33 TO 35 ON WORKER10: OK
>> STARTING SUBJECTS 36 TO 39 ON WORKER11: OK
>> STARTING SUBJECTS 40 TO 42 ON WORKER12: OK
>> STARTING SUBJECTS 43 TO 46 ON WORKER13: OK
>> STARTING SUBJECTS 47 TO 50 ON WORKER14: OK
>> STARTING SUBJECTS 51 TO 53 ON WORKER15: OK
>> STARTING SUBJECTS 54 TO 58 ON WORKER16: OK
>> STARTING SUBJECTS 59 TO 62 ON WORKER17: OK
>> STARTING SUBJECTS 63 TO 66 ON WORKER18: OK
>> STARTING SUBJECTS 67 TO 70 ON WORKER19: OK
>> STARTING SUBJECTS 71 TO 71 ON WORKER20: OK
>> STARTING SUBJECTS 72 TO 74 ON WORKER21: OK
>> STARTING SUBJECTS 75 TO 77 ON WORKER22: OK
>> STARTING SUBJECTS 78 TO 80 ON WORKER23: OK
>> STARTING SUBJECTS 81 TO 84 ON WORKER24: OK
>> STARTING SUBJECTS 85 TO 86 ON WORKER25: OK
>> STARTING SUBJECTS 87 TO 88 ON WORKER26: OK
>> STARTING SUBJECTS 89 TO 90 ON WORKER27: OK
>> STARTING SUBJECTS 91 TO 93 ON WORKER28: OK
>> STARTING SUBJECTS 94 TO 96 ON WORKER29: OK
>> STARTING SUBJECTS 97 TO 99 ON WORKER30: OK
>> STARTING SUBJECTS 100 TO 103 ON WORKER31: OK
>> COLLECTING SUBJECTS 1 TO 4 ON MANAGER
>> COLLECTING SUBJECTS 5 TO 8 ON WORKER1
>> COLLECTING SUBJECTS 9 TO 11 ON WORKER2
>> COLLECTING SUBJECTS 12 TO 15 ON WORKER3
>> COLLECTING SUBJECTS 16 TO 18 ON WORKER4
>> COLLECTING SUBJECTS 19 TO 20 ON WORKER5
>> COLLECTING SUBJECTS 21 TO 24 ON WORKER6
>> COLLECTING SUBJECTS 25 TO 27 ON WORKER7
>> COLLECTING SUBJECTS 28 TO 29 ON WORKER8
>> COLLECTING SUBJECTS 30 TO 32 ON WORKER9
>> COLLECTING SUBJECTS 33 TO 35 ON WORKER10
>> COLLECTING SUBJECTS 36 TO 39 ON WORKER11
>> COLLECTING SUBJECTS 40 TO 42 ON WORKER12
>> COLLECTING SUBJECTS 43 TO 46 ON WORKER13
>> COLLECTING SUBJECTS 47 TO 50 ON WORKER14
>> COLLECTING SUBJECTS 51 TO 53 ON WORKER15
>> COLLECTING SUBJECTS 54 TO 58 ON WORKER16
>> COLLECTING SUBJECTS 59 TO 62 ON WORKER17
>> COLLECTING SUBJECTS 63 TO 66 ON WORKER18
>> COLLECTING SUBJECTS 67 TO 70 ON WORKER19
>> COLLECTING SUBJECTS 71 TO 71 ON WORKER20
>> COLLECTING SUBJECTS 72 TO 74 ON WORKER21
>> COLLECTING SUBJECTS 75 TO 77 ON WORKER22
>> COLLECTING SUBJECTS 78 TO 80 ON WORKER23
>> COLLECTING SUBJECTS 81 TO 84 ON WORKER24
>> COLLECTING SUBJECTS 85 TO 86 ON WORKER25
>> COLLECTING SUBJECTS 87 TO 88 ON WORKER26
>> COLLECTING SUBJECTS 89 TO 90 ON WORKER27
>> COLLECTING SUBJECTS 91 TO 93 ON WORKER28
>> COLLECTING SUBJECTS 94 TO 96 ON WORKER29
>> COLLECTING SUBJECTS 97 TO 99 ON WORKER30
>> COLLECTING SUBJECTS 100 TO 103 ON WORKER31
>>
>> The result is a lot of disk I/O and a file in the GB+ range. It dwarfs
>> the file containing the MCMC samples. Is there some way to suppress that
>> file or reduce what gets written to it?
>>
>> Thanks,
>> Bill
>>
>> William R Gillespie, VP Strategic Modeling & Simulation
>> Metrum Research Group LLC
>> 2 Tunxis Road, Tariffville, CT 06081
>> Direct & FAX: 919-371-2786, Main: 860-735-7043
>> [email protected]
>> www.metrumrg.com
>>
>
>