any means to suppress "Reading frame" numbers (not to output) in inverse.log - creates huge logs, hard to check on the run

[abrukhno]

Hi Votcians,

Is there any way to suppress outputting the running progress of Gromacs and Votca tools in inverse.log. This "Reading frame" numbers accumulate into huge logs, which are hard to check on the fly after a couple of iterations, especially on clusters. Moreover, it constantly writes to disk, reducing the performance.

I have tried to redirect stderr with "2>/dev/null" (for csg_inverse.sh),
or putting "1>/dev/null 2>/dev/null" for inverse.gromacs.log tag, but the latter removes the logs altogether, which is not desired.

Thanks

Andrey

[Christoph Junghans]

2015-01-09 8:19 GMT-07:00 abrukhno <abru...@googlemail.com>:
> Hi Votcians,
>
> Is there any way to suppress outputting the running progress of Gromacs and Votca tools in inverse.log. This "Reading frame" numbers accumulate into huge logs, which are hard to check on the fly after a couple of iterations, especially on clusters. Moreover, it constantly writes to disk, reducing the performance.

There is not easy way, "Reading frame" comes out of libgmx. Looking at
printcount_ in src/gmxlib/trxio.c there does not seem to be a way to
quiet it.

> I have tried to redirect stderr with "2>/dev/null" (for csg_inverse.sh),
> or putting "1>/dev/null 2>/dev/null" for inverse.gromacs.log tag, but the latter removes the logs altogether, which is not desired.

inverse.gromacs.log is a separate log file for gromacs program, but
not for csg_inverse itself, so that won't work here.
"Reading frame" goes to stderr, but forwarding that to /dev/null, one
would loss all other error messages.
You might be able to create a post-update script, which does something like:
sync && sed -e '/Reading frame/d' -i $CSGLOG

Christoph

>
> Thanks
>
> Andrey
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "votca" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to votca+un...@googlegroups.com.
> To post to this group, send email to vo...@googlegroups.com.
> Visit this group at http://groups.google.com/group/votca.
> For more options, visit https://groups.google.com/d/optout.



--
Christoph Junghans
Web: http://www.compphys.de

[abrukhno]

Thanks for prompt replying Christoph!

On Friday, January 9, 2015 at 4:31:38 PM UTC, Christoph Junghans wrote:
> 2015-01-09 8:19 GMT-07:00 abrukhno <abru...@googlemail.com>:
> > Hi Votcians,
> >
> > Is there any way to suppress outputting the running progress of Gromacs and Votca tools in inverse.log. This "Reading frame" numbers accumulate into huge logs, which are hard to check on the fly after a couple of iterations, especially on clusters. Moreover, it constantly writes to disk, reducing the performance.
> There is not easy way, "Reading frame" comes out of libgmx. Looking at
> printcount_ in src/gmxlib/trxio.c there does not seem to be a way to
> quiet it.

- That is really inconvenient and annoying. On one of the clusters I am using this "feature" does not allow me (in any practical terms) to see inverse.log for some minutes (after a dozen or more iterations). I can only use "tail" then, while I would prefer "less", but this is not the only my concern...

> > I have tried to redirect stderr with "2>/dev/null" (for csg_inverse.sh),
> > or putting "1>/dev/null 2>/dev/null" for inverse.gromacs.log tag, but the latter removes the logs altogether, which is not desired.
> inverse.gromacs.log is a separate log file for gromacs program, but
> not for csg_inverse itself, so that won't work here.
> "Reading frame" goes to stderr, but forwarding that to /dev/null, one
> would loss all other error messages.
> You might be able to create a post-update script, which does something like:
> sync && sed -e '/Reading frame/d' -i $CSGLOG

- This could be a solution, if the performance of analysis tools was not affected, but writing so often to disk (after every single frame read!) should drastically increase the execution time during the reading stage, and it does take quite some time - about 20-30 mins for about 10 mln frames in trajectory in my latest case (which is relatively light in terms of the number of beads).

- may I ask you what "exec 3>&1 4>&2 >> "$CSGLOG" 2>&1" does in enable_logging() (functions_common.sh)? I mean I could not find out the meaning of file descriptors "3" and "4" (I reckon exec redirects them).

Andrey

[abrukhno]

I guess this sounds like general moaning, so I will probably stick to 2>/dev/null after making sure the iteration goes well.

However, I am not sure - where do I put it?
Inserting it in "cg.inverse.log_file" tag does not help, it simply includes it into the files name so it becomes "inverse.log 2>/dev/null", which is weird.

If instead I use "csg_inverse.sh ... <options> ... 1> csg_inverse.log 2>/dev/null", then nothing changes. Basically, csg_inverse.sh does not substitute or rename inverse.log but creates another, less comprehensive log file.

I am still puzzled how to achieve suppressing "errors" (let say all of them).

Cheers

[Christoph Junghans]

2015-01-09 9:53 GMT-07:00 abrukhno <abru...@googlemail.com>:
> Thanks for prompt replying Christoph!
>
> On Friday, January 9, 2015 at 4:31:38 PM UTC, Christoph Junghans wrote:
>> 2015-01-09 8:19 GMT-07:00 abrukhno <abru...@googlemail.com>:
>> > Hi Votcians,
>> >
>> > Is there any way to suppress outputting the running progress of Gromacs and Votca tools in inverse.log. This "Reading frame" numbers accumulate into huge logs, which are hard to check on the fly after a couple of iterations, especially on clusters. Moreover, it constantly writes to disk, reducing the performance.
>> There is not easy way, "Reading frame" comes out of libgmx. Looking at
>> printcount_ in src/gmxlib/trxio.c there does not seem to be a way to
>> quiet it.
>
> - That is really inconvenient and annoying. On one of the clusters I am using this "feature" does not allow me (in any practical terms) to see inverse.log for some minutes (after a dozen or more iterations). I can only use "tail" then, while I would prefer "less", but this is not the only my concern...
>
>> > I have tried to redirect stderr with "2>/dev/null" (for csg_inverse.sh),
>> > or putting "1>/dev/null 2>/dev/null" for inverse.gromacs.log tag, but the latter removes the logs altogether, which is not desired.
>> inverse.gromacs.log is a separate log file for gromacs program, but
>> not for csg_inverse itself, so that won't work here.
>> "Reading frame" goes to stderr, but forwarding that to /dev/null, one
>> would loss all other error messages.
>> You might be able to create a post-update script, which does something like:
>> sync && sed -e '/Reading frame/d' -i $CSGLOG
>
> - This could be a solution, if the performance of analysis tools was not affected, but writing so often to disk (after every single frame read!) should drastically increase the execution time during the reading stage, and it does take quite some time - about 20-30 mins for about 10 mln frames in trajectory in my latest case (which is relatively light in terms of the number of beads).
>
> - may I ask you what "exec 3>&1 4>&2 >> "$CSGLOG" 2>&1" does in enable_logging() (functions_common.sh)? I mean I could not find out the meaning of file descriptors "3" and "4" (I reckon exec redirects them).

We create two new file descriptors ("3" and "4") and 3 is linked to
1(stdout) and 4 to 2(stderr), while the current 1 (stdout) is appended
to "$CSGLOG" and the current 2(stderr) is appended to the current 1
(stdout).
This way, if you do use stdout/stderr in a script it will end up in
the log file and not on the screen, while if you use fd 3,4 will end
up of the screen (that is what 'msg' is doing).

I guess
exec 3>&1 4>&2 >> "$CSGLOG" 2>/dev/null
will do what you want.

We could add an option like cg.inverse.error_log_file and use
exec 3>&1 4>&2 >> "$CSGLOG" 2> "$CSG_ERRORLOG"

[abrukhno]

- that might be useful, but maybe you don't put quotes around $CSG_ERRORLOG ?
otherwise, can it create a file named "/dev/null" instead of dumping the output into the null device? I've tried "inverse.log 2>inverse.err" for $CSGLOG (via tag cg.inverse.log_file in settings.xml) and got the file named "inverse.log 2>inverse.err" (without quotes)

- out of curiosity, do you ever actually use descriptors 3 and/or 4, or is it there just in case for the future uses?

Thanks

[Christoph Junghans]

2015-01-09 14:41 GMT-07:00 abrukhno <abru...@googlemail.com>:
> On Friday, January 9, 2015 at 6:52:55 PM UTC, Christoph Junghans wrote:
>> 2015-01-09 9:53 GMT-07:00 abrukhno <abru...@googlemail.com>:
>> > Thanks for prompt replying Christoph!
>> >
>> > On Friday, January 9, 2015 at 4:31:38 PM UTC, Christoph Junghans wrote:
>> >> 2015-01-09 8:19 GMT-07:00 abrukhno <abru...@googlemail.com>:
>> >> > Hi Votcians,
>> >> >
>> >> > Is there any way to suppress outputting the running progress of Gromacs and Votca tools in inverse.log. This "Reading frame" numbers accumulate into huge logs, which are hard to check on the fly after a couple of iterations, especially on clusters. Moreover, it constantly writes to disk, reducing the performance.
>> >> There is not easy way, "Reading frame" comes out of libgmx. Looking at
>> >> printcount_ in src/gmxlib/trxio.c there does not seem to be a way to
>> >> quiet it.
>> >
>> > - That is really inconvenient and annoying. On one of the clusters I am using this "feature" does not allow me (in any practical terms) to see inverse.log for some minutes (after a dozen or more iterations). I can only use "tail" then, while I would prefer "less", but this is not the only my concern...
>> >
>> >> > I have tried to redirect stderr with "2>/dev/null" (for csg_inverse.sh),
>> >> > or putting "1>/dev/null 2>/dev/null" for inverse.gromacs.log tag, but the latter removes the logs altogether, which is not desired.
>> >> inverse.gromacs.log is a separate log file for gromacs program, but
>> >> not for csg_inverse itself, so that won't work here.
>> >> "Reading frame" goes to stderr, but forwarding that to /dev/null, one
>> >> would loss all other error messages.
>> >> You might be able to create a post-update script, which does something like:
>> >> sync && sed -e '/Reading frame/d' -i $CSGLOG
>> >
>> > - This could be a solution, if the performance of analysis tools was not affected, but writing so often to disk (after every single frame read!) should drastically increase the execution time during the reading stage, and it does take quite some time - about 20-30 mins for about 10 mln frames in trajectory in my latest case (which is relatively light in terms of the number of beads).
>> >

It don't think that this is a a problem as the IO is buffered.

>> > - may I ask you what "exec 3>&1 4>&2 >> "$CSGLOG" 2>&1" does in enable_logging() (functions_common.sh)? I mean I could not find out the meaning of file descriptors "3" and "4" (I reckon exec redirects them).
>> We create two new file descriptors ("3" and "4") and 3 is linked to
>> 1(stdout) and 4 to 2(stderr), while the current 1 (stdout) is appended
>> to "$CSGLOG" and the current 2(stderr) is appended to the current 1
>> (stdout).
>> This way, if you do use stdout/stderr in a script it will end up in
>> the log file and not on the screen, while if you use fd 3,4 will end
>> up of the screen (that is what 'msg' is doing).
>>
>> I guess
>> exec 3>&1 4>&2 >> "$CSGLOG" 2>/dev/null
>> will do what you want.
>>
>> We could add an option like cg.inverse.error_log_file and use
>> exec 3>&1 4>&2 >> "$CSGLOG" 2> "$CSG_ERRORLOG"
>
> - that might be useful, but maybe you don't put quotes around $CSG_ERRORLOG ?
> otherwise, can it create a file named "/dev/null" instead of dumping the output into the null device? I've tried "inverse.log 2>inverse.err" for $CSGLOG (via tag cg.inverse.log_file in settings.xml) and got the file named "inverse.log 2>inverse.err" (without quotes)

The quotes are to support spaces in the file names. And your trick
won't work in bash anyhow:
$ xxx="1 2> 2"
$ echo hallo > $xxx
-bash: $xxx: ambiguous redirect

>
> - out of curiosity, do you ever actually use descriptors 3 and/or 4, or is it there just in case for the future uses?

This is use in the "msg" function:
$ grep ">&[34]" functions_common.sh
echo -e "${color}$*${off}" >&4
echo -e "${color}$*${off}" >&3
Their use is that inside the script the redirect to log file is
automatic and not every command needs to redirect with
"&>" separately.

Christoph

[abrukhno]

> >> I guess
> >> exec 3>&1 4>&2 >> "$CSGLOG" 2>/dev/null
> >> will do what you want.
> >>
> >> We could add an option like cg.inverse.error_log_file and use
> >> exec 3>&1 4>&2 >> "$CSGLOG" 2> "$CSG_ERRORLOG"
> >
> > - that might be useful, but maybe you don't put quotes around $CSG_ERRORLOG ?

- For the time being I've simply overridden inverse.sh within my_scripts directory, replacing the call to enable_logging() with its body altered as follows (since I cannot override functions_common.sh this way):
===
...
export CSGLOG="$log"
if [[ -f $CSGLOG ]]; then

exec 3>&1 4>&2 >> "$CSGLOG" 2>&1

echo "\n\n#################################"
echo "# Appending to existing logfile #"
echo "#################################\n\n"
msg --color blue "Appending to existing logfile ${CSGLOG##*/}"
else

exec 3>&1 4>&2 >> "$CSGLOG" 2>/dev/null

echo "\n\n#################################"
echo "# Creating new logfile & dumping errors #"
echo "#################################\n\n"
msg "For a more verbose log see: ${CSGLOG##*/}"
fi
===
Works fine, where I can chose to dump errors or not, by either having or not pre-existing inverse.log (so that if I've got an error, next time this will show up in the appended inverse.log)

BTW, why do you need "\n\n" in echo strings? - they show up "as is", i.e. no extra lines around the message (if that was the purpose).

> > otherwise, can it create a file named "/dev/null" instead of dumping the output into the null device? I've tried "inverse.log 2>inverse.err" for $CSGLOG (via tag cg.inverse.log_file in settings.xml) and got the file named "inverse.log 2>inverse.err" (without quotes)
> The quotes are to support spaces in the file names. And your trick
> won't work in bash anyhow:
> $ xxx="1 2> 2"
> $ echo hallo > $xxx
> -bash: $xxx: ambiguous redirect

- what you say, I won't be able to set $CSG_ERRORLOG to "/dev/null" anyway, but could only split the log into stdout and stderr outputs? not sure if this would be that useful (if only to ease viewing inverse.log without the tools' progress reports)

Perhaps, my last question/suggestion here: isn't the starting bit of enable_logging() supposed to invoke "2>/dev/null" after the name provided in cg.inverse.log_file tag (maybe when it wasn't found?)? - it doesn't though.
===
local log
if [[ -z $1 ]]; then
log="$(csg_get_property cg.inverse.log_file 2>/dev/null)"
else
log="$1"
fi
===
Anyways one could get around the errors logging by doing "2>/dev/null" only in the case when cg.inverse.log_file is not found in settings.xml (or has a specific name, like inverse_no_stderr.log), and "2>&1" otherwise.

Andrey

[Christoph Junghans]

We had overwriting of function_common.sh at some point, but it was too
error-prone as the user would have to re-incorporate our changes all
the time.

>
> BTW, why do you need "\n\n" in echo strings? - they show up "as is", i.e. no extra lines around the message (if that was the purpose).

Should be "echo -e", now fixed in the code.

>
>> > otherwise, can it create a file named "/dev/null" instead of dumping the output into the null device? I've tried "inverse.log 2>inverse.err" for $CSGLOG (via tag cg.inverse.log_file in settings.xml) and got the file named "inverse.log 2>inverse.err" (without quotes)
>> The quotes are to support spaces in the file names. And your trick
>> won't work in bash anyhow:
>> $ xxx="1 2> 2"
>> $ echo hallo > $xxx
>> -bash: $xxx: ambiguous redirect
>
> - what you say, I won't be able to set $CSG_ERRORLOG to "/dev/null" anyway, but could only split the log into stdout and stderr outputs? not sure if this would be that useful (if only to ease viewing inverse.log without the tools' progress reports)

What I was saying is that setting cg.inverse.log_file to something
line "file1 2> file2" will never work.

>
> Perhaps, my last question/suggestion here: isn't the starting bit of enable_logging() supposed to invoke "2>/dev/null" after the name provided in cg.inverse.log_file tag (maybe when it wasn't found?)? - it doesn't though.
> ===
> local log
> if [[ -z $1 ]]; then
> log="$(csg_get_property cg.inverse.log_file 2>/dev/null)"
> else
> log="$1"
> fi
> ===
> Anyways one could get around the errors logging by doing "2>/dev/null" only in the case when cg.inverse.log_file is not found in settings.xml (or has a specific name, like inverse_no_stderr.log), and "2>&1" otherwise.

No, that should like too much magic to me.

As said previously, we could add something like
cg.inverse.error_log_file, however I think your case is pretty
special.
Can you do a benchmark, how much faster using /dev/null actually is?

Beside the fact I don't like the idea of throwing all error messages away.

Christoph

[abrukhno]

- never mind, I have it working the way I like by now :)

> >
> > BTW, why do you need "\n\n" in echo strings? - they show up "as is", i.e. no extra lines around the message (if that was the purpose).
> Should be "echo -e", now fixed in the code.
>
> >
> >> > otherwise, can it create a file named "/dev/null" instead of dumping the output into the null device? I've tried "inverse.log 2>inverse.err" for $CSGLOG (via tag cg.inverse.log_file in settings.xml) and got the file named "inverse.log 2>inverse.err" (without quotes)
> >> The quotes are to support spaces in the file names. And your trick
> >> won't work in bash anyhow:
> >> $ xxx="1 2> 2"
> >> $ echo hallo > $xxx
> >> -bash: $xxx: ambiguous redirect
> >
> > - what you say, I won't be able to set $CSG_ERRORLOG to "/dev/null" anyway, but could only split the log into stdout and stderr outputs? not sure if this would be that useful (if only to ease viewing inverse.log without the tools' progress reports)
> What I was saying is that setting cg.inverse.log_file to something
> line "file1 2> file2" will never work.

- ok, but what the beginning of enable_logging() does then? (see next below)

> >
> > Perhaps, my last question/suggestion here: isn't the starting bit of enable_logging() supposed to invoke "2>/dev/null" after the name provided in cg.inverse.log_file tag (maybe when it wasn't found?)? - it doesn't though.
> > ===
> > local log
> > if [[ -z $1 ]]; then
> > log="$(csg_get_property cg.inverse.log_file 2>/dev/null)"
> > else
> > log="$1"
> > fi
> > ===
> > Anyways one could get around the errors logging by doing "2>/dev/null" only in the case when cg.inverse.log_file is not found in settings.xml (or has a specific name, like inverse_no_stderr.log), and "2>&1" otherwise.
> No, that should like too much magic to me.

- in fact, I prefer it the way I did by overriding.

> As said previously, we could add something like
> cg.inverse.error_log_file, however I think your case is pretty
> special.
> Can you do a benchmark, how much faster using /dev/null actually is?

- let me see some finished iterations done with different outputs,
but even if on clusters buffering takes care of the overheads, I often tune the settings/options by running preliminary iterations on my PC or laptop, where there buffering is only modest.

Besides, as I said before, getting and browsing a log of some hundreds Mb of "reading frame" progress reports, is not a good style anyway.

> Beside the fact I don't like the idea of throwing all error messages away.

- no problem, I can get the errors by re-running for them the second time.
It would be actually cumbersome to edit settings.xml every time to switch between the two modes (+/-errors)

Cheers & Happy New Year btw! :)

[abrukhno]

...about this:

log="$(csg_get_property cg.inverse.log_file 2>/dev/null)"

I've just noticed that the ending "2>/dev/null" is not added to the string produced but instead dumps the errors of the command in parenthesis (earlier I was "blind", not seeing these).

AB

[Christoph Junghans]

Yeah, $( ) (and ` `) only fetch stdin, but not stderr.
$ x=; x=$(echo yeah); echo x=$x
x=yeah
$ x=; x=$(echo yeah >&2); echo x=$x
yeah
x=

In our case csg_get_property prints a couple of extra information,
like when a default values is used, to stderr and we don't want the
user to see it. Later stderr will just go to the log file, so it won't
matter.

Christoph
>
> AB

[Andrey Brukhno]

I have just found out that even if I use "1>inverse.log 2>/dev/null"
the errors are still end up in csg_errors.log that I provide for

csg_inverse.sh script. So not much is missing.

Andrey

You received this message because you are subscribed to a topic in the Google Groups "votca" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/votca/OTEEUvhUavU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to votca+un...@googlegroups.com.