Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Re: What should go to stdout/stderr and why Python logging write everything to stderr?
1,933 views
Skip to first unread message
Thomas Passin
Jan 3, 2023, 11:27:28 AM1/3/23
to
On 1/3/2023 10:35 AM, c.b...@posteo.jp wrote:
> Hello,
>
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
>
> The logging module write everything to stderr no matter which logging
> level is used.
>
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
>
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
I do not have any special insight into the developers' minds, but here
is my own reaction. Logging output is normally not the primary output
of a program. For example, you might want to capture the program's
output and do some processing on it. In that case you don't want to
have to parse out the logging text so as to avoid processing it. If not
to stdout, then where should logging output go? The only two
possibilities seem to be stderr, or a known file.
Also, I think it would be confusing to sometimes have logging output go
to stdout and other times go to stderr.
In the case of argparse, the messages are the intended program output,
so to me it makes sense that error announcements should go to stderr and
requests for help information should go to stdout.
There is probably no single right way to do it, but the way it's being
done makes sense to me, FWIW.
>
> Of course I could modify the handlers etc. to workaround this. But I
> assume that there was something in mind of the Python developers when
> they decided that.
>
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.
> Hello,
>
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
>
> The logging module write everything to stderr no matter which logging
> level is used.
>
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
>
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
I do not have any special insight into the developers' minds, but here
is my own reaction. Logging output is normally not the primary output
of a program. For example, you might want to capture the program's
output and do some processing on it. In that case you don't want to
have to parse out the logging text so as to avoid processing it. If not
to stdout, then where should logging output go? The only two
possibilities seem to be stderr, or a known file.
Also, I think it would be confusing to sometimes have logging output go
to stdout and other times go to stderr.
In the case of argparse, the messages are the intended program output,
so to me it makes sense that error announcements should go to stderr and
requests for help information should go to stdout.
There is probably no single right way to do it, but the way it's being
done makes sense to me, FWIW.
>
> Of course I could modify the handlers etc. to workaround this. But I
> assume that there was something in mind of the Python developers when
> they decided that.
>
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.
MRAB
Jan 3, 2023, 11:40:20 AM1/3/23
to
On 2023-01-03 15:35, c.b...@posteo.jp wrote:
> Hello,
>
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
>
> The logging module write everything to stderr no matter which logging
> level is used.
>
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
>
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
>
> Hello,
>
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
>
> The logging module write everything to stderr no matter which logging
> level is used.
>
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
>
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
>
> Of course I could modify the handlers etc. to workaround this. But I
> assume that there was something in mind of the Python developers when
> they decided that.
>
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.
The purpose of stderr is to display status messages, logging and error
> assume that there was something in mind of the Python developers when
> they decided that.
>
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.
messages, even user prompts, and not mess up the program's actual
output. This is important on a *nix system where you might be piping the
output of one program into the input of another.
Richard Damon
Jan 3, 2023, 11:41:07 AM1/3/23
to
> On Jan 3, 2023, at 10:38 AM, c.b...@posteo.jp wrote:
> Hello,
>
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
>
> The logging module write everything to stderr no matter which logging
> level is used.
>
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
>
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
>
> Of course I could modify the handlers etc. to workaround this. But I
> assume that there was something in mind of the Python developers when
> they decided that.
>
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.
> https://mail.python.org/mailman/listinfo/python-list
My thought are, that a “Log” should be a COMPLETE output of what is logged. Logged messages are NOT intended to be a message to the current user, but for a post operation analysis. Remember, a message sent by logging.info may never be seen, as that level of logging may be turned off. (You are providing ways to control how much is being set to the log, especially if going to the “screen”).
As such, stderr is a often better spot to send the log than stdout, as the program may well be using stdout to interact with the user, and you don’t want the interaction cluttered with the logging.
Also, I am fairly sure you can configure the logger to do what you want by just configuring two different logging handlers, one for stderr that does just errors, and one for stdout that shows all messages. If you don’t want the errors on stderr and stdout, just log the errors to a different logger which is on an error.* hierarchy and don’t let that hierarchy propagate up to the root logger.
Weatherby,Gerard
Jan 3, 2023, 12:18:27 PM1/3/23
to
Really doesn’t have much to do with Python and very much with standard streams, which go back as far as the 1950s. https://en.wikipedia.org/wiki/Standard_streams
When a user types -h / --help to a Python argparse script, the output of the script is the help message. The standard behavior is to print that and exit, regardless of any other arguments that are passed. So its behavior is consistent with the concept of standard out.
From: Python-list <python-list-bounces+gweatherby=uchc...@python.org> on behalf of MRAB <pyt...@mrabarnett.plus.com>
Date: Tuesday, January 3, 2023 at 11:41 AM
To: pytho...@python.org <pytho...@python.org>
Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr?
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***
On 2023-01-03 15:35, c.b...@posteo.jp wrote:
> Hello,
>
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
>
> The logging module write everything to stderr no matter which logging
> level is used.
>
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
>
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
>
> Of course I could modify the handlers etc. to workaround this. But I
> assume that there was something in mind of the Python developers when
> they decided that.
>
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jni-tegcOXxLEkj6ohH0wbjz0Gd2OmHirJln0S1hX5z3nT6MUnqw4-ZMgaJGkzT228O9zSczGws4tyRERCMntNGR$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jni-tegcOXxLEkj6ohH0wbjz0Gd2OmHirJln0S1hX5z3nT6MUnqw4-ZMgaJGkzT228O9zSczGws4tyRERCMntNGR$>
When a user types -h / --help to a Python argparse script, the output of the script is the help message. The standard behavior is to print that and exit, regardless of any other arguments that are passed. So its behavior is consistent with the concept of standard out.
From: Python-list <python-list-bounces+gweatherby=uchc...@python.org> on behalf of MRAB <pyt...@mrabarnett.plus.com>
Date: Tuesday, January 3, 2023 at 11:41 AM
To: pytho...@python.org <pytho...@python.org>
Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr?
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***
On 2023-01-03 15:35, c.b...@posteo.jp wrote:
> Hello,
>
> this posting isn't about asking for a technical solution. My intention
> is to understand the design decision Python's core developers made in
> context of that topic.
>
> The logging module write everything to stderr no matter which logging
> level is used.
>
> The argparse module does it more differentiated. If arguments are
> mandatory but none are given then argparse produce a short version of
> the usage info; on stderr. If the user request the usage info via "-h"
> it goes to stdout. This is the behavior I would expect.
>
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
>
> Of course I could modify the handlers etc. to workaround this. But I
> assume that there was something in mind of the Python developers when
> they decided that.
>
> My goal is not to divide between the use of print() or logging.info()
> in my own code. This would mess up a lot.
The purpose of stderr is to display status messages, logging and error
messages, even user prompts, and not mess up the program's actual
output. This is important on a *nix system where you might be piping the
output of one program into the input of another.
--
messages, even user prompts, and not mess up the program's actual
output. This is important on a *nix system where you might be piping the
output of one program into the input of another.
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jni-tegcOXxLEkj6ohH0wbjz0Gd2OmHirJln0S1hX5z3nT6MUnqw4-ZMgaJGkzT228O9zSczGws4tyRERCMntNGR$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jni-tegcOXxLEkj6ohH0wbjz0Gd2OmHirJln0S1hX5z3nT6MUnqw4-ZMgaJGkzT228O9zSczGws4tyRERCMntNGR$>
Eryk Sun
Jan 3, 2023, 1:32:02 PM1/3/23
to
On 1/3/23, c.b...@posteo.jp <c.b...@posteo.jp> wrote:
>
> If the user request the usage info via "-h" it goes to stdout.
The standard file for application output is sys.stdout. Note that
>
> If the user request the usage info via "-h" it goes to stdout.
sys.stdout may be buffered, particularly if it isn't a tty. When a
file is buffered, writes are aggregated and only written to the OS
file when the buffer fills up or is manually flushed.
> Why does logging behave different? DEBUG and INFO imho should go to
> stdout not stderr.
is sys.stderr. This file should never be buffered.
Keith Thompson
Jan 3, 2023, 1:46:06 PM1/3/23
to
MRAB <pyt...@mrabarnett.plus.com> writes:
[...]
system-specific stream like the current tty, not to stderr. If a
program has user prompts, it probably doesn't make sense to pipe its
output to the input of another.
--
Keith Thompson (The_Other_Keith) Keith.S.T...@gmail.com
Working, but not speaking, for XCOM Labs
void Void(void) { Void(); } /* The recursive call of the void */
[...]
> The purpose of stderr is to display status messages, logging and error
> messages, even user prompts, and not mess up the program's actual
> output. This is important on a *nix system where you might be piping
> the output of one program into the input of another.
I would expect user prompts to be written to stdout, or perhaps to some
> messages, even user prompts, and not mess up the program's actual
> output. This is important on a *nix system where you might be piping
> the output of one program into the input of another.
system-specific stream like the current tty, not to stderr. If a
program has user prompts, it probably doesn't make sense to pipe its
output to the input of another.
--
Keith Thompson (The_Other_Keith) Keith.S.T...@gmail.com
Working, but not speaking, for XCOM Labs
void Void(void) { Void(); } /* The recursive call of the void */
Thomas Passin
Jan 3, 2023, 2:34:18 PM1/3/23
to
On 1/3/2023 11:51 AM, Stefan Ram wrote:
> so error messages written to the output might get appended
> to some result file and might not be seen by the user.
Yes, and in Windows, too. Note that stderr messages can be written or
redirected to a file, too, and then would not be seen on the terminal.
> Therefore, UNIX and C provide the unbuffered "stderr" stream
> that can be redirected separately.
>
> The following example shows how logging might write to stdout.
>
> import logging
> import sys
> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
> logging.error( 'example' )
This is starting to look too much like java. Yuck!
> Thomas Passin <li...@tompassin.net> writes:
>> On 1/3/2023 10:35 AM, c.b...@posteo.jp wrote:
>> Also, I think it would be confusing to sometimes have logging output go
>> to stdout and other times go to stderr.
>
> In UNIX, the output of a program can be redirected,
>> On 1/3/2023 10:35 AM, c.b...@posteo.jp wrote:
>> Also, I think it would be confusing to sometimes have logging output go
>> to stdout and other times go to stderr.
>
> so error messages written to the output might get appended
> to some result file and might not be seen by the user.
Yes, and in Windows, too. Note that stderr messages can be written or
redirected to a file, too, and then would not be seen on the terminal.
> Therefore, UNIX and C provide the unbuffered "stderr" stream
> that can be redirected separately.
>
> The following example shows how logging might write to stdout.
>
> import logging
> import sys
> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
> logging.error( 'example' )
This is starting to look too much like java. Yuck!
Weatherby,Gerard
Jan 3, 2023, 2:52:48 PM1/3/23
to
If sys.stdout is a tty, it typically flushes on newline. e. g.
!/usr/bin/env python3
import time
import sys
print("No flush",end='',file=sys.stdout)
time.sleep(2)
print("flushed",file=sys.stdout)
time.sleep(5)
will print the “flushed” 5 seconds before the script ends
From: Python-list <python-list-bounces+gweatherby=uchc...@python.org> on behalf of Eryk Sun <ery...@gmail.com>
Date: Tuesday, January 3, 2023 at 1:33 PM
To: c.b...@posteo.jp <c.b...@posteo.jp>
Cc: pytho...@python.org <pytho...@python.org>
Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr?
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jGBVNrRuUThZCKMmShbuvBgggwv7FBDL9h2vW-vvehPnBHdfkrJUhohhZCgsCAqlRrDluk9c526jABrLjg$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jGBVNrRuUThZCKMmShbuvBgggwv7FBDL9h2vW-vvehPnBHdfkrJUhohhZCgsCAqlRrDluk9c526jABrLjg$>
!/usr/bin/env python3
import time
import sys
print("No flush",end='',file=sys.stdout)
time.sleep(2)
print("flushed",file=sys.stdout)
time.sleep(5)
will print the “flushed” 5 seconds before the script ends
From: Python-list <python-list-bounces+gweatherby=uchc...@python.org> on behalf of Eryk Sun <ery...@gmail.com>
Date: Tuesday, January 3, 2023 at 1:33 PM
To: c.b...@posteo.jp <c.b...@posteo.jp>
Cc: pytho...@python.org <pytho...@python.org>
Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr?
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jGBVNrRuUThZCKMmShbuvBgggwv7FBDL9h2vW-vvehPnBHdfkrJUhohhZCgsCAqlRrDluk9c526jABrLjg$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!jGBVNrRuUThZCKMmShbuvBgggwv7FBDL9h2vW-vvehPnBHdfkrJUhohhZCgsCAqlRrDluk9c526jABrLjg$>
Eryk Sun
Jan 3, 2023, 3:30:09 PM1/3/23
to
On 1/3/23, Weatherby,Gerard <gweat...@uchc.edu> wrote:
> If sys.stdout is a tty, it typically flushes on newline. e. g.
Sorry if I wasn't clear. Yes, a tty file will be buffered, but it's
> If sys.stdout is a tty, it typically flushes on newline. e. g.
line buffering, which isn't an issue as long as lines are written to
stdout. I was referring to full buffering of pipe and disk files. This
can be a problem when printing diagnostics. We went the information
written to the file immediately, not whenever the buffer happens to
fill up. Thus sys.stderr is unbuffered.
c.b...@posteo.jp
Jan 3, 2023, 4:24:50 PM1/3/23
to
Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:
> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
But I don't want to make all log levels go to stdout. Just DEBUG and
INFO. But this would be a workaround.
The main question here is why does Python deciecded to make all logs go
to stderr?
Maybe I totally misunderstood the intention of logging.info()?! Isn't
this the "usual applicaton output"?
If not, what messages should go into logging.info()? Can you name me
some examples?
> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
But I don't want to make all log levels go to stdout. Just DEBUG and
INFO. But this would be a workaround.
The main question here is why does Python deciecded to make all logs go
to stderr?
Maybe I totally misunderstood the intention of logging.info()?! Isn't
this the "usual applicaton output"?
If not, what messages should go into logging.info()? Can you name me
some examples?
Cameron Simpson
Jan 3, 2023, 4:30:12 PM1/3/23
to
On Jan 3, 2023, at 10:38 AM, c.b...@posteo.jp wrote:
>> this posting isn't about asking for a technical solution. My
>> intention is to understand the design decision Python's core developers made in
>> context of that topic.
>>
>> The logging module write everything to stderr no matter which logging
>> level is used.
Sure. They're logs.
>> intention is to understand the design decision Python's core developers made in
>> context of that topic.
>>
>> The logging module write everything to stderr no matter which logging
>> level is used.
>> The argparse module does it more differentiated. If arguments are
>> mandatory but none are given then argparse produce a short version of
>> the usage info; on stderr. If the user request the usage info via "-h"
>> it goes to stdout. This is the behavior I would expect.
>> Why does logging behave different? DEBUG and INFO imho should go to
>> stdout not stderr.
Consider a shell pipeline:
cmd1 | cmd2 | cmd3
where `cmd2` and `cmd3` parse data coming down the pipes. Letting
logging info into those data streams would produce garbage. This
separation between core programme output and error/logging output is the
basic reason that the two streams exist, and the reason logging goes to
stderr by default.
You're conflating "normal output" with INFO, which is not a substitute
for eg `print()`.
>> Of course I could modify the handlers etc. to workaround this. But I
>> assume that there was something in mind of the Python developers when
>> they decided that.
to stdout. The log level doesn't change that characteristic.
>> My goal is not to divide between the use of print() or logging.info()
>> in my own code. This would mess up a lot.
resist doing if it is your goal.
But keep in mind that if you do that globally (eg in the root logger) it
will affect _all_ logging calls, not merely your own, and other
libraries will assume they can log at whatever level and not pollute
stdout with their logs.
Basicly, logging isn't "output".
Cheers,
Cameron Simpson <c...@cskk.id.au>
Chris Angelico
Jan 3, 2023, 5:00:17 PM1/3/23
to
On Wed, 4 Jan 2023 at 08:25, <c.b...@posteo.jp> wrote:
>
> Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:
> > logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
>
> But I don't want to make all log levels go to stdout. Just DEBUG and
> INFO. But this would be a workaround.
But why are DEBUG logs part of stdout? That doesn't make any sense.
>
> Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:
> > logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
>
> But I don't want to make all log levels go to stdout. Just DEBUG and
> INFO. But this would be a workaround.
> The main question here is why does Python deciecded to make all logs go
> to stderr?
> Maybe I totally misunderstood the intention of logging.info()?! Isn't
> this the "usual applicaton output"?
>
> If not, what messages should go into logging.info()? Can you name me
> some examples?
ChrisA
Michael Torrie
Jan 3, 2023, 5:16:47 PM1/3/23
to
On 1/3/23 11:45, Keith Thompson wrote:
> MRAB <pyt...@mrabarnett.plus.com> writes:
> [...]
>> The purpose of stderr is to display status messages, logging and error
>> messages, even user prompts, and not mess up the program's actual
>> output. This is important on a *nix system where you might be piping
>> the output of one program into the input of another.
>
> I would expect user prompts to be written to stdout, or perhaps to some
> system-specific stream like the current tty, not to stderr. If a
> program has user prompts, it probably doesn't make sense to pipe its
> output to the input of another.
I can't think of a specific example, but I know I have piped the output
> MRAB <pyt...@mrabarnett.plus.com> writes:
> [...]
>> The purpose of stderr is to display status messages, logging and error
>> messages, even user prompts, and not mess up the program's actual
>> output. This is important on a *nix system where you might be piping
>> the output of one program into the input of another.
>
> I would expect user prompts to be written to stdout, or perhaps to some
> system-specific stream like the current tty, not to stderr. If a
> program has user prompts, it probably doesn't make sense to pipe its
> output to the input of another.
of a program while at the same time interacting with a prompt on stderr.
A rare thing, though.
Maybe some day an interface and shell syntax will be developed to
interact with an arbitrary number of standard streams. Current piping
syntax really only works well with one stream and even trying to use
stderr and stdout with pipes and filters in a shell is awkward.
2QdxY4Rz...@potatochowder.com
Jan 3, 2023, 5:20:54 PM1/3/23
to
On 2023-01-03 at 21:24:20 +0000,
that place.
> Maybe I totally misunderstood the intention of logging.info()?! Isn't
> this the "usual applicaton output"?
Aha. This seems to be the misconception. "Usual" in the sense of "the
application is running as usual," but not in the sense of "the output
the users usually see."
Logging is the process where applications write details that ordinarily
aren't visible to the user. Logs are for forensics, after the fact.
There is an entire universe of software for managing (writing, reading,
archiving, retrieving, searching, etc.) log files.
At my last job, we wrote gigabytes of compressed logs every hour, and
read roughly two or three out of a million entries to track down
questions from our users. Users wouldn't have understood 99% of what
was in the logs anyway.
I also worked a lot in embedded systems, like telephone switches. Users
(i.e., the people with the telephones) didn't have access to the
switches, but when things went wrong, I could look at the switches' logs
to get the their view of what happened.
INFO level messages usually track "regular" happy-path progress through
your application: reading config files, calculating results, writing
report. Again, not meant for users (who would have their own progress
meters, textual or graphical), but for tracking down problems when there
are problems. For one example, the name of the config file, even if you
only tell the users that you found a config file.
HTH
c.b...@posteo.jp wrote:
> The main question here is why does Python deciecded to make all logs
> go to stderr?
It makes sense to send all logging to one destination, and stderr is
> The main question here is why does Python deciecded to make all logs
> go to stderr?
that place.
> Maybe I totally misunderstood the intention of logging.info()?! Isn't
> this the "usual applicaton output"?
application is running as usual," but not in the sense of "the output
the users usually see."
Logging is the process where applications write details that ordinarily
aren't visible to the user. Logs are for forensics, after the fact.
There is an entire universe of software for managing (writing, reading,
archiving, retrieving, searching, etc.) log files.
At my last job, we wrote gigabytes of compressed logs every hour, and
read roughly two or three out of a million entries to track down
questions from our users. Users wouldn't have understood 99% of what
was in the logs anyway.
I also worked a lot in embedded systems, like telephone switches. Users
(i.e., the people with the telephones) didn't have access to the
switches, but when things went wrong, I could look at the switches' logs
to get the their view of what happened.
INFO level messages usually track "regular" happy-path progress through
your application: reading config files, calculating results, writing
report. Again, not meant for users (who would have their own progress
meters, textual or graphical), but for tracking down problems when there
are problems. For one example, the name of the config file, even if you
only tell the users that you found a config file.
HTH
Chris Angelico
Jan 3, 2023, 5:41:32 PM1/3/23
to
On Wed, 4 Jan 2023 at 09:17, Michael Torrie <tor...@gmail.com> wrote:
>
> On 1/3/23 11:45, Keith Thompson wrote:
>
> On 1/3/23 11:45, Keith Thompson wrote:
> > MRAB <pyt...@mrabarnett.plus.com> writes:
> > [...]
> >> The purpose of stderr is to display status messages, logging and error
> >> messages, even user prompts, and not mess up the program's actual
> >> output. This is important on a *nix system where you might be piping
> >> the output of one program into the input of another.
> >
> > I would expect user prompts to be written to stdout, or perhaps to some
> > system-specific stream like the current tty, not to stderr. If a
> > program has user prompts, it probably doesn't make sense to pipe its
> > output to the input of another.
>
> > [...]
> >> The purpose of stderr is to display status messages, logging and error
> >> messages, even user prompts, and not mess up the program's actual
> >> output. This is important on a *nix system where you might be piping
> >> the output of one program into the input of another.
> >
> > I would expect user prompts to be written to stdout, or perhaps to some
> > system-specific stream like the current tty, not to stderr. If a
> > program has user prompts, it probably doesn't make sense to pipe its
> > output to the input of another.
>
> I can't think of a specific example, but I know I have piped the output
> of a program while at the same time interacting with a prompt on stderr.
> A rare thing, though.
That can certainly happen, but you may more often see things that go
> of a program while at the same time interacting with a prompt on stderr.
> A rare thing, though.
directly to the TTY rather than using the standard streams at all. For
example, ssh and sudo will prompt on the terminal if they need a
password, to avoid interfering with the standard streams (which can
and often will be redirected).
> Maybe some day an interface and shell syntax will be developed to
> interact with an arbitrary number of standard streams. Current piping
> syntax really only works well with one stream and even trying to use
> stderr and stdout with pipes and filters in a shell is awkward.
to backport it into bash. You have at least ten available, with more
being possible but potentially causing conflicts:
https://www.gnu.org/software/bash/manual/html_node/Redirections.html
ChrisA
Grant Edwards
Jan 3, 2023, 5:51:52 PM1/3/23
to
>> MRAB <pyt...@mrabarnett.plus.com> writes:
>> [...]
>>> The purpose of stderr is to display status messages, logging and error
>>> messages, even user prompts, and not mess up the program's actual
>>> output. This is important on a *nix system where you might be piping
>>> the output of one program into the input of another.
>>
>> I would expect user prompts to be written to stdout, or perhaps to some
>> system-specific stream like the current tty, not to stderr. If a
>> program has user prompts, it probably doesn't make sense to pipe its
>> output to the input of another.
>
>> [...]
>>> The purpose of stderr is to display status messages, logging and error
>>> messages, even user prompts, and not mess up the program's actual
>>> output. This is important on a *nix system where you might be piping
>>> the output of one program into the input of another.
>>
>> I would expect user prompts to be written to stdout, or perhaps to some
>> system-specific stream like the current tty, not to stderr. If a
>> program has user prompts, it probably doesn't make sense to pipe its
>> output to the input of another.
>
> I can't think of a specific example, but I know I have piped the output
> of a program while at the same time interacting with a prompt on stderr.
> A rare thing, though.
Programs that ask for passwords often do it by reading/writing to fd 0
> of a program while at the same time interacting with a prompt on stderr.
> A rare thing, though.
or /dev/tty so that stdout is unaffected.
--
Grant
Chris Angelico
Jan 3, 2023, 9:26:34 PM1/3/23
to
On Wed, 4 Jan 2023 at 09:52, Grant Edwards <grant.b...@gmail.com> wrote:
>
> On 2023-01-03, Michael Torrie <tor...@gmail.com> wrote:
> > On 1/3/23 11:45, Keith Thompson wrote:
>
> On 2023-01-03, Michael Torrie <tor...@gmail.com> wrote:
> > On 1/3/23 11:45, Keith Thompson wrote:
> >> MRAB <pyt...@mrabarnett.plus.com> writes:
> >> [...]
> >>> The purpose of stderr is to display status messages, logging and error
> >>> messages, even user prompts, and not mess up the program's actual
> >>> output. This is important on a *nix system where you might be piping
> >>> the output of one program into the input of another.
> >>
> >> I would expect user prompts to be written to stdout, or perhaps to some
> >> system-specific stream like the current tty, not to stderr. If a
> >> program has user prompts, it probably doesn't make sense to pipe its
> >> output to the input of another.
> >
> >> [...]
> >>> The purpose of stderr is to display status messages, logging and error
> >>> messages, even user prompts, and not mess up the program's actual
> >>> output. This is important on a *nix system where you might be piping
> >>> the output of one program into the input of another.
> >>
> >> I would expect user prompts to be written to stdout, or perhaps to some
> >> system-specific stream like the current tty, not to stderr. If a
> >> program has user prompts, it probably doesn't make sense to pipe its
> >> output to the input of another.
> >
> > I can't think of a specific example, but I know I have piped the output
> > of a program while at the same time interacting with a prompt on stderr.
> > A rare thing, though.
>
> Programs that ask for passwords often do it by reading/writing to fd 0
> or /dev/tty so that stdout is unaffected.
>
Reading/writing the FD is the same as using stdout (technically you'd
> > of a program while at the same time interacting with a prompt on stderr.
> > A rare thing, though.
>
> Programs that ask for passwords often do it by reading/writing to fd 0
> or /dev/tty so that stdout is unaffected.
>
write to fd 1 and read from fd 0), but yes, can use /dev/tty to reach
for the console directly.
ChrisA
Chris Angelico
Jan 3, 2023, 9:28:41 PM1/3/23
to
On Wed, 4 Jan 2023 at 10:28, Stefan Ram <r...@zedat.fu-berlin.de> wrote:
> posted on a web site, maybe it was "rkachach".
Yet another shining example of bad code that does what someone asks,
but will make life harder for the next person to come along and try to
work with that program. It's like if someone asked for a way to make
True equal to -1 instead of 1, for compatibility with BASIC. Bad idea.
ChrisA
>
> c.b...@posteo.jp writes:
> >But I don't want to make all log levels go to stdout. Just DEBUG and
> >INFO.
>
> The following was stripped down from something some guy
> c.b...@posteo.jp writes:
> >But I don't want to make all log levels go to stdout. Just DEBUG and
> >INFO.
>
> posted on a web site, maybe it was "rkachach".
Yet another shining example of bad code that does what someone asks,
but will make life harder for the next person to come along and try to
work with that program. It's like if someone asked for a way to make
True equal to -1 instead of 1, for compatibility with BASIC. Bad idea.
ChrisA
Eryk Sun
Jan 3, 2023, 11:11:32 PM1/3/23
to
Except stdout may be buffered. One should probably flush the buffer
before each raw write to the file descriptor.
> use /dev/tty to reach for the console directly.
On Windows, open "CON" (read or write), "CONIN$" (read-write), or
"CONOUT$" (read-write). Or use the more explicit device paths
"\\.\CON", "\\.\CONIN$", and "\\.\CONOUT$". If the process has no
console, one can call WinAPI AllocConsole() or
AttachConsole(process_id) to obtain one.
before each raw write to the file descriptor.
> use /dev/tty to reach for the console directly.
"CONOUT$" (read-write). Or use the more explicit device paths
"\\.\CON", "\\.\CONIN$", and "\\.\CONOUT$". If the process has no
console, one can call WinAPI AllocConsole() or
AttachConsole(process_id) to obtain one.
Grant Edwards
Jan 4, 2023, 12:20:33 AM1/4/23
to
On 2023-01-04, Chris Angelico <ros...@gmail.com> wrote:
> On Wed, 4 Jan 2023 at 09:52, Grant Edwards <grant.b...@gmail.com> wrote:
>>
>>> I can't think of a specific example, but I know I have piped the output
>>> of a program while at the same time interacting with a prompt on stderr.
>>> A rare thing, though.
>>
>> Programs that ask for passwords often do it by reading/writing to
>> fd 0 or /dev/tty so that stdout is unaffected.
>
> Reading/writing the FD is the same as using stdout.
> On Wed, 4 Jan 2023 at 09:52, Grant Edwards <grant.b...@gmail.com> wrote:
>>
>>> I can't think of a specific example, but I know I have piped the output
>>> of a program while at the same time interacting with a prompt on stderr.
>>> A rare thing, though.
>>
>> Programs that ask for passwords often do it by reading/writing to
>> fd 0 or /dev/tty so that stdout is unaffected.
>
No, stdout is fd 1.
> (technically you'd write to fd 1 and read from fd 0),
the program's output). The way I remember it was that you wrote the
prompt to fd 0, and then read the password from fd 0. That way it
worked even when fd 1 (stdout) was redirected. It still failed if
stdin was redirected...
IIRC, you can't usually write to the stdin stream in C (or Python), so
you had to write(0,promptstr,strlen(promptstr)) instead of
printf("%s",promptstr) or fputs(promptstr,stdin).
> but yes, can use /dev/tty to reach for the console directly.
That's definitely a better option, but I'm pretty sure I've seen
multiple examples over the decades where fd 0 was used instead. Was
/dev/tty a "recent" enough invention that I would have seen
application code that was written before it existed? Or maybe I was
just looking at code written by people who weren't aware of /dev/tty?
--
Grant
Chris Angelico
Jan 4, 2023, 12:47:09 AM1/4/23
to
On Wed, 4 Jan 2023 at 15:11, Eryk Sun <ery...@gmail.com> wrote:
>
> On 1/3/23, Chris Angelico <ros...@gmail.com> wrote:
> >
> > writing the FD is the same as using stdout
>
> Except stdout may be buffered. One should probably flush the buffer
> before each raw write to the file descriptor.
FDs can also be buffered. If it's buffering you want to avoid, don't
>
> On 1/3/23, Chris Angelico <ros...@gmail.com> wrote:
> >
> > writing the FD is the same as using stdout
>
> Except stdout may be buffered. One should probably flush the buffer
> before each raw write to the file descriptor.
mess around with exactly which one you're writing to, just flush.
> > use /dev/tty to reach for the console directly.
>
> On Windows, open "CON" (read or write), "CONIN$" (read-write), or
> "CONOUT$" (read-write). Or use the more explicit device paths
> "\\.\CON", "\\.\CONIN$", and "\\.\CONOUT$". If the process has no
> console, one can call WinAPI AllocConsole() or
> AttachConsole(process_id) to obtain one.
Yeah, /dev/tty is the Unix option, Windows has its own way, but
> "CONOUT$" (read-write). Or use the more explicit device paths
> "\\.\CON", "\\.\CONIN$", and "\\.\CONOUT$". If the process has no
> console, one can call WinAPI AllocConsole() or
> AttachConsole(process_id) to obtain one.
whatever platform you're on, there's a way to directly reach the
console.
ChrisA
Chris Angelico
Jan 4, 2023, 12:54:16 AM1/4/23
to
On Wed, 4 Jan 2023 at 16:21, Grant Edwards <grant.b...@gmail.com> wrote:
>
> On 2023-01-04, Chris Angelico <ros...@gmail.com> wrote:
> > On Wed, 4 Jan 2023 at 09:52, Grant Edwards <grant.b...@gmail.com> wrote:
> >>
> >>> I can't think of a specific example, but I know I have piped the output
> >>> of a program while at the same time interacting with a prompt on stderr.
> >>> A rare thing, though.
> >>
> >> Programs that ask for passwords often do it by reading/writing to
> >> fd 0 or /dev/tty so that stdout is unaffected.
> >
> > Reading/writing the FD is the same as using stdout.
>
> No, stdout is fd 1.
>
> > (technically you'd write to fd 1 and read from fd 0),
I assumed that you were using a shorthand. My bad.
>
> On 2023-01-04, Chris Angelico <ros...@gmail.com> wrote:
> > On Wed, 4 Jan 2023 at 09:52, Grant Edwards <grant.b...@gmail.com> wrote:
> >>
> >>> I can't think of a specific example, but I know I have piped the output
> >>> of a program while at the same time interacting with a prompt on stderr.
> >>> A rare thing, though.
> >>
> >> Programs that ask for passwords often do it by reading/writing to
> >> fd 0 or /dev/tty so that stdout is unaffected.
> >
> > Reading/writing the FD is the same as using stdout.
>
> No, stdout is fd 1.
>
> > (technically you'd write to fd 1 and read from fd 0),
> No, the whole point is _not_ to write to stdout (which would corrupt
> the program's output). The way I remember it was that you wrote the
> prompt to fd 0, and then read the password from fd 0. That way it
> worked even when fd 1 (stdout) was redirected. It still failed if
> stdin was redirected...
TTY, and (b) it hasn't been redirected. Not a reliable assumption.
rosuav@sikorsky:~/tmp$ cat testfds.py
import os
for fd in range(3):
os.write(fd, b"This is on FD %d\n" % fd)
rosuav@sikorsky:~/tmp$ python3 testfds.py
This is on FD 0
This is on FD 1
This is on FD 2
rosuav@sikorsky:~/tmp$ echo | python3 testfds.py
Traceback (most recent call last):
File "/home/rosuav/tmp/testfds.py", line 3, in <module>
os.write(fd, b"This is on FD %d\n" % fd)
OSError: [Errno 9] Bad file descriptor
Writing to fd 0 might work if stdout is redirected, but it won't work
if stdin is redirected, so honestly, I don't think this hack is at all
reliable.
> > but yes, can use /dev/tty to reach for the console directly.
>
> That's definitely a better option, but I'm pretty sure I've seen
> multiple examples over the decades where fd 0 was used instead. Was
> /dev/tty a "recent" enough invention that I would have seen
> application code that was written before it existed? Or maybe I was
> just looking at code written by people who weren't aware of /dev/tty?
stdout redirected and goes "oh hey, writing to fd 0 still works", and
never tries it with stdin redirected. It's very common for programmers
to see that something works without knowing WHY it works :)
By the way, I can reinstate the behaviour of "FD 0 writes to the TTY"
in bash by redirecting it:
rosuav@sikorsky:~/tmp$ python3 testfds.py >/dev/null 0<>/dev/tty
This is on FD 0
This is on FD 2
which is another neat demonstration that bash has more power than you
will ever need...
(though redirecting FDs above 2 in read/write mode can be *incredibly* useful.)
ChrisA
Eryk Sun
Jan 4, 2023, 1:19:54 AM1/4/23
to
On 1/3/23, Grant Edwards <grant.b...@gmail.com> wrote:
>
> That's definitely a better option, but I'm pretty sure I've seen
> multiple examples over the decades where fd 0 was used instead. Was
> /dev/tty a "recent" enough invention that I would have seen
> application code that was written before it existed? Or maybe I was
> just looking at code written by people who weren't aware of /dev/tty?
POSIX has required "/dev/tty" for 20 years, or longer. It's in the
>
> That's definitely a better option, but I'm pretty sure I've seen
> multiple examples over the decades where fd 0 was used instead. Was
> /dev/tty a "recent" enough invention that I would have seen
> application code that was written before it existed? Or maybe I was
> just looking at code written by people who weren't aware of /dev/tty?
2004 edition of the spec, which was a minor update to the 2001
edition.
https://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap10.html
> No, the whole point is _not_ to write to stdout (which would corrupt
> the program's output). The way I remember it was that you wrote the
> prompt to fd 0, and then read the password from fd 0. That way it
> worked even when fd 1 (stdout) was redirected. It still failed if
> stdin was redirected...
files can be opened with read-write access, but it's for a different
purpose.
A console input buffer can be read from via read() or WinAPI
ReadFile(). It can also be read from using IOCTLs that work with
16-bit wide-character strings, which is the basis of WinAPI
ReadConsoleW() and ReadConsoleInputW(). A console input buffer can be
*written to* via the IOCTL-based function WriteConsoleInputW().
A console screen buffer can be written to via write() or WinAPI
WriteFile(). It can also be written to using IOCTLs that work with
16-bit wide-character strings, which is the basis of WriteConsoleW()
and WriteConsoleOutputW(). A console screen buffer can be *read from*
via the IOCTL-based function ReadConsoleOutputW().
Eryk Sun
Jan 4, 2023, 1:26:29 AM1/4/23
to
On 1/3/23, Chris Angelico <ros...@gmail.com> wrote:
>
> FDs can also be buffered. If it's buffering you want to avoid, don't
> mess around with exactly which one you're writing to, just flush.
I meant to flush a C FILE stream or Python file before writing
>
> FDs can also be buffered. If it's buffering you want to avoid, don't
> mess around with exactly which one you're writing to, just flush.
directly to the file descriptor, in order to avoid out-of-sequence and
interlaced writes that make no sense. Any OS buffering on the file
doesn't matter in this regard.
Chris Angelico
Jan 4, 2023, 1:46:58 AM1/4/23
to
OS buffering can most certainly make a difference. And it doesn't hurt
to explicitly flush when you need it flushed.
I've known some systems to have a trigger of "reading on FD 0 flushes
FD 1", but that's not going to work when you use certain
keyboard-input routines (ISTR readline was one of the few things that
*didn't* have that problem, but I may be misremembering), so the
general strategy was: print your prompt, flush stdout, THEN request
input.
ChrisA
Weatherby,Gerard
Jan 4, 2023, 7:03:27 AM1/4/23
to
Dealing with stdout / stderr is bash is just syntax. I don’t always remember it off the top of my head but … stackoverflow.com.
On Linux at least it’s possible to pipe to arbitrary streams, it seems. The following uses bash to write “Hi” to the file “third” opened by Python. I determined the file was 5 empirically.
import os
import subprocess
command= 'echo Hi >/dev/fd/5'
fd = os.open("third",os.O_WRONLY|os.O_CREAT)
os.set_inheritable(fd,True)
subprocess.run(('/usr/bin/bash','-c',command),close_fds=False)
From: Python-list <python-list-bounces+gweatherby=uchc...@python.org> on behalf of Michael Torrie <tor...@gmail.com>
Date: Tuesday, January 3, 2023 at 5:18 PM
To: pytho...@python.org <pytho...@python.org>
Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr?
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!j8qV9yx1DJI_G2F-q1fQz2LfnVYoMi40Qpk_h8bxrOcw50rVXpwScpFJSyZ212Tm9rj6T7vKgJjaIEgLRw$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!j8qV9yx1DJI_G2F-q1fQz2LfnVYoMi40Qpk_h8bxrOcw50rVXpwScpFJSyZ212Tm9rj6T7vKgJjaIEgLRw$>
On Linux at least it’s possible to pipe to arbitrary streams, it seems. The following uses bash to write “Hi” to the file “third” opened by Python. I determined the file was 5 empirically.
import os
import subprocess
command= 'echo Hi >/dev/fd/5'
fd = os.open("third",os.O_WRONLY|os.O_CREAT)
os.set_inheritable(fd,True)
subprocess.run(('/usr/bin/bash','-c',command),close_fds=False)
From: Python-list <python-list-bounces+gweatherby=uchc...@python.org> on behalf of Michael Torrie <tor...@gmail.com>
Date: Tuesday, January 3, 2023 at 5:18 PM
To: pytho...@python.org <pytho...@python.org>
Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr?
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***
Maybe some day an interface and shell syntax will be developed to
interact with an arbitrary number of standard streams. Current piping
syntax really only works well with one stream and even trying to use
stderr and stdout with pipes and filters in a shell is awkward.
--
interact with an arbitrary number of standard streams. Current piping
syntax really only works well with one stream and even trying to use
stderr and stdout with pipes and filters in a shell is awkward.
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!j8qV9yx1DJI_G2F-q1fQz2LfnVYoMi40Qpk_h8bxrOcw50rVXpwScpFJSyZ212Tm9rj6T7vKgJjaIEgLRw$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!j8qV9yx1DJI_G2F-q1fQz2LfnVYoMi40Qpk_h8bxrOcw50rVXpwScpFJSyZ212Tm9rj6T7vKgJjaIEgLRw$>
2QdxY4Rz...@potatochowder.com
Jan 4, 2023, 7:20:33 AM1/4/23
to
On 2023-01-04 at 12:02:55 +0000,
> On Linux at least it’s possible to pipe to arbitrary streams, it
> seems. The following uses bash to write “Hi” to the file “third”
> opened by Python. I determined the file was 5 empirically.
Empirically?
> import os
> import subprocess
> command= 'echo Hi >/dev/fd/5'
> fd = os.open("third",os.O_WRONLY|os.O_CREAT)
command = f"echo Hi >/dev/fd/{fd}"
Or:
command = f"echo Hi >&{fd}"
> os.set_inheritable(fd,True)
>
> subprocess.run(('/usr/bin/bash','-c',command),close_fds=False)
By the time I'm that far down that path, I usually write a separate
function to call fork, open, close, and exec myself (old habits die
hard). In the end, there's no reward for writing and maintaining shell
programs in Python.
"Weatherby,Gerard" <gweat...@uchc.edu> wrote:
> Dealing with stdout / stderr is bash is just syntax. I don’t always
> remember it off the top of my head but … stackoverflow.com.
https://tldp.org/LDP/abs/html/io-redirection.html
> Dealing with stdout / stderr is bash is just syntax. I don’t always
> remember it off the top of my head but … stackoverflow.com.
> On Linux at least it’s possible to pipe to arbitrary streams, it
> seems. The following uses bash to write “Hi” to the file “third”
> opened by Python. I determined the file was 5 empirically.
> import os
> import subprocess
> command= 'echo Hi >/dev/fd/5'
> fd = os.open("third",os.O_WRONLY|os.O_CREAT)
Or:
command = f"echo Hi >&{fd}"
> os.set_inheritable(fd,True)
>
> subprocess.run(('/usr/bin/bash','-c',command),close_fds=False)
function to call fork, open, close, and exec myself (old habits die
hard). In the end, there's no reward for writing and maintaining shell
programs in Python.
Barry Scott
Jan 4, 2023, 10:29:58 AM1/4/23
to
On 04/01/2023 02:26, Chris Angelico wrote:
> Reading/writing the FD is the same as using stdout (technically you'd
> write to fd 1 and read from fd 0), but yes, can use /dev/tty to reach
> for the console directly.
I think the logic is more like checking that stdin is a tty then using
the tty it to read and write the tty. This works with stdout redirected.
Also stdin is likely only open with read access.
For example the way to prevent ssh-add from prompting in the terminal is
this:
$ ssh-add ~/.ssh/id_rsa </dev/null
It will then use the SSH_ASKPASS program.
Barry
Barry Scott
Jan 4, 2023, 10:36:28 AM1/4/23
to
On 04/01/2023 06:46, Chris Angelico wrote:
> I've known some systems to have a trigger of "reading on FD 0 flushes
> FD 1"
Quote from https://en.cppreference.com/w/cpp/io/cin
"Once |std::cin| is constructed, std::cin.tie() returns &std::cout
<http://en.cppreference.com/w/cpp/io/cout>, and likewise,
std::wcin.tie() returns &std::wcout
<http://en.cppreference.com/w/cpp/io/cout>. This means that any
formatted input operation on |std::cin| forces a call to std::cout
<http://en.cppreference.com/w/cpp/io/cout>.flush() if any characters are
pending for output."
Barry
Peter J. Holzer
Jan 4, 2023, 11:15:18 AM1/4/23
to
On 2023-01-04 16:46:37 +1100, Chris Angelico wrote:
"stdout" as a shorthand for "standard output" which refers to file
descriptor 1, not just to the C FILE* of that name. Or Python's
sys.stdout which is a different thing again (but also is a frontend to
FD 1).
> > One should probably flush the buffer
> > before each raw write to the file descriptor.
>
to a file on a disk will not usually affect the disk, true. But it will
be immediately visible to other processes reading that file.
A write to a terminal attached via a serial cable will also be written
to a buffer first (in the OS or maybe inside the UART), but it will
be sent to the terminal as fast as the hardware allows.
So. sure, the writes are "buffered", but is that meaningful in this
context?
Reading from a terminal also often involves a buffer, but in a very
different sense: Here the OS buffers each line to implement line editing
functions. If the kernel didn't do that each application which could
possibly read from a terminal would have to implement that itself. and
it was decided back in the 1970's that this wasn't a good use of
resources. (Of course these days many programs which didn't do that
originally (like shells) do implement line editing (often via
readline).)
> If it's buffering you want to avoid, don't
> mess around with exactly which one you're writing to, just flush.
I don't think fsync() will have an effect on an FD connected to a
terminal (or a pipe or a socket).
hp
--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | h...@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"
> On Wed, 4 Jan 2023 at 15:11, Eryk Sun <ery...@gmail.com> wrote:
> > On 1/3/23, Chris Angelico <ros...@gmail.com> wrote:
> > On 1/3/23, Chris Angelico <ros...@gmail.com> wrote:
> > > writing the FD is the same as using stdout
> >
> >
> > Except stdout may be buffered.
Maybe I'm overly lax in my terminology, but I frequently use the term
"stdout" as a shorthand for "standard output" which refers to file
descriptor 1, not just to the C FILE* of that name. Or Python's
sys.stdout which is a different thing again (but also is a frontend to
FD 1).
> > One should probably flush the buffer
> > before each raw write to the file descriptor.
>
> FDs can also be buffered.
That depends on what you mean by "buffered". A write to an fd referring
to a file on a disk will not usually affect the disk, true. But it will
be immediately visible to other processes reading that file.
A write to a terminal attached via a serial cable will also be written
to a buffer first (in the OS or maybe inside the UART), but it will
be sent to the terminal as fast as the hardware allows.
So. sure, the writes are "buffered", but is that meaningful in this
context?
Reading from a terminal also often involves a buffer, but in a very
different sense: Here the OS buffers each line to implement line editing
functions. If the kernel didn't do that each application which could
possibly read from a terminal would have to implement that itself. and
it was decided back in the 1970's that this wasn't a good use of
resources. (Of course these days many programs which didn't do that
originally (like shells) do implement line editing (often via
readline).)
> If it's buffering you want to avoid, don't
> mess around with exactly which one you're writing to, just flush.
terminal (or a pipe or a socket).
hp
--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | h...@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"
Peter J. Holzer
Jan 4, 2023, 11:22:15 AM1/4/23
to
On 2023-01-03 21:24:20 +0000, c.b...@posteo.jp wrote:
> The main question here is why does Python deciecded to make all logs go to
> stderr?
It doesn't.
> The main question here is why does Python deciecded to make all logs go to
> stderr?
The Python logging system provides a plethora of handlers to log to lots
of different places. Only the StreamHandler defaults to writing to
stderr, and even that can log to any stream - just pass it as an
argument.
If none of existing handlers does what you want you can always write
your own.
Thomas Passin
Jan 4, 2023, 12:33:13 PM1/4/23
to
On 1/3/2023 10:35 AM, c.b...@posteo.jp wrote:
> The logging module write everything to stderr no matter which logging
> level is used.
The OP wrote this, but it's not so by default. There is a HOW-TO page
> level is used.
that explains this and how to get the logging package to log everything
to a file, along with many other useful bits of information -
https://docs.python.org/3/howto/logging.html
Barry Scott
Jan 4, 2023, 12:50:23 PM1/4/23
to
On 03/01/2023 21:24, c.b...@posteo.jp wrote:
> Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:
>> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
>
> But I don't want to make all log levels go to stdout. Just DEBUG and
> INFO. But this would be a workaround.
>
>
> The main question here is why does Python deciecded to make all logs
> go to stderr?
> go to stderr?
> Maybe I totally misunderstood the intention of logging.info()?! Isn't
> this the "usual applicaton output"?
>
It is up to you, the designer of an application, to decide how it works.
> this the "usual applicaton output"?
>
You will take into account conventions that your users will expect you
to follow.
If the logging module helps you then use it, but you are not forced by
logging to
design your app is a particular way. The default behavior of the logging
module is
a generally useful default, but its only a default. You are free to
setup logging to
meet your needs.
Barry
Barry Scott
Jan 4, 2023, 1:05:47 PM1/4/23
to
On 03/01/2023 21:24, c.b...@posteo.jp wrote:
> Am 03.01.2023 17:51 schrieb r...@zedat.fu-berlin.de:
>> logging.getLogger().addHandler( logging.StreamHandler( sys.stdout ))
>
> But I don't want to make all log levels go to stdout. Just DEBUG and
> INFO. But this would be a workaround.
>
> The main question here is why does Python deciecded to make all logs
> go to stderr?
> Maybe I totally misunderstood the intention of logging.info()?! Isn't
> this the "usual applicaton output"?
>
> If not, what messages should go into logging.info()? Can you name me
> some examples?
write an app that prints the contents of a file.
The application output is the contents of the file.
The logging might be this when all is working:
INFO About to open <filename>
INFO Wrote <number> bytes from <filename>
The logging might be this when there is a problem:
INFO About to open <filename>
ERROR Failed to open <filename> - <reason>
Does that help?
Barry
Chris Angelico
Jan 4, 2023, 1:45:23 PM1/4/23
to
INFO: Loading module XYZ
WARN: Unrecognized module option frobnicate=1
INFO: Using default spamination of 4
ERROR: Insufficient ham to spaminate
This tells a hypothetical (but only slightly hypothetical) story of a
module that was given an incorrect or misspelled option, and which
then failed to load as a consequence. Without the INFO lines, it would
be harder to figure out what the problem was, but they are still part
of logging, not output, and belong in the same place as the warnings
and errors.
ChrisA
Peter J. Holzer
Jan 5, 2023, 6:31:31 AM1/5/23
to
On 2023-01-04 12:32:40 -0500, Thomas Passin wrote:
> On 1/3/2023 10:35 AM, c.b...@posteo.jp wrote:
> > The logging module write everything to stderr no matter which logging
> > level is used.
>
> The OP wrote this, but it's not so by default.
By default it is - sort of.
> On 1/3/2023 10:35 AM, c.b...@posteo.jp wrote:
> > The logging module write everything to stderr no matter which logging
> > level is used.
>
> The OP wrote this, but it's not so by default.
That is all log messages go to stderr, but not all log levels are
logged.
Thomas Passin
Jan 5, 2023, 8:32:15 AM1/5/23
to
On 1/5/2023 6:27 AM, Peter J. Holzer wrote:
> On 2023-01-04 12:32:40 -0500, Thomas Passin wrote:
>> On 1/3/2023 10:35 AM, c.b...@posteo.jp wrote:
>>> The logging module write everything to stderr no matter which logging
>>> level is used.
>>
>> The OP wrote this, but it's not so by default.
>
> By default it is - sort of.
>
> That is all log messages go to stderr, but not all log levels are
> logged.
Actually, I misread something else that made me think that non-warning
> On 2023-01-04 12:32:40 -0500, Thomas Passin wrote:
>> On 1/3/2023 10:35 AM, c.b...@posteo.jp wrote:
>>> The logging module write everything to stderr no matter which logging
>>> level is used.
>>
>> The OP wrote this, but it's not so by default.
>
> By default it is - sort of.
>
> That is all log messages go to stderr, but not all log levels are
> logged.
messages would go to stdout, but I was mistaken. As it happens, the
page in the Python docs only mention stderr twice, both in the same
paragraph:
"logging.lastResort
A “handler of last resort” is available through this attribute. This is
a StreamHandler writing to sys.stderr with a level of WARNING, and is
used to handle logging events in the absence of any logging
configuration. The end result is to just print the message to sys.stderr."
Nowhere else on this page is the printed output destination mentioned,
except that it gets printed on the console, which could be either stderr
or stdout.
The logging system is so configurable that a user could set a different
destination for each level of logging. So it seems that the O.P.'s
original question about why the package's developers choose stderr for
all levels can be answered: "They didn't".
Peter J. Holzer
Jan 5, 2023, 2:18:30 PM1/5/23
to
On 2023-01-05 08:31:40 -0500, Thomas Passin wrote:
> The logging system is so configurable that a user could set a different
> destination for each level of logging. So it seems that the O.P.'s original
> question about why the package's developers choose stderr for all levels can
> be answered: "They didn't".
Which is almost exactly what I wrote in
> The logging system is so configurable that a user could set a different
> destination for each level of logging. So it seems that the O.P.'s original
> question about why the package's developers choose stderr for all levels can
> be answered: "They didn't".
<20230104162154....@hjp.at> ;-)
Thomas Passin
Jan 5, 2023, 2:42:25 PM1/5/23
to
On 1/5/2023 2:18 PM, Peter J. Holzer wrote:
> On 2023-01-05 08:31:40 -0500, Thomas Passin wrote:
>> The logging system is so configurable that a user could set a different
>> destination for each level of logging. So it seems that the O.P.'s original
>> question about why the package's developers choose stderr for all levels can
>> be answered: "They didn't".
>
> Which is almost exactly what I wrote in
> <20230104162154....@hjp.at> ;-)
Yup, two minds are better than one :)
> On 2023-01-05 08:31:40 -0500, Thomas Passin wrote:
>> The logging system is so configurable that a user could set a different
>> destination for each level of logging. So it seems that the O.P.'s original
>> question about why the package's developers choose stderr for all levels can
>> be answered: "They didn't".
>
> Which is almost exactly what I wrote in
> <20230104162154....@hjp.at> ;-)
Grant Edwards
Jan 5, 2023, 3:29:59 PM1/5/23
to
On 2023-01-05, Thomas Passin <li...@tompassin.net> wrote:
> The logging system is so configurable that...
I find it almost impossible to use unless I copy a working example I
find somewhere. ;)
I'm not at all surprised that the OP didn't understand how it
works. I've been writing Python programs for over 20 years, and it's
beyond me.
--
Grant
> The logging system is so configurable that...
I find it almost impossible to use unless I copy a working example I
find somewhere. ;)
I'm not at all surprised that the OP didn't understand how it
works. I've been writing Python programs for over 20 years, and it's
beyond me.
--
Grant
Thomas Passin
Jan 5, 2023, 4:09:34 PM1/5/23
to
I inherited a java project that uses Log4j. Talk about obscure! The
python docs are a wonder of clarity beside Log4j. So most of my Log4j
configuration has been by copy-pasta, with occasional surprises and
Log4j bug bites. Like a handler that deleted its own previous log file
when it rolled over at midnight - a "feature" that wasn't advertised.
I will say, though, that having used a bit of Log4j must have been
helpful, since much of what is in the Python logging package docs seemed
somewhat familiar and even seemed to make sense.
BTW, I was able to re-create that java project using Jython for about
90% of it. Way better!
Weatherby,Gerard
Jan 5, 2023, 4:29:03 PM1/5/23
to
logging.basicConfig()
logging.info(“Nice to know”)
logging.debug(“Details for when things are funky”)
logging.warn(“Trouble is brewing”)
From: Python-list <python-list-bounces+gweatherby=uchc...@python.org> on behalf of Grant Edwards <grant.b...@gmail.com>
Date: Thursday, January 5, 2023 at 3:31 PM
To: pytho...@python.org <pytho...@python.org>
Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr?
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***
Grant
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l6vSXQFKppEuLS0R5gYeLYaiHyVFfs2Rapqm1oPGEtvnZ5ivQyApZcnJyNTnnH9zEVY80ajNb-HfYkNwgw8fMtsnlSOT$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l6vSXQFKppEuLS0R5gYeLYaiHyVFfs2Rapqm1oPGEtvnZ5ivQyApZcnJyNTnnH9zEVY80ajNb-HfYkNwgw8fMtsnlSOT$>
logging.info(“Nice to know”)
logging.debug(“Details for when things are funky”)
logging.warn(“Trouble is brewing”)
From: Python-list <python-list-bounces+gweatherby=uchc...@python.org> on behalf of Grant Edwards <grant.b...@gmail.com>
Date: Thursday, January 5, 2023 at 3:31 PM
To: pytho...@python.org <pytho...@python.org>
Subject: Re: What should go to stdout/stderr and why Python logging write everything to stderr?
*** Attention: This is an external email. Use caution responding, opening attachments or clicking on links. ***
On 2023-01-05, Thomas Passin <li...@tompassin.net> wrote:
> The logging system is so configurable that...
I find it almost impossible to use unless I copy a working example I
find somewhere. ;)
I'm not at all surprised that the OP didn't understand how it
works. I've been writing Python programs for over 20 years, and it's
beyond me.
--
> The logging system is so configurable that...
I find it almost impossible to use unless I copy a working example I
find somewhere. ;)
I'm not at all surprised that the OP didn't understand how it
works. I've been writing Python programs for over 20 years, and it's
beyond me.
Grant
--
https://urldefense.com/v3/__https://mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l6vSXQFKppEuLS0R5gYeLYaiHyVFfs2Rapqm1oPGEtvnZ5ivQyApZcnJyNTnnH9zEVY80ajNb-HfYkNwgw8fMtsnlSOT$<https://urldefense.com/v3/__https:/mail.python.org/mailman/listinfo/python-list__;!!Cn_UX_p3!l6vSXQFKppEuLS0R5gYeLYaiHyVFfs2Rapqm1oPGEtvnZ5ivQyApZcnJyNTnnH9zEVY80ajNb-HfYkNwgw8fMtsnlSOT$>
Grant Edwards
Jan 5, 2023, 4:59:47 PM1/5/23
to
On 2023-01-05, Weatherby,Gerard <gweat...@uchc.edu> wrote:
> logging.basicConfig()
> logging.info(“Nice to know”)
> logging.debug(“Details for when things are funky”)
> logging.warn(“Trouble is brewing”)
I always seem to need something slightly more complex. Usually something like:
> logging.basicConfig()
> logging.info(“Nice to know”)
> logging.debug(“Details for when things are funky”)
> logging.warn(“Trouble is brewing”)
* Specify a log level on the command line.
* Allow logging to a file as specified on the command line.
* Optional timestamps for log messages.
I know, without doubt, that it can easily do all those things. I just
never seem to be able to figure out how unless I can find example code
to look at.
--
Grant
Thomas Passin
Jan 5, 2023, 5:32:11 PM1/5/23
to
On 1/5/2023 4:28 PM, Weatherby,Gerard wrote:
> logging.basicConfig()
> logging.info(“Nice to know”)
> logging.debug(“Details for when things are funky”)
> logging.warn(“Trouble is brewing”)
Not quite -
> logging.basicConfig()
> logging.info(“Nice to know”)
> logging.debug(“Details for when things are funky”)
> logging.warn(“Trouble is brewing”)
>>> import logging
>>> logging.basicConfig()
>>> logging.info("Nice to know")
>>> logging.debug("Details for when things are funky")
>>> logging.warn("Trouble is brewing")
<stdin>:1: DeprecationWarning: The 'warn' function is deprecated, use
>>> logging.info("Nice to know")
>>> logging.debug("Details for when things are funky")
>>> logging.warn("Trouble is brewing")
'warning' instead
WARNING:root:Trouble is brewing
Only warning level messages are displayed, unless you do some
configuration, as mentioned in the logging Howto page:
"The default level is WARNING, which means that only events of this
level and above will be tracked, unless the logging package is
configured to do otherwise."
So following the Howto:
>>> logging.basicConfig(level=logging.DEBUG)
>>> logging.info('So should this')
INFO:root:So should this
: Finally!
Not quite straightforward, though it is in the Howto. Now about those
handlers ...
Thomas Passin
Jan 5, 2023, 5:47:22 PM1/5/23
to
On 1/5/2023 4:24 PM, Stefan Ram wrote:
> You often can replace threads in tkinter by coroutines using
> asyncio when you write a replacement for the mainloop of
> tkinter that uses asyncio. Now, try to read only the official
> documentation of asyncio and tkinter and figure out only from
> this how to get such a program to work!
Cool! Can we have a link, please?
> You often can replace threads in tkinter by coroutines using
> asyncio when you write a replacement for the mainloop of
> tkinter that uses asyncio. Now, try to read only the official
> documentation of asyncio and tkinter and figure out only from
> this how to get such a program to work!
Cool! Can we have a link, please?
Thomas Passin
Jan 6, 2023, 1:05:49 AM1/6/23
to
On 1/5/2023 7:52 PM, Stefan Ram wrote:
> I do not post links, but tried to create a minimal example app.
Thanks! That's not exactly obvious, is it? Congratulations on getting
it working.
> import asyncio
> import logging
> import tkinter
>
> # This program was tested on Windows, it is experimental.
> # It should open a tkinter root window.
> # Ctrl-E should create a task. You should be able to create
> # a new task even while one task is already running.
> # Each task will end after about 10 seconds.
> # Ctrl-X should exit the process.
> # Or, use the menu "Action".
>
> # I (Stefan Ram) designed and wrote the program, but especially
> # the code in "app_async_mainloop" and "terminate_tasks" was
> # written following educational material from the World-Wide Web.
>
> class app_class( tkinter.Tk ):
> def __init__( self, *args, **kwargs ):
> self.is_running = 1_000_000
> super().__init__( *args, **kwargs )
> root = self
> root.protocol( "WM_DELETE_WINDOW", self.app_exit_macro )
> async def app_example_task( self, example_label ):
> try:
> for i in range( 10 ):
> example_label[ 'text' ]=str( i )
> await asyncio.sleep( 1 )
> except asyncio.CancelledError:
> pass
> example_label.destroy()
> def app_example_macro( self ):
> root = self
> example_label=tkinter.Label( root )
> example_label.pack()
> asyncio.get_running_loop().create_task\
> ( self.app_example_task( example_label ))
> root = self
> def terminate_tasks( self ):
> loop = asyncio.get_running_loop()
> pending = asyncio.all_tasks( loop=loop )
> label_tasks = []
> for task in pending:
> if 'app_example_task' in str( task ):
> label_tasks.append( task )
> task.cancel()
> group = asyncio.gather( *label_tasks, return_exceptions=True )
> try:
> loop.run_until_complete( group )
> except AssertionError:
> # ignoring an assertion error on Windows I do not understand
> pass
> def app_exit_macro( self ):
> self.terminate_tasks()
> self.is_running = 99
> root = self
> root.destroy()
> def app_add_basemenu( self, example=False ):
> root = self
> menubar = tkinter.Menu( root )
> menu = tkinter.Menu( menubar, tearoff=0 )
> name = "Actions"; menu = tkinter.Menu( menubar, tearoff=0 )
> if example:
> text = "Example";
> callback = self.app_example_macro; menu.add_command\
> ( label=text, underline=0, command=callback,
> accelerator="Control-e" );
> root.bind\
> ( "<Control-e>", lambda event, callback=callback:callback() )
> text = "Exit";
> callback = self.app_exit_macro
> menu.add_command\
> ( label=text, underline=1,
> command=callback, accelerator="Control-x" )
> root.bind\
> ( "<Control-x>", lambda event,callback=callback: callback() )
> menubar.add_cascade( label=name, underline=0, menu=menu )
> root.config( menu=menubar )
> async def app_async_mainloop( self ):
> root = self
> root.willdispatch()
> root.tk.dooneevent( tkinter._tkinter.DONT_WAIT )
> while self.is_running > 0:
> if self.is_running < 1_000_000: self.is_running -= 1
> try:
> await asyncio.sleep\
> ( tkinter._tkinter.getbusywaitinterval() / 10_000 )
> root.tk.dooneevent( tkinter._tkinter.DONT_WAIT )
> root.update()
> except tkinter.TclError:
> break
> async def app_async_main( self ):
> try:
> await self.app_async_mainloop()
> except asyncio.CancelledError:
> logging.debug( f'asyncio.CancelledError' )
> def app_async_run( self ):
> asyncio.run( self.app_async_main() )
>
> app = app_class()
> app.app_add_basemenu(example=True)
> app.app_async_run()
>
>
>
Thanks! That's not exactly obvious, is it? Congratulations on getting
it working.
> import asyncio
> import logging
> import tkinter
>
> # This program was tested on Windows, it is experimental.
> # It should open a tkinter root window.
> # Ctrl-E should create a task. You should be able to create
> # a new task even while one task is already running.
> # Each task will end after about 10 seconds.
> # Ctrl-X should exit the process.
> # Or, use the menu "Action".
>
> # I (Stefan Ram) designed and wrote the program, but especially
> # the code in "app_async_mainloop" and "terminate_tasks" was
> # written following educational material from the World-Wide Web.
>
> class app_class( tkinter.Tk ):
> def __init__( self, *args, **kwargs ):
> self.is_running = 1_000_000
> super().__init__( *args, **kwargs )
> root = self
> root.protocol( "WM_DELETE_WINDOW", self.app_exit_macro )
> async def app_example_task( self, example_label ):
> try:
> for i in range( 10 ):
> example_label[ 'text' ]=str( i )
> await asyncio.sleep( 1 )
> except asyncio.CancelledError:
> pass
> example_label.destroy()
> def app_example_macro( self ):
> root = self
> example_label=tkinter.Label( root )
> example_label.pack()
> asyncio.get_running_loop().create_task\
> ( self.app_example_task( example_label ))
> root = self
> def terminate_tasks( self ):
> loop = asyncio.get_running_loop()
> pending = asyncio.all_tasks( loop=loop )
> label_tasks = []
> for task in pending:
> if 'app_example_task' in str( task ):
> label_tasks.append( task )
> task.cancel()
> group = asyncio.gather( *label_tasks, return_exceptions=True )
> try:
> loop.run_until_complete( group )
> except AssertionError:
> # ignoring an assertion error on Windows I do not understand
> pass
> def app_exit_macro( self ):
> self.terminate_tasks()
> self.is_running = 99
> root = self
> root.destroy()
> def app_add_basemenu( self, example=False ):
> root = self
> menubar = tkinter.Menu( root )
> menu = tkinter.Menu( menubar, tearoff=0 )
> name = "Actions"; menu = tkinter.Menu( menubar, tearoff=0 )
> if example:
> text = "Example";
> callback = self.app_example_macro; menu.add_command\
> ( label=text, underline=0, command=callback,
> accelerator="Control-e" );
> root.bind\
> ( "<Control-e>", lambda event, callback=callback:callback() )
> text = "Exit";
> callback = self.app_exit_macro
> menu.add_command\
> ( label=text, underline=1,
> command=callback, accelerator="Control-x" )
> root.bind\
> ( "<Control-x>", lambda event,callback=callback: callback() )
> menubar.add_cascade( label=name, underline=0, menu=menu )
> root.config( menu=menubar )
> async def app_async_mainloop( self ):
> root = self
> root.willdispatch()
> root.tk.dooneevent( tkinter._tkinter.DONT_WAIT )
> while self.is_running > 0:
> if self.is_running < 1_000_000: self.is_running -= 1
> try:
> await asyncio.sleep\
> ( tkinter._tkinter.getbusywaitinterval() / 10_000 )
> root.tk.dooneevent( tkinter._tkinter.DONT_WAIT )
> root.update()
> except tkinter.TclError:
> break
> async def app_async_main( self ):
> try:
> await self.app_async_mainloop()
> except asyncio.CancelledError:
> logging.debug( f'asyncio.CancelledError' )
> def app_async_run( self ):
> asyncio.run( self.app_async_main() )
>
> app = app_class()
> app.app_add_basemenu(example=True)
> app.app_async_run()
>
>
>
Peter J. Holzer
Jan 6, 2023, 1:56:57 PM1/6/23
to
On 2023-01-05 12:29:33 -0800, Grant Edwards wrote:
> On 2023-01-05, Thomas Passin <li...@tompassin.net> wrote:
>
> > The logging system is so configurable that...
>
> I find it almost impossible to use unless I copy a working example I
> find somewhere. ;)
I usually copy the working example from my previous project ;-).
> On 2023-01-05, Thomas Passin <li...@tompassin.net> wrote:
>
> > The logging system is so configurable that...
>
> I find it almost impossible to use unless I copy a working example I
> find somewhere. ;)
I think the general structure is clear enough. You need a formatter to
format your log messages, a handler to actually write them and finally a
logger to determine what goes where. I can look up the details in the
docs but generally the logging is always the same (except for path names
and log levels), so I can just copy the config from last time and adjust
those.
So I might have a config.py like this:
# ...
logging = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"standard": {
"format": "%(asctime)s %(levelname)s %(name)s %(funcName)s %(lineno)d | %(message)s"
}
},
"handlers": {
"file": {
"class": "switchinglogfilehandlers.TimeoutSwitchingFileHandler",
"formatter": "standard",
"filename": "/var/log/www/XXX.hjp.at/XXX.",
},
},
"loggers": {
"": {
"handlers": ["file"],
"level": "INFO"
}
}
}
# ...
And then my application would start like this:
import logging
import logging.config
import config
logging.config.dictConfig(config.logging)
log = logging.getLogger(__name__)
Plus typically every other source file contains
import logging
log = logging.getLogger(__name__)
somewhere near the start.
Then I can just scatter my log.debug(...) or whatever whereever I want.
When I decide that I need debug output from one module, I'll just add a
logger. Or if some library is too chatty I'll add another logger to shut
it up - no programming, just a few extra lines in a dict.
(Instead of a config.py I might use a json file, especially if I expect
the config to change often.)
> I'm not at all surprised that the OP didn't understand how it
> works.
is very similar, although I find Python logging easier to use (but then
I never did much Java programming so that's probably just a matter of
familiarity.
hp
PS: The TimeoutSwitchingFileHandler mentioned above is one I wrote
myself. You can find it on PyPI, but be warned that the
documentation is (still) quite lacking.
0 new messages