info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Re: Defensive programming
8 views
Skip to first unread message
Dmitry A. Kazakov
Sep 23, 2022, 2:51:54 PM9/23/22
to
On 2022-09-23 19:57, Stefan Ram wrote:
> I wrote a small Python script to append a line to the end
> of a log file. This script is invoked from a VBA subroutine
> to append a line to that log file.
>
> In the same VBA subroutine, I then read the last line from
> the same log file and compare it with the line to be written.
> If they do not agree, the VBA subroutine opens a new message
> box informing the user.
That is an awful idea IMO. With disastrous performance as it would block
the logging file for a long time.
The key point is that logging is secondary to functionality. Thus
whatever might happen with the logs, that shall not disrupt normal
operation.
For example, in a real-time application logger queues messages instead
of writing them. Then a lower priority task flushes the log messages
queue to the disk in background.
> This might sound overly defensive, but in fact it is not
> infrequent that I happen to see that dialog! Reasons might include
> problems with the character encoding, full disks, or recently,
> a possible spurious error message for reasons (yet) unknown
> (maybe due to an overload of the system, which led to timeouts).
You should simply flush the log file periodically (keeping it open of
course).
Modern file systems do caching, mirroring, journaling,
"never-overwrite-same-block" stuff. You cannot do any better from a VBA
application! You even cannot know if the stuff you read back was
actually physically written down to the physical medium and not sitting
in some N-th order cache of some M-th controller in between... (Mission
critical logger solutions like blackboxes may use special hardware
write-once mediums etc)
--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de
> I wrote a small Python script to append a line to the end
> of a log file. This script is invoked from a VBA subroutine
> to append a line to that log file.
>
> In the same VBA subroutine, I then read the last line from
> the same log file and compare it with the line to be written.
> If they do not agree, the VBA subroutine opens a new message
> box informing the user.
That is an awful idea IMO. With disastrous performance as it would block
the logging file for a long time.
The key point is that logging is secondary to functionality. Thus
whatever might happen with the logs, that shall not disrupt normal
operation.
For example, in a real-time application logger queues messages instead
of writing them. Then a lower priority task flushes the log messages
queue to the disk in background.
> This might sound overly defensive, but in fact it is not
> infrequent that I happen to see that dialog! Reasons might include
> problems with the character encoding, full disks, or recently,
> a possible spurious error message for reasons (yet) unknown
> (maybe due to an overload of the system, which led to timeouts).
You should simply flush the log file periodically (keeping it open of
course).
Modern file systems do caching, mirroring, journaling,
"never-overwrite-same-block" stuff. You cannot do any better from a VBA
application! You even cannot know if the stuff you read back was
actually physically written down to the physical medium and not sitting
in some N-th order cache of some M-th controller in between... (Mission
critical logger solutions like blackboxes may use special hardware
write-once mediums etc)
--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de
Dmitry A. Kazakov
Sep 23, 2022, 3:25:35 PM9/23/22
to
On 2022-09-23 20:58, Stefan Ram wrote:
> So to append a line, I always open the file for appending, write
> the line, and close it immediately.
Keep it open. Flush after each write since it is infrequent.
If the file was opened for shared access then after flush you should be
able to read the file from outside, e.g. from Notepad++ etc.
> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:
>> You should simply flush the log file periodically (keeping it open of
>> course).
>
> This log file gets infrequent writes. Not more than 1000 per day.
>> You should simply flush the log file periodically (keeping it open of
>> course).
>
> So to append a line, I always open the file for appending, write
> the line, and close it immediately.
Keep it open. Flush after each write since it is infrequent.
If the file was opened for shared access then after flush you should be
able to read the file from outside, e.g. from Notepad++ etc.
Dmitry A. Kazakov
Sep 23, 2022, 3:31:54 PM9/23/22
to
On 2022-09-23 21:21, Stefan Ram wrote:
Then you need something like a database to manage transactions to the
log in order to keep in consistent and properly interlock. If the file
is opened for shared write, you will get a mess. If not, you will fail
to open a locked file. The behavior will depend on the OS. E.g. Linux
does not even have a proper locking mechanism.
P.S. It is a bad idea anyway. Each process should have its own log file.
Supply messages with a time stamp if you want to be able to generate
some combined report later.
> r...@zedat.fu-berlin.de (Stefan Ram) writes:
>> This log file gets infrequent writes. Not more than 1000 per day.
>> So to append a line, I always open the file for appending, write
>> the line, and close it immediately.
>
> But several different processes could append to that log file.
>> This log file gets infrequent writes. Not more than 1000 per day.
>> So to append a line, I always open the file for appending, write
>> the line, and close it immediately.
>
Then you need something like a database to manage transactions to the
log in order to keep in consistent and properly interlock. If the file
is opened for shared write, you will get a mess. If not, you will fail
to open a locked file. The behavior will depend on the OS. E.g. Linux
does not even have a proper locking mechanism.
P.S. It is a bad idea anyway. Each process should have its own log file.
Supply messages with a time stamp if you want to be able to generate
some combined report later.
Dmitry A. Kazakov
Sep 23, 2022, 3:33:27 PM9/23/22
to
On 2022-09-23 21:30, Stefan Ram wrote:
> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:
>> Keep it open. Flush after each write since it is infrequent.
>
> What's the problem this is supposed to be the solution to?
> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:
>> Keep it open. Flush after each write since it is infrequent.
>
That upon system crash you won't lose all file, since it is never closed.
Dmitry A. Kazakov
Sep 23, 2022, 3:35:52 PM9/23/22
to
On 2022-09-23 21:29, Stefan Ram wrote:
> r...@zedat.fu-berlin.de (Stefan Ram) writes:
>> But several different processes could append to that log file.
>
> Including batch files using ">>".
> r...@zedat.fu-berlin.de (Stefan Ram) writes:
>> But several different processes could append to that log file.
>
Then mess is guaranteed! Batch files are perfect at this... (:-))
Dmitry A. Kazakov
Sep 23, 2022, 3:50:13 PM9/23/22
to
On 2022-09-23 21:41, Stefan Ram wrote:
> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:
>>> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:
>>>> Keep it open. Flush after each write since it is infrequent.
>>> What's the problem this is supposed to be the solution to?
>> That upon system crash you won't lose all file, since it is never closed.
>
> That's why I always keep it open as short as possible;
> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:
>>> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> writes:
>>>> Keep it open. Flush after each write since it is infrequent.
>>> What's the problem this is supposed to be the solution to?
>> That upon system crash you won't lose all file, since it is never closed.
>
> I immediately close it after every append. I think, ">>"
> does this too. It might be open for append access few
> seconds per day.
This explains well the problems you experience.
0 new messages