Reading a poor man's live feed

[Luc]

There is a file being constantly rewritten at /dev/shm that I need to
read from Tcl. Likely many times per second. I don't want it to miss
a beat.

My first idea:

while 1 {
set fp [open file r]
set content [read $fp
close $fp
}

Somehow I suspect this is a bad idea. Probably inefficient or maybe
even dangerous (due to lock ups)?

What do you think? Is there a better way?

--
Luc
>>

[pd]

El martes, 6 de febrero de 2024 a las 23:55:33 UTC+1, Luc escribió:

> Somehow I suspect this is a bad idea. Probably inefficient or maybe
> even dangerous (due to lock ups)?
>
> What do you think? Is there a better way?

Maybe a file event be a better way

[Rich]

Luc <l...@sep.invalid> wrote:
> There is a file being constantly rewritten at /dev/shm that I need to
> read

This is not going to work out well in the end. Sometimes you will get
the contents, sometimes you'll get an empty file, sometimes you may get
partial contents. And every now and then, seemingly at random, you'll
find there is no file present at all.

> from Tcl. Likely many times per second. I don't want it to miss
> a beat.

Yeah, good luck with that (many times a second). You'd have a hard
time keeping consistency with a speedy C program, but you'll very
likely quickly find this is a bad idea overall, and the overhead of the
Tcl interpreter will not help.

> My first idea:
>
> while 1 {
> set fp [open file r]
> set content [read $fp
> close $fp
> }
>
> Somehow I suspect this is a bad idea.

It is.

> Probably inefficient

It is.

> or maybe even dangerous (due to lock ups)?

Under linux you won't encounter lockups, but you will encounter all
possibilibites from "not there" up to missing some data.

Under windows you'll run smack into windows mandatory file locking and
encounter either lockups or outright refusal to open.

> What do you think? Is there a better way?

The only way for us to formulate an answer to that question is for you
to tell us what you are really trying to do, instead of this Rube
Goldberg machine you've presently described.

[Luc]

On Wed, 7 Feb 2024 02:06:15 -0000 (UTC), Rich wrote:

>Under linux you won't encounter lockups, but you will encounter all
>possibilibites from "not there" up to missing some data.

I've been watching the "Rube Goldberg machine" as you describe it :-)
and never had a case of incomplete data. But missing file quite a few
times. I had to wrap catch around it.

But the machine froze and became unresponsive on two occasions. I had
to hard reset!

>> What do you think? Is there a better way?
>
>The only way for us to formulate an answer to that question is for you
>to tell us what you are really trying to do, instead of this Rube
>Goldberg machine you've presently described.

It's a Windows application running on Wine. It can log torrents of data
and I want to monitor that data, possibly triggering alarms. It's the
only way I can make it communicate with Tcl. At least I'm not making it
write to disk.

Would a Linux socket be more suitable? With Tcl socket capabilities?


--
Luc
>>

[Rich]

Does it (the windows app) actually 'rewrite' the file (as in delete and
create another, or rewind to zero length and start writing again)?

If yes, you don't have a lot of choices, the 'rewriting' part is going
to cause you all kinds of pain.

> Would a Linux socket be more suitable? With Tcl socket capabilities?

If it is a 'log file' in the usual sense (opens, then a constant
torrent of writes occur) you might be able to point it at a named pipe
(read the 'man fifo' man-page) then open the named pipe in Tcl and just
read out the data it writes.

Alternately, if it won't write to a fifo then let it log to a disk file
somewhere and just use "tail -f" on the log file to ship the contents
into Tcl. I.e.:

set fd [open {|tail -f /tmp/logfile} RDONLY]

while {[gets $fd line] > -1} {
#do something with $line
}

Or setup file events on $fd and let the event loop trigger when the
file has data to read.

[rene]

You could try "tko::file::tail" from https://chiselapp.com/user/rene/repository/tko/file?name=library/tko_file.tcl&ci=b597f7efa44e7ad53e11018f743e183b6cf69c1a9c2bcd0ef2d449b25cb480d0

Rene

[Luc]

On Wed, 7 Feb 2024 04:37:29 -0000 (UTC), Rich wrote:

>Does it (the windows app) actually 'rewrite' the file (as in delete and
>create another, or rewind to zero length and start writing again)?

I don't know. Is there a way to find out?

I've seen my Tcl script error out (before I added 'catch') complaining
that the file was not found so I guess it is deleted and rewritten from
scracth every time. But it's just my guess.

> set fd [open {|tail -f /tmp/logfile} RDONLY]
>
> while {[gets $fd line] > -1} {
> #do something with $line
> }

That code doesn't work. No output. I only get output with my previous
approach:

proc every {ms body} {
eval $body
after $ms [info level 0]
}

every 1 {
catch {p.loop}
}

proc p.loop {} {
if {![file exists $::feedfile]} {return}
set fp [open $::feedfile r]
set data [read $fp]
puts $data
}

--
Luc
>>

[Ralf Fassel]

* Luc <l...@sep.invalid>

| proc p.loop {} {
| if {![file exists $::feedfile]} {return}
| set fp [open $::feedfile r]

Note that there still is a chance that [file exists] returns true and
the [open] failing due to ENOENT (file has been removed between the two
calls).

A more robust approach is to simply open the file and handle ENOENT, see
the 'try' manpage for an example:

Handle different reasons for a file to not be openable for reading:

try {
set f [open /some/file/name w]
} trap {POSIX EISDIR} {} {
puts "failed to open /some/file/name: it's a directory"
} trap {POSIX ENOENT} {} {
puts "failed to open /some/file/name: it doesn't exist"
}

HTH
R'

[Rich]

Luc <l...@sep.invalid> wrote:
> On Wed, 7 Feb 2024 04:37:29 -0000 (UTC), Rich wrote:
>
>>Does it (the windows app) actually 'rewrite' the file (as in delete and
>>create another, or rewind to zero length and start writing again)?
>
> I don't know. Is there a way to find out?

You could strace wine while the windows app is logging and see what
file access calls are being made.

> I've seen my Tcl script error out (before I added 'catch')
> complaining that the file was not found so I guess it is deleted and
> rewritten from scracth every time. But it's just my guess.
>
>> set fd [open {|tail -f /tmp/logfile} RDONLY]
>>
>> while {[gets $fd line] > -1} {
>> #do something with $line
>> }
>
> That code doesn't work. No output. I only get output with my previous
> approach:

That implies a delete and recreate approach (which is an odd way to do
it, but it /is/ a windows program, so....), which will be *much* harder
for you to keep up with.

[Luc]

On Thu, 8 Feb 2024 19:17:34 -0000 (UTC), Rich wrote:

>> That code doesn't work. No output. I only get output with my previous
>> approach:
>
>That implies a delete and recreate approach (which is an odd way to do
>it, but it /is/ a windows program, so....), which will be *much* harder
>for you to keep up with.

I was going to give up because it's just an experiment, but something
odd is happening and I would like to understand it.


set feedfile "/dev/shm/myfeed.txt"
set fifofile "/dev/shm/myfeed.fifo"
exec tail -f $feedfile > $fifofile &

(build Tk GUI with text widget to display the data)

while {[gets $fd line] > -1} {

puts $line
return

(text widget insertion here, not in use yet, return aborts it)
}

I get no output. Nothing happens.

Then I go to the directory where the fifo file is in a regular shell
terminal:

$ cat myfeed.fifo

After running that command, I get all the output (ok, makes sense) but:

1. Only in the shell terminal, not in the IDE's compiler output pane
which usually captures the output.

2. AND the Tk GUI with the text widget shows up on the screen. Empty,
but visible.

Why?


Moreover, if I remove the 'return' line, then the IDE captures the
output (because of the 'puts' line), but the Tk GUI never shows up.

Again, why?


--
Luc
>>

[Rich]

Luc <l...@sep.invalid> wrote:
> On Thu, 8 Feb 2024 19:17:34 -0000 (UTC), Rich wrote:
>
>>> That code doesn't work. No output. I only get output with my previous
>>> approach:
>>
>>That implies a delete and recreate approach (which is an odd way to
>>do it, but it /is/ a windows program, so....), which will be *much*
>>harder for you to keep up with.
>
> I was going to give up because it's just an experiment, but something
> odd is happening and I would like to understand it.
>
>
> set feedfile "/dev/shm/myfeed.txt"
> set fifofile "/dev/shm/myfeed.fifo"
> exec tail -f $feedfile > $fifofile &

If using tail to copy from the wine created 'feed' to the fifo works
(you say it does when you indicate you 'cat' the fifo below) then you
should just be able to open the feed file and read/gets from it.

> (build Tk GUI with text widget to display the data)
>
> while {[gets $fd line] > -1} {
> puts $line
> return
>
> (text widget insertion here, not in use yet, return aborts
> it)
> }
>
> I get no output. Nothing happens.

That while loop above only ever runs for one iteration because of the
'return'. I suspect you meant to use 'continue' not 'return'.

> Then I go to the directory where the fifo file is in a regular shell
> terminal:
>
> $ cat myfeed.fifo
>
> After running that command, I get all the output (ok, makes sense)
> but:
>
> 1. Only in the shell terminal, not in the IDE's compiler output pane
> which usually captures the output.

If the code you posted here is the same you are running in the IDE,
then only running the while for a singe gets/puts iteration is why.

> 2. AND the Tk GUI with the text widget shows up on the screen. Empty,
> but visible.
>
> Why?

Because the return terminates the while loop.

> Moreover, if I remove the 'return' line, then the IDE captures the
> output (because of the 'puts' line), but the Tk GUI never shows up.
>
> Again, why?

Because the while loop never lets the event loop run. With the event
loop paused no Tk GUI will ever show up.