Little script

[jonny]

Hello all,

I have this simple eggdrop tcl script:

bind msg - "hello" hello_response

proc hello_response {nick uhost handle text} {

putserv "privmsg $nick :Hello !"
}


This script send a reply every time it receve "hello".
I want it reply to $nick only one time.

How may I do it ?

Thx

[heinrichmartin]

On Wednesday, May 8, 2019 at 10:41:27 PM UTC+2, jonny wrote:
> This script send a reply every time it receve "hello".
> I want it reply to $nick only one time.
>
> How may I do it ?

Two approaches cross my mind immediately:
* A well-known (global or namespace) variable that keeps track of whether it is the first time. Criterion could be existence or the value.
* Use a coroutine for the callback.

[Rich]

This works, but be aware that in order to keep track, the size of this
variable must grow (each new nick adds a little bit of memory usage in
order to track 'firstness'). So if the process is long running, it
could eventually exhaust memory.

> * Use a coroutine for the callback.

How does using a coroutine help with tracking "this nick has been seen
this before"?

[heinrichmartin]

On Thursday, May 9, 2019 at 1:15:10 PM UTC+2, Rich wrote:

> heinrichmartin wrote:
> > On Wednesday, May 8, 2019 at 10:41:27 PM UTC+2, jonny wrote:
> >> This script send a reply every time it receve "hello".
> >> I want it reply to $nick only one time.
> >>
> >> How may I do it ?
> >
> > Two approaches cross my mind immediately:
>
> > * A well-known (global or namespace) variable that keeps track of
> > whether it is the first time. Criterion could be existence or the
> > value.
>
> This works, but be aware that in order to keep track, the size of this
> variable must grow (each new nick adds a little bit of memory usage in
> order to track 'firstness'). So if the process is long running, it
> could eventually exhaust memory.

I missed that the same binding serves multiple nicks. But on the other hand, you are looking for session management; so you cannot avoid growing data. (Implement session cleanup for long-running services.)

You can use tcllib's struct::set or an array (or a dict) for tracking.

> > * Use a coroutine for the callback.
>
> How does using a coroutine help with tracking "this nick has been seen
> this before"?

It's just a container for the variable in this case. If it was one coroutine per session, one could yield
1. on creation (i.e. before the binding),
2. before the hello, and
3. in an empty loop thereafter.

[onem...@gmail.com]

Without going to the overhead of a coroutine, the simplest method is a global array or dict. Using an array, it would look something like:

proc hello_response {nick uhost handle text} {

global seen
if {![info exists seen($nick)]} {
set seen($nick) true

putserv "privmsg $nick :Hello !"
}

# If you want to do something different if it's not the first time,
# it goes in the else clause here.
}

Note that it must be global (or static in some other namespace) to persist across the multiple invocations for each message.

[Bezoar]

One way not mentioned it just to redefine hello_response and append a default noreply list to the argument list like so :

#!/bin/sh
# the next line restarts using wish \
exec /opt/usr8.6b.2/bin/tclsh8.6 "$0" ${1+"$@"}

proc putserv { msg } {
puts "$msg"
}

proc hello_response { nick uhost handle text { noreply {} } } {
set body [ info body hello_response ]
#puts "noreply: $noreply"
if { $nick ni $noreply } {

putserv "privmsg $nick :Hello !"

lappend noreply $nick
uplevel #0 [ subst -nocommands {
proc ::hello_response \{ nick uhost handle test \{ noreply \{ $noreply \} \} \} \{
$body
\}
} ]
}
}
hello_response "chuck" x fd0 "text"
hello_response "chucky" x fd0 "text"
hello_response "chuckie" x fd0 "text"
hello_response "chuck" x fd0 "text"
hello_response "chucky" x fd0 "text"
hello_response "chuck" x fd0 "text"
hello_response "chuckles" x fd0 "text"

##################################
# OUTPUT
#privmsg chuck :Hello !
#privmsg chucky :Hello !
#privmsg chuckie :Hello !
#privmsg chuckles :Hello !

[heinrichmartin]

On Monday, May 13, 2019 at 4:13:50 AM UTC+2, Bezoar wrote:
> One way not mentioned it just to redefine hello_response and append a default noreply list to the argument list like so :

Rewriting the proc or rebinding the event (to no-op) is a very good pattern if we are looking at separate sessions per nick. "Hiding" the data in the default value of a parameter is a nice showcase, but I'd prefer a variable for code clarity. Data is still eating memory and can hardly be cleaned up.

[two...@gmail.com]

Is memory really an issue here?

I have a tcl/tk program running on a rasp pi that monitors connections and throughput on my home router. It writes to a console window and has an ongoing stip-chart in another window. It does a couple http connections to the router every 10 seconds to grab the stats page.

This has been running now for 48 days and my PI doesn't have much memory. I see no signs that memory is a concern.

Perhaps I don't understand the OP's application. How long will it run between restarts of the program?

[Rich]

two...@gmail.com wrote:
> Is memory really an issue here?

Yes.

> Perhaps I don't understand the OP's application.

The OP proposed a system that remembers every nickname it sees in an
IRC channel.

Absent additional information, a busy IRC channel will generate a
constant stream of new nicknames to remember. Each new nickname
remembered adds to the amount of memory used by the Tcl script. The
growth is slow (as IRC nicknames are generally on the shorter end of
the scale), but this is an ever increasing memory use situation (even if
it never exceeds the capacity of the RAM of the computer upon which it
runs).

> How long will it run between restarts of the program?

Only the OP can answer that. But, remembering *all* nicknames in a busy
IRC channel for a long enough length of time would use up all memory in
whatever system the script is running.

[two...@gmail.com]

On Monday, May 13, 2019 at 12:21:44 PM UTC-7, Rich wrote:

>
> Only the OP can answer that. But, remembering *all* nicknames in a busy
> IRC channel for a long enough length of time would use up all memory in
> whatever system the script is running.

I guess it depends on how many new sign ons will be occurring.

I've got a program to solve wheel of fortune puzzles, it loads 3
dictionaries, of 500k, 144k, and 80k words - last one with definitions.
It all loads up into several tcl arrays does several sorts and it
all runs on my $80 phone under androwish.

I'm pretty amazed at how well tcl manages memory. It's hard to imagine
that monitoring an irc channel will result in more array settings than
my word solving program.

Maybe more info from the OP would be interesting.

[Rich]

Run this loop in a new Tcl process:

while {true} {
set array([incr i]) $i
}

And watch the memory used by the process as it runs.

The OP's wish (and all the proposed solutions) are simply a /slower/
growing version of the loop above. It is a design with an ever
increasing memory usage footprint.

Reality is that whether the ever increasing memory usage is an issue
depends up how busy the IRC channel is, and how long the OP plans to
leave it running. They may never leave it running long enough to ever
be an issue, but that does not change the fact that the basic design is
one of ever increasing memory usage.

> Maybe more info from the OP would be interesting.

Yes, we are working on incomplete information from the OP.

[two...@gmail.com]

On Monday, May 13, 2019 at 2:02:06 PM UTC-7, Rich wrote:

> Run this loop in a new Tcl process:
>
> while {true} {
> set array([incr i]) $i
> }

Ok, I've got a lot of time on my hands :)

I tried it and then also with the below code,

windows 10, 32 bit tcl 16,000,000 stopped with dialog
windows 10, 64 bit tcl 73,000,000 killed it, too slow
rasp pi raspian 1 gig ram 6,000,000 hung, had to reset

with longer index (as below) and longer values

windows 10, 32 bit tcl 10,000,000 stopped with dialog
windows 10, 64 bit tcl 42,000,000 slowed by 60x

64 bit win10 kept going, but got 60x slower
30 seconds to do a million, where first few took 1/2 second
(7g working set, 20g virtual size, pagefile on ssd)



while {true} {
set array(abcdefghijklmn[incr i]) abcdefghijklmn$i
if { [expr {( ($i % 1000000) == 0 )}]} {
puts "i = $i size = [array size array] [clock format [clock seconds]]"
update
after 50

}
}

[two...@gmail.com]

I concede that using an array will grow memory.

How about the following approach then. I don’t know
how to do this in tcl, but in C, I would:

Allocate a large bit array, say 1gb or 8 gbit. Init all to 0.

Hash each nick as you get it into that 8gb space, read the bit,
using modulo 8g where needed.

If the bit was set don't reply, else set the bit to 1 and reply.

Of course more than 1 nick hashing to the same thing and you’ll miss sending
out a reply.

Only the OP knows if this is acceptable. There won’t
be any extra replys however.

So, with 8 gbit, if I did my math right, and we want to cover
100 million nicks, then the likihood of missing a reply would be
about 1 in 800.

[heinrichmartin]

On Tuesday, May 14, 2019 at 7:31:55 AM UTC+2, two...@gmail.com wrote:
> How about the following approach then. I don’t know
> how to do this in tcl, but in C, I would:
>
> Allocate a large bit array, say 1gb or 8 gbit. Init all to 0.

Not at script level (https://www.tcl-lang.org/cgi-bin/tct/tip/418.html), but in the C interface: https://www.tcl-lang.org/man/tcl/TclLib/ByteArrObj.htm

> Hash each nick

This can be used in any other strategy to define the range of keys. The hashes can still be stored in an array/dict with session data; or in a bitfield as you suggested.

I agree we are missing information, but my guts feeling suggests to prepare for more data (e.g. count messages per nick).

Btw, if we are really looking at a 24/7 service and session data is a factor, then OP could even consider using the disk (db, file, ...) optionally along with a cache.

[Rich]

two...@gmail.com wrote:
> I concede that using an array will grow memory.
>

> How about the following approach then. I don?t know

> how to do this in tcl, but in C, I would:
>
> Allocate a large bit array, say 1gb or 8 gbit. Init all to 0.
>
> Hash each nick as you get it into that 8gb space, read the bit,
> using modulo 8g where needed.
>
> If the bit was set don't reply, else set the bit to 1 and reply.
>

> Of course more than 1 nick hashing to the same thing and you?ll miss sending
> out a reply.
>
> Only the OP knows if this is acceptable. There won?t

> be any extra replys however.
>
> So, with 8 gbit, if I did my math right, and we want to cover
> 100 million nicks, then the likihood of missing a reply would be
> about 1 in 800.

Depends upon whether the OP needs "perfection" or "close enough" (and
on just how long the OP plans to have this running between restarts of
the process, but then a restart would also restart the sending of
automated "hi" messages again). But yes, this method would constrain
the memory usage to a fixed amount, at the expense of missing saying
"hi" to some percentage of nick's.

An alternative, which simply changes where the space is used, would be
to link in the sqlite driver and setup an indexed sqlite table to store
seen nicks, then the 'growth' would be on disk instead of in RAM. But
as most systems today have much more free disk than free RAM, this
saves the RAM usage aspect.

Another alternative is to store not only the nick, but a timestamp for
"last seen" - then periodically expire nick's with a last seen date
older than some amount (week, month, six months, etc.). The expiration
process would act as a garbage collector to keep the memory usage
reasonably bounded. But again this means that some nick's will get an
automated "hi" all over again.

But we don't know what the OP really wanted. I suspect they did not
want absolute "perfection". If they were to ever explain in an
understandable way exactly what they want to accomplish, I suspect we'd
learn that the "say hi once" requirement was really:

Don't automatically say hi more often than once a week

or something similar. I.e., I suspect the "expire old lastseen nick's"
method I suggest above would be exactly what the OP wants. The
frequent nick's only get one automated hi, the infrequent ones would
get more than one, but for an infrequent nick, this is probably not a
bad thing.

[Rich]

heinrichmartin <martin....@frequentis.com> wrote:
> On Tuesday, May 14, 2019 at 7:31:55 AM UTC+2, two...@gmail.com wrote:

>> How about the following approach then. I don?t know

>> how to do this in tcl, but in C, I would:
>>
>> Allocate a large bit array, say 1gb or 8 gbit. Init all to 0.
>
> Not at script level
> (https://www.tcl-lang.org/cgi-bin/tct/tip/418.html), but in the C
> interface: https://www.tcl-lang.org/man/tcl/TclLib/ByteArrObj.htm

It /can/ be done at script level, just not as clean as in C nor as
compact as in C (this below assumes a 64-bit Tcl):

set bit_array [lrepeat 134217728 [expr 0]]

And one has a 1GiB bit array (which will consume much more than 1GiB of
core due to Tcl list pointer overhead when all the 64-bit words start
getting filled in).

Set a bit is uglier than in C:

lset bit_array 123456 [expr {[lindex $bit_array 123456] | 0x0000000100000000}]

And test a bit:

if {0 != [expr {[lindex $bit_array 123456] & 0x0000100000000000}]} {
# ...
}

Possible to do, and if one were to do this I'd suggest wrapping it up
in some convience procs to make usage easy to write (and easy to
understand later (i.e., test-bit, set-bit, clear-bit).

> I agree we are missing information, but my guts feeling suggests to
> prepare for more data (e.g. count messages per nick).

This is a possible next requirement -- if we ever hear anything back
from the OP.

> Btw, if we are really looking at a 24/7 service and session data is a
> factor, then OP could even consider using the disk (db, file, ...)
> optionally along with a cache.

Yes - and sqlite integrates nicely with Tcl, so even for maintaining a
count of messages per nick, this would be a good use for sqlite.

[heinrichmartin]

On Tuesday, May 14, 2019 at 2:57:03 PM UTC+2, Rich wrote:
> heinrichmartin wrote:

> > Not at script level
> > (https://www.tcl-lang.org/cgi-bin/tct/tip/418.html), but in the C
> > interface: https://www.tcl-lang.org/man/tcl/TclLib/ByteArrObj.htm
>
> It /can/ be done at script level, just not as clean as in C nor as
> compact as in C (this below assumes a 64-bit Tcl):
>
> set bit_array [lrepeat 134217728 [expr 0]]
>
> And one has a 1GiB bit array (which will consume much more than 1GiB of
> core due to Tcl list pointer overhead when all the 64-bit words start
> getting filled in).
>
> Set a bit is uglier than in C:
>
> lset bit_array 123456 [expr {[lindex $bit_array 123456] | 0x0000000100000000}]
>
> And test a bit:
>
> if {0 != [expr {[lindex $bit_array 123456] & 0x0000100000000000}]} {
> # ...
> }
>
> Possible to do, and if one were to do this I'd suggest wrapping it up
> in some convience procs to make usage easy to write (and easy to
> understand later (i.e., test-bit, set-bit, clear-bit).

I'd call this "messing with lists of binary numbers", not "using a bytearray" ;-)

If these wrappers were available, we'd be close enough.

Another idea just came up: https://core.tcl.tk/tcllib/doc/trunk/embedded/md/tcllib/files/modules/virtchannel_base/tcllib_memchan.md (but which uses [set content [string replace $content ...]] for writing, i.e. temporarily doubles memory usage if I assume correctly (copy on write))

[Rich]

heinrichmartin <martin....@frequentis.com> wrote:
> On Tuesday, May 14, 2019 at 2:57:03 PM UTC+2, Rich wrote:
>> heinrichmartin wrote:
>> > Not at script level
>> > (https://www.tcl-lang.org/cgi-bin/tct/tip/418.html), but in the C
>> > interface: https://www.tcl-lang.org/man/tcl/TclLib/ByteArrObj.htm
>>
>> It /can/ be done at script level, just not as clean as in C nor as
>> compact as in C (this below assumes a 64-bit Tcl):
>>
>> set bit_array [lrepeat 134217728 [expr 0]]
>>
>> And one has a 1GiB bit array (which will consume much more than 1GiB of
>> core due to Tcl list pointer overhead when all the 64-bit words start
>> getting filled in).
>>
>> Set a bit is uglier than in C:
>>
>> lset bit_array 123456 [expr {[lindex $bit_array 123456] | 0x0000000100000000}]
>>
>> And test a bit:
>>
>> if {0 != [expr {[lindex $bit_array 123456] & 0x0000100000000000}]} {
>> # ...
>> }
>>
>> Possible to do, and if one were to do this I'd suggest wrapping it up
>> in some convience procs to make usage easy to write (and easy to
>> understand later (i.e., test-bit, set-bit, clear-bit).
>
> I'd call this "messing with lists of binary numbers", not "using a
> bytearray" ;-)

To be fair, I did say: "It /can/ be done at script level, just not as
clean as in C nor as compact as in C".

> If these wrappers were available, we'd be close enough.

Hmm, are you suggesting another Tcllib module, the "bitfield" or
"bitarray" module?

> Another idea just came up:
> https://core.tcl.tk/tcllib/doc/trunk/embedded/md/tcllib/files/modules/virtchannel_base/tcllib_memchan.md
> (but which uses [set content [string replace $content ...]] for
> writing, i.e. temporarily doubles memory usage if I assume correctly
> (copy on write))

Yeah, that writing method would look to consume double memory for each
update. My "list of integers" also does the same amount of copy on
write, but only for an individual 64-bit integer at a time, so much
less memory expansion.

But realistically, if one wanted to do this for any sizeable bitarray
value, a C extension would be best, rather than coding it in pure Tcl.

[two...@gmail.com]

On Tuesday, May 14, 2019 at 6:55:26 AM UTC-7, Rich wrote:

>
> Hmm, are you suggesting another Tcllib module, the "bitfield" or
> "bitarray" module?
>

Perhaps this could be part of the core [array] command. In addition
to associative arrays, we'd have the good old fashioned kind. Maybe
both bit and byte, or even any adjustable width.

Since array variables are already distinct from regular variables, there
must be some bookkeeping. Perhaps a second type of array variable.

array allocate arrayName -type bit/byte/... quantity
array deallocate arrayName

then

array set arrayName index value
array get arrayName index

could overload the array command and subcommands, where the array
type would determine the operation and arguments.

>
> But realistically, if one wanted to do this for any sizeable bitarray
> value, a C extension would be best, rather than coding it in pure Tcl.

Or just built in as above. Tcl 9.0 maybe?

[heinrichmartin]

On Tuesday, May 14, 2019 at 6:45:40 PM UTC+2, two...@gmail.com wrote:
> On Tuesday, May 14, 2019 at 6:55:26 AM UTC-7, Rich wrote:
>
> >
> > Hmm, are you suggesting another Tcllib module, the "bitfield" or
> > "bitarray" module?
> >
>
> Perhaps this could be part of the core [array] command.

Actually, https://www.tcl-lang.org/cgi-bin/tct/tip/418.html already describes an API.

A small (scripted) wrapper could provide bitfields using [binary poke] and [binary scan]. Btw, what would be the name of the reading subcommand that mirrors [binary poke], i.e. read at an index? pull? tear? seekscan?

> array allocate arrayName -type bit/byte/... quantity
> array deallocate arrayName

allocate and deallocate are not tcl'ish.

> array set arrayName index value
> array get arrayName index
>
> could overload the array command and subcommands, where the array
> type would determine the operation and arguments.

I'd object overloading such different behavior. Note that legacy set and get transform the array from and to dicts (actually lists of even length).

[two...@gmail.com]

On Tuesday, May 14, 2019 at 2:21:08 PM UTC-7, heinrichmartin wrote:
> Actually, https://www.tcl-lang.org/cgi-bin/tct/tip/418.html already describes an API.

This tip is 7 years old, is there any chance it will be implemented?

After a quick perusal, it seems somewhat complex. I was thinking
something simpler that everyone is familiar with, the basic numerical array.

The goals would be compactness, efficiency, and simplicity - plus safety.

> allocate and deallocate are not tcl'ish.

Do you object to the particular words themselves,
or the idea of explicitly allocating and de-allocating
chunks of memory? If the former, perhaps,

array binary arrayName size
array unset arrayName ;# just uses existing unset

> > array set arrayName index value
> > array get arrayName index

> I'd object overloading such different behavior. Note that legacy set and get transform the array from and to dicts (actually lists of even length).

I see your point. What about setbin/getbin peek/poke or some variation?

I hesitate to suggest this, but I also wondered about overloading the
current array notation with array(index) and $array(index) so the
actual [array] command would not be needed except to create the
new type of binary array variable.

I can see lots of uses for a faster more compact array mechanism. But I
have no idea how difficult it would be to implement. Fun to think about.

[Rich]

two...@gmail.com wrote:
> On Tuesday, May 14, 2019 at 2:21:08 PM UTC-7, heinrichmartin wrote:
>> Actually, https://www.tcl-lang.org/cgi-bin/tct/tip/418.html already
>> describes an API.
>
> This tip is 7 years old, is there any chance it will be implemented?

Implementation requires someone to *do* the work. Are you
volunteering?

> After a quick perusal, it seems somewhat complex. I was thinking
> something simpler that everyone is familiar with, the basic numerical
> array.
>
> The goals would be compactness, efficiency, and simplicity - plus
> safety.

For that, the vectcl extension might be what you are after.

> I hesitate to suggest this, but I also wondered about overloading the
> current array notation with array(index) and $array(index) so the
> actual [array] command would not be needed except to create the new
> type of binary array variable.

I'd agree with Heinrich here, adding "perl like" magic hidden behavior
based upon the underlying TclObj 'type' does not seem the right way to
go here. I'd go with making bitfield stuff explicit, not 'magic'.

[two...@gmail.com]

On Tuesday, May 14, 2019 at 4:56:58 PM UTC-7, Rich wrote:
>
> Implementation requires someone to *do* the work. Are you
> volunteering?

I'm more known for hair-brained suggestions. My implementation
skills are pretty lacking. And I've not coded in C for many years.
Please ignore me when I get too carried away.

> For that, the vectcl extension might be what you are after.

Ahhh, didn't even know I already had that. Cool.

> I'd agree with Heinrich here, adding "perl like" magic hidden behavior
> based upon the underlying TclObj 'type' does not seem the right way to
> go here. I'd go with making bitfield stuff explicit, not 'magic'.

Makes sense. I also now recall that blt had vectors, though I don't
see a version that runs in 8.6.

[jonny]

Il 09/05/19 22:21, onem...@gmail.com ha scritto:

> Without going to the overhead of a coroutine, the simplest method is a global array or dict. Using an array, it would look something like:
>
> proc hello_response {nick uhost handle text} {
> global seen
> if {![info exists seen($nick)]} {
> set seen($nick) true
> putserv "privmsg $nick :Hello !"
> }
> # If you want to do something different if it's not the first time,
> # it goes in the else clause here.
> }
>
> Note that it must be global (or static in some other namespace) to persist across the multiple invocations for each message.

Ok, thanks