INN the only software?

[hurst]

Is INN the only usenet news software for linux? I've done some searching
around and can't seam to find much of anything else. There is citadel
but that is a entire groupware sweet in it self. There is leafnode but I
don't know if you can host local groups with that or not.

[Matthias Meyser]

INN
Diablo
DNews
C News
WendzelNNTPd

Historic Software

B News
A News



--
Diese E-Mail wurde von AVG auf Viren geprüft.
http://www.avg.com

[G.K.]

On 7/29/22 07:22, Matthias Meyser wrote:
> INN
> Diablo
> DNews
> C News
> WendzelNNTPd
>
> Historic Software
>
> B News
> A News

# Rslite
has PHP front end and automated email registration system
usenet group : rocksolid.nodes
web : https://novabbs.com
faq : https://novabbs.com/common/faq.txt
code : https://github.com/novabbs/rocksolid-light

# Papercut NNTP server
has forum front end and automated registration system
web : http://pessoal.org/papercut/
code : https://github.com/jpm/papercut

--

G.K.

[jdanield]

Le 04/08/2022 à 11:08, noel a écrit :

> format INN understands so we have shelved the idea, not too worried since
> ipv4 despite exhaustion will be round for a very long time yet anyway.

not so long. My Linux User Group FAI (REDbySFR) just set our box as full
IPV6 - no more IPV4 usable address for servers :-( - and no warning :-(

jdd

[yamo']

Hi,

noel a tapoté le 04/08/2022 11:08:
> We still run it (since the 90s) but all our attempts in past couple years
> to move the spool into INN which is ipv6 and 64bit capable, has thus far
> failed, it bogs down, and trying to move near TB of data would take
> months, as there is no apparent way to convert DNews's spool into a

> format INN understands so we have shelved the idea, not too worried since
> ipv4 despite exhaustion will be round for a very long time yet anyway.

The only simple solution would to find how to feed all articles from
Dnews server to the INN2 server.

Or with suck on the new server?
<https://netwinsite.com/dnews/faq3.htm#10>

I had never used suck but there may some users here.


--
Stéphane

[Jesse Rehmer]

On Aug 4, 2022 at 4:49:14 AM CDT, "yamo'" in <tcg4ng$3fr$1...@rasp.pasdenom.info>
wrote:

Using any of the single threaded tools like suck or pullnews to move TB's
worth of articles is tedious, highly error-prone, and takes forever. There is
a multi-threaded app called multisuck, but there is zero documentation about
it (looks incomplete perhaps) and I couldn't figure out how to make it work.
The main issue is using anything that uses a single thread or connection is
going to be a huge bottleneck when you have millions/billions of articles to
move.

I opted to use multiple instances of pullnews to initially seed the spool. It
crashed a lot and had difficulty with very large groups. After doing as much
as I could with pullnews I started using suck since it uses the history
database and deduplicates before downloading every article. This made filling
in the holes that were left with pullnews less taxing.

It's been a few months of going heavy at it and then a week in between of
validating, testing, etc. I think I'm about 1/3 of the way through the Big8
hierarchies.

[Julien ÉLIE]

Hi Jesse,

> I opted to use multiple instances of pullnews to initially seed the spool. It
> crashed a lot and had difficulty with very large groups.

Do you remember the errors you got?
They are maybe worthwhile fixing in pullnews.

What kind of "difficulty" is there with large groups? (resuming after a
crash? normally the article numbers of latest downloaded articles are
saved in the pullnews.marks file, making it easier to continue)

> After doing as much
> as I could with pullnews I started using suck since it uses the history
> database and deduplicates before downloading every article.

"pullnews -O" does that too.

--
Julien ÉLIE

« Ibi etsi uis te non esse sed es ibi. »

[Jesse Rehmer]

On Aug 4, 2022 at 10:55:29 AM CDT, "Julien ÉLIE" in

<tcgq5i$31omo$1...@news.trigofacile.com> wrote:

> Hi Jesse,
>
>> I opted to use multiple instances of pullnews to initially seed the spool. It
>> crashed a lot and had difficulty with very large groups.
>
> Do you remember the errors you got?
> They are maybe worthwhile fixing in pullnews.

It could be related to the server on the other end (pulling from a commercial
entity). The one situation that happened a lot, especially with large groups,
was at some point in the session pullnews would just start spewing x's to the
terminal in a loop and not stop until killed. It never gave me an error.
Killing and resuming always seemed to work.

In comparison, suck seems to handle whatever issue pullnews runs into by
eventually cleaning up and reporting a socket error. This made it easier to
see which instances weren't running anymore. When I had 20-50 pullnews
instances running at the same time I would have to connect to each screen to
make sure it wasn't barfing.

>> After doing as much
>> as I could with pullnews I started using suck since it uses the history
>> database and deduplicates before downloading every article.
>
> "pullnews -O" does that too.

It does but I did a few comparisons with pullnews and suck, even when using
the -O flag with pullnews, suck is a lot faster in the overall runtime of
pulling an entire group (tested the same group on empty servers). Internally I
don't know how the NNTP operations compare, but pullnews wasn't as efficient
as suck's method of building the list of articles, deduping, grabbing all
articles, and posting as serial operations led to getting the job done faster.

[Julien ÉLIE]

Hi Jesse,

> It could be related to the server on the other end (pulling from a commercial
> entity). The one situation that happened a lot, especially with large groups,
> was at some point in the session pullnews would just start spewing x's to the
> terminal in a loop and not stop until killed. It never gave me an error.
> Killing and resuming always seemed to work.

Interesting.
I see in the code that if pullnews receives an unexpected response code
when asking for article #i, it will print "x" and try #(i+1).

I've tested a timeout, as you suggested, and I confirm "pullnews -d2"
(debug mode) prints "x" and goes on...

x
DEBUGGING 1 421 [Net::NNTP] Connection closed
x
DEBUGGING 2 421 [Net::NNTP] Connection closed
x
DEBUGGING 3 421 [Net::NNTP] Connection closed
[...]

This is a bug.
I would just suggest in that case to verify whether the connection were
still active by sending a DATE command through Net::NNTP::date(). If
there is no answer, then we can consider the connection is no longer
here, and we just bail out.
And if there is an answer to DATE, hmmm, we can still be in a deadloop
with a repeated unknown error. Maybe we could also bail out after 10
attempts for instance?

Unfortunately, I do not see how to check that the connection is no
longer active with the Net::NNTP module pullnews uses :-/
It seems to just answer "undef" to each command, but it could be for
another reason. That's why I suggest to ask for the DATE.

> I did a few comparisons with pullnews and suck, even when using
> the -O flag with pullnews, suck is a lot faster in the overall runtime of
> pulling an entire group (tested the same group on empty servers). Internally I
> don't know how the NNTP operations compare, but pullnews wasn't as efficient
> as suck's method of building the list of articles, deduping, grabbing all
> articles, and posting as serial operations led to getting the job done faster.

I've just had a glance at suck source code. It directly runs the native
C function provided by INN (dbzexists) on the history file. No wonder
it is more efficient than pullnews which just runs an NNTP command
(STAT) on the local news server...

Thanks again for your valuable bug report!
Any other issue you remember with pullnews?

[Jesse Rehmer]

On Aug 4, 2022 at 1:57:05 PM CDT, "Julien ÉLIE" in

<tch4q1$31vln$1...@news.trigofacile.com> wrote:

> I've just had a glance at suck source code. It directly runs the native
> C function provided by INN (dbzexists) on the history file. No wonder
> it is more efficient than pullnews which just runs an NNTP command
> (STAT) on the local news server...
>
> Thanks again for your valuable bug report!
> Any other issue you remember with pullnews?

I think that was the primary issue I ran into. It would be great if an option
could be used to start a new connection should one be considered dead or
unusable.

Resource-wise, pullnews uses exponentially less memory and CPU than suck, so
there's that. Probably a memory leak, but suck really sucks up the memory. ;)

[Julien ÉLIE]

Hi Noel,

> We tried suck, its the only way they might get into INN, thats where its
> bogging down

I've found out in the Dnews documentation
<https://netwinsite.com/dnews/tellnews.htm> the following command:

"""
tellnews refeed feed.name

Description: This command will re-send all current articles to a
particular news feed. This command is fairly cpu intensive as the
entire history file (maybe 80MB) must be parsed. Also, it does not
apply all the rules associated with the feed you have specified, it only
checks that the group name matches. This command is particularly useful
when setting up the FTS (Full Text Searching) system on a running
server. It allows you to 're-index' all existing items. It should not
be used for other purposes.
"""

Did you try it? Maybe by setting the feed.name entry in newsfeeds.conf
and testing it with a single newsgroup to check whether it works? (and
if it is, then "group *")





Also, DNews seems to store all the articles in files named "db*.itm".
Did you try to have a look at the contents of these files? Maybe the
articles could be extracted from them with an appropriate (simple) script?

I've found out a message of the DNews support:
https://groups.google.com/g/netwin.dnews/c/ISgH6dc5baM/m/CrLz-bU4hDMJ

"""
Create a feed entry like this:
site xxx
groups *
And then write a program to read 'xxx.feed' or 'xxx.send' files
which list each new item, and gives the
offset into the bucket and item length so you can read direct from teh
.itm files, this method is not gauranteed to
work with future versiosn of dnews :-)
"""

Well, that does not seem easy... I'm very surprised by such a response
from the official DNews support. They could at least have natively
provided a tool to export their proprietary data spool. Seems like they
wanted their customes to be captive :-/

--
Julien ÉLIE

« La grippe, ça dure huit jours si on la soigne et une semaine si on ne
fait rien. » (Raymond Devos)

[Julien ÉLIE]

Hi Noel,

> We tried suck, its the only way they might get into INN, thats where its
> bogging down

What problem do you encounter with suck? (except that it may take a long
time to run)

Did you eventually try to run suck in its multifile mode "suck -m"
directly on your DNews news server to extract the articles?
It can also generate batch files suitable for innxmit or rnews with
respectively the "-bi" and "-br" flags.

--
Julien ÉLIE

« Un voyage de mille lieues commence toujours par un premier pas. » (Lao
Zi)

[hurst]

On 8/6/22 06:51, Julien ÉLIE wrote:
> Hi Noel,
>
>> We tried suck, its the only way they might get into INN, thats where its
>> bogging down
>
> What problem do you encounter with suck? (except that it may take a long
> time to run)
>
> Did you eventually try to run suck in its multifile mode "suck -m"
> directly on your DNews news server to extract the articles?
> It can also generate batch files suitable for innxmit or rnews with
> respectively the "-bi" and "-br" flags.
>

I would like to try Dnews running arch linux. How much does it cost and
how would you go about installing it. I was using INN running 1 group
for a nntp only access bbs. Could use wendzelnntp but that does not
allow for multiple servers or linking. I was using mystic bbs to link
the news server to fidonet.

[Julien ÉLIE]

Hi Noel,

>> Also, DNews seems to store all the articles in files named "db*.itm".
>> Did you try to have a look at the contents of these files? Maybe the
>> articles could be extracted from them with an appropriate (simple)
>> script?
>

> many of these bucket files are up to 195mb in zise, the majorty though
> are around 60-90mb, I've found one from 2018 thats about 10mb gz'd to 2mb
> if your curious to see what one looks like.
>
> https://members.ausics.net/noelb/db_1615_2.itm.gz

Very interesting, thanks for this sample!

These .itm files contain plain-text articles, dot-stuffed, and each line
ending with CRLF, separated with:

^~00001957:0000389823:017829:

The first field is the number of bytes of the article.
The second field is the article number the article has in the newsgroup
internally represented in DNews by the ID in the third field.

Converting these files to a format understandable by INN is not complicated.

For instance, each .itm file could be converted to an rnews batch file.
The tricky part is that one cannot directly rewrite the DNews
beginning line to "#! rnews 1957" because rnews expects articles in
native format (not dot-stuffed, and each line ending with LF) so the
number of bytes has to be recomputed after the translation.

Would it be something that would interest you, or do you prefer to keep
your DNews installation that already fits your needs?
At least you now know that you can transfer your spool to INN if you
wish for whatever reason in the future:
1/ create the conversion script itm -> rnews format (less than half a
day of work)
2/ run that script to convert all your itm files to rnews batches
3/ transfer these rnews batches to another server (INN or another one)
4/ locally run rnews, which will happily inject your spool in the new
news server

--
Julien ÉLIE

« – Heureusement, nous aurons bientôt le TGV !
– C'est quoi le tégévé ?
– C'est le Transport Gaulois Véloce. » (Astérix)

[Julien ÉLIE]

Hi Noel,

>> Did you try it? Maybe by setting the feed.name entry in newsfeeds.conf
>> and testing it with a single newsgroup to check whether it works? (and
>> if it is, then "group *")
>

> I'm very aware of that command and have used it before over the 25 years
> of running dnews, it was the first thing I tried and was also a failure
> in this case, perhaps it was the INN storage I selected, cant remember
> what it was.

It would have been interesting to have INN logs about your attempt, to
see if it really got it and why it rejected it (unrecognized command?
missing configuration somewhere? ...)

Maybe the refeed command works as-is, and could really be usable.

If you want, I can create a newsgroup on my news server and you can try
to refeed this newsgroup to me (a small newsgroup of a few articles
would be enough to see what's going on - or you could create a
local.refeed testing group, write a few articles there, and refeed it to
me afterwards).

--
Julien ÉLIE

« – Ouvre l'œil, et le bon !
– L'autre, je peux pas encore l'ouvrir, je risque pas de me
tromper ! » (Astérix)

[Julien ÉLIE]

Hi Noel,

>> What problem do you encounter with suck? (except that it may take a long
>> time to run)
>

> Its time to run, strangely, and this is likely memory related, it starts
> off fast enough, but over time, its just gets slow

Like Jesse found out too (high memory consumption).


> and not sure I can
> pause new articles for the 6 months it would take to be done, I'd have no
> one left using it with that long no new posts lol, I say that because I
> dpont want to upset counters, I dont want the sucking or pulling to be
> through 2010 articles trhen import this hours new stuff and go back to
> 2010 :)

Also exactly what Jesse wishes for article numbers being "time-ordered" :)

--
Julien ÉLIE

« Faites le mur, pas la guerre. » (Astérix)

[Jesse Rehmer]

On Aug 7, 2022 at 5:12:29 AM CDT, "Julien ÉLIE" in

<tco36d$3724r$1...@news.trigofacile.com> wrote:

> Hi Noel,
>
>>> What problem do you encounter with suck? (except that it may take a long
>>> time to run)
>>
>> Its time to run, strangely, and this is likely memory related, it starts
>> off fast enough, but over time, its just gets slow
>
> Like Jesse found out too (high memory consumption).

Most suck processes I've watched use 2+GB of memory, even with a small list of
groups. I found you can decrease its consumption by disabling the killfiles
(-K).

You may benefit from smaller batch postings. Here is the suck command I'm
using to pull from commercial NNTP servers that do strange things with
connection handling at times:

suck source.host.name -U username -P 'password' -bP 100 -hl localhost -HF
/usr/local/news/db/history -f -lr

These options will have suck do the following:

- Establish connections
- Build the list of articles from each group defined in sucknewsrc
- De-duplicate against the local history database (on an empty server you
should skip this)
- Re-establish the remote connection after de-duplication operations (I found
this was required because the de-dupe can take a long time)
- Download articles from oldest to newest (depending on source article
numbering this may be unreliable)
- After downloading 100 articles it will batch post them to the localhost

If suck gets 'stuck', or otherwise dies due to some error, when you run it
again, add the '-R' flag, it will skip steps two through four above.

>> and not sure I can
>> pause new articles for the 6 months it would take to be done, I'd have no
>> one left using it with that long no new posts lol, I say that because I
>> dpont want to upset counters, I dont want the sucking or pulling to be
>> through 2010 articles trhen import this hours new stuff and go back to
>> 2010 :)
>
> Also exactly what Jesse wishes for article numbers being "time-ordered" :)

It not the best user experience when they load 10,000 articles and get a mix
of date ranges all incomplete until the entire list of articles is processed
by the newsreader.

Cheers,

Jesse

[Julien ÉLIE]

Hi Jesse and Noel,

>>>> What problem do you encounter with suck? (except that it may take a long
>>>> time to run)
>>>
>>> Its time to run, strangely, and this is likely memory related, it starts
>>> off fast enough, but over time, its just gets slow
>>
>> Like Jesse found out too (high memory consumption).
>
> Most suck processes I've watched use 2+GB of memory, even with a small list of
> groups. I found you can decrease its consumption by disabling the killfiles
> (-K).

A recent discussion on a French newsgroup about the newsx program makes
me think that it could maybe interest you?

http://www.kvaleberg.no/newsx.html
http://www.kvaleberg.no/newsx-man.html

It seems to perform like suck, and perhaps better. It could be
worthwhile giving it a try. (It is unfortunately not packaged on all
distributions.)

--
Julien ÉLIE

« À vaincre sans péril, on évite des ennuis ! » (légionnaire romain)

[Julien ÉLIE]

Hi Jesse,

>>> I opted to use multiple instances of pullnews to initially seed the spool. It
>>> crashed a lot and had difficulty with very large groups.
>>
>> Do you remember the errors you got?
>> They are maybe worthwhile fixing in pullnews.
>
> It could be related to the server on the other end (pulling from a commercial
> entity). The one situation that happened a lot, especially with large groups,
> was at some point in the session pullnews would just start spewing x's to the
> terminal in a loop and not stop until killed. It never gave me an error.
> Killing and resuming always seemed to work.
>

> It would be great if an option could be used to start a new
> connection should one be considered dead or unusable.

I've just had a look at a fix.
I suggest to re-use the "-t" parameter. It will then be the maximum
number of attempts to connect to a server or reconnect to a server if
the socket is unexpectedly closed.

I think it answers your needs. Do not hesitate to tell if it is not the
case. The fix will be present in the next release (INN 2.7.1).


Here is the new behaviour:

% ./pullnews -t2
Sat Jan 21 15:31:44 2023 start

Connecting to downstream host: localhost port: 119... done.

Legend: ``.'' is an article the downstream server refused
``*'' is an article the downstream server rejected
``+'' is an article the downstream server accepted
``x'' is an article the upstream server couldn't give out
``m'' is an article skipped due to headers (-a, -m or -P)

Connecting to upstream server news.trigofacile.com... done.

trigofacile.test:
Last checked: Sat Jan 21 15:23:25 2023
608 articles available (first 7, last 712)
Our current highest: 30 (at most 608 articles to get)
..
Article retrieval failed ([Net::NNTP] Connection closed)

Let's attempt again.
Connecting to upstream server news.trigofacile.com... done.

trigofacile.test:
Last checked: Sat Jan 21 15:32:00 2023
608 articles available (first 7, last 712)
Our current highest: 32 (at most 608 articles to get)
......xx
Article retrieval failed ([Net::NNTP] Connection closed)

Let's attempt again.
Connecting to upstream server news.trigofacile.com... done.

trigofacile.test:
Last checked: Sat Jan 21 15:33:13 2023
608 articles available (first 7, last 712)
Our current highest: 40 (at most 608 articles to get)
.
Article retrieval failed ([Net::NNTP] Connection closed)

Let's attempt again.
Connecting to upstream server news.trigofacile.com... done.

trigofacile.test:
Last checked: Sat Jan 21 15:33:30 2023
608 articles available (first 7, last 712)
Our current highest: 41 (at most 608 articles to get)
..
Article retrieval failed ([Net::NNTP] Connection closed)


Saving config

11 articles were offered to server on localhost
11 articles were refused

Upstream server news.trigofacile.com:
trigofacile.test 11

Done Sat Jan 21 15:34:03 2023






And before, you had an infinite loop of "x" appearing in the output...




I've noted that pullnews does not currently support TLS connections.
I'll add an option for it for the downstream server, as it may be
useful. We can already configure a port for the downstream server (via
-p or -s).

As for upstream servers, pullnews.marks has the following syntax:

server [username password]

I would propose to extend it to:

server[:port][_TLS] [username password]

so as to ask for TLS, and also a different port. (Note that "server"
may also contain colons, if written as an IPv6 address.)
We would had:

news.server.com:563_TLS
news.server.com:433

Unless someone has a better idea to specify whether implicit TLS should
be used for a given upstream server?

--
Julien ÉLIE

« Perl programming is an *empirical* science. » (Larry Wall)

[Julien ÉLIE]

> Here is the new behaviour:
>
> % ./pullnews -t2

The behaviour I copied/pasted was for "-t3" as there are 3 reconnections.

--
Julien ÉLIE

« Don't marry for money; you can borrow it cheaper. » (proverbe
écossais)

[Julien ÉLIE]

Hi Jesse,

>> The one situation that happened a lot, especially with large
>> groups, was at some point in the session pullnews would just start
>> spewing x's to the terminal in a loop and not stop until killed.
>>

>> It would be great if an option could be used to start a new
>> connection should one be considered dead or unusable.
>
> I've just had a look at a fix.
>

> % ./pullnews -t2
[...]

> Article retrieval failed ([Net::NNTP] Connection closed)
>
> Let's attempt again.
> Connecting to upstream server news.trigofacile.com... done.

Works fine, I've retested it today.

> I've noted that pullnews does not currently support TLS connections.

Now implemented for downstream server:

-s to-server[:port][_tlsmode]
Normally, pullnews will feed the articles it retrieves to the news
server running on localhost. To connect to a different host,
specify a server with the -s flag. You can also specify the port
with this same flag or use -p. Default port is 119.

The connection is by default unencrypted. To negotiate a TLS
encryption layer, you can set *tlsmode* to "TLS" for implicit TLS
(negotiated immediately upon connection on a dedicated port) or
"STARTTLS" for explicit TLS (the appropriate command will be sent
before authenticating or feeding messages). Examples of use are:

pullnews -s news.server.com
pullnews -R -s news.server.com_STARTTLS
pullnews -s news.server.com:563_TLS


As well as for upstream servers in pullnews.marks:

<host>[:<port>][_<tlsmode>] [<username> <password>]
<group> [<time> <high>]
<group> [<time> <high>]


A typical configuration file would be:

# Format: group date high
data.pa.vix.com
rec.bicycles.racing 908086612 783
rec.humor.funny 908086613 18
comp.programming.threads
nnrp.vix.com pull sekret
comp.std.lisp
news.server.com:433_TLS joe password
news.software.nntp

The connection to news.server.com will be encrypted with implicit
TLS on
port 433. Joe's password won't be sent in plaintext.



This feature will be present in INN 2.7.1.

For those who want it on older releases, grabbing pullnews from sources
<https://raw.githubusercontent.com/InterNetNews/inn/main/frontends/pullnews.in>
and just modifying the first 2 lines to fit the Perl installation is
enough to make it work.

--
Julien ÉLIE

« – Où vous croyez-vous ici ?
– Où je me trouve, je sais. » (Astérix)