INN rejecting articles

[Chris Knipe]

Hi All,

I have a strange issue with INN 2.5.3

Stats from one of my peers:
ip address: x.x.x.x
seconds: 2349 duplicates: 0 max allowed cxns: 10
offered: 75491 uw newsgroups: 0 active cxns: 8
accepted: 0 uw distributions: 0 sleeping cxns: 0
refused: 6434 unapproved: 0 want streaming: Yes
rejected: 69057 filtered: 0 is streaming: Yes
size: 0.0kb bad sites: 0 duplicate size: 0.0kb
reject size: 202.3kb
Protocol:
Ihave: 0 SendIt[335]: 0 Got[435]: 0 Deferred[436]: 0
Check: 75554 SendIt[238]: 69120 Got[438]: 6434 Deferred[431]: 0
Takethis: 69057 Ok[239]: 0 Error[439]: 69057

This only happens to this one single peer. For some reason, the peer is
offering articles to me which I do not have. Looking at TCP dumps I can see
that my INN talks correctly with the peer, and that my INN requests the remote
peer to send me the particles.

The conversation goes like this (specific message example):
PEER -> ME: check <IIDK99jQdl...@JBinUp.local>
ME -> PEER: 238 <IIDK99jQdl...@JBinUp.local>
PEER -> ME: takethis <IIDK99jQdl...@JBinUp.local>
ME -> PEER: 439 <IIDK99jQdl...@JBinUp.local>

I have no idea why my INN is reporting an 439 back to the peer. I have checked
every single log file that I can think of, and I have no reference in any log
file for that specific Message-ID (all transferred messages are logged).

Can anyone perhaps aid in assisting as to why all the articles from this
peer is resulting in errors? Or even just to give some guidance to determine
what exactly the error is so that I can research possible ways to fix it?

All I know at this stage is that it is failing, and INN is not giving me any
indication at this stage as to why :-(

Thank you all,
Chris.

[Xavier Roche]

On 05/17/2013 10:50 AM, Chris Knipe wrote:
> PEER -> ME: takethis <IIDK99jQdl...@JBinUp.local>
> ME -> PEER: 439 <IIDK99jQdl...@JBinUp.local>

Do you have this error for every single message ? It might be rejected
messages for various reasons (unwanted group, malformed newsgroups, date
or lines header, unapproved message in a moderated group, or simply
duplicate due to a no-check site which sent the message in the meantime)

> I have no idea why my INN is reporting an 439 back to the peer. I have checked
> every single log file that I can think of, and I have no reference in any log
> file for that specific Message-ID (all transferred messages are logged).

Nothing relevant in /var/log/news/news ? (grep 439 /var/log/news/news)

[sav...@savage.za.org]

> <kn4r0n$a2f$1...@nntpspool01.opticnetworks.net>

>On 05/17/2013 10:50 AM, Chris Knipe wrote:
>> PEER -> ME: takethis <IIDK99jQdl...@JBinUp.local>
>> ME -> PEER: 439 <IIDK99jQdl...@JBinUp.local>
>
>Do you have this error for every single message ? It might be rejected
>messages for various reasons (unwanted group, malformed newsgroups, date
>or lines header, unapproved message in a moderated group, or simply
>duplicate due to a no-check site which sent the message in the meantime)

It's pretty much for 99.9% of all messages from this one single peer yes
(and it's an big full peer inclusive of binaries). All other peers are
working fine.

>> I have no idea why my INN is reporting an 439 back to the peer. I have checked
>> every single log file that I can think of, and I have no reference in any log
>> file for that specific Message-ID (all transferred messages are logged).
>
>Nothing relevant in /var/log/news/news ? (grep 439 /var/log/news/news)

The specific Message-ID posted in this example does not feature in my news log
file at all. However, after going through the news log, I can see that there
are quite a few issues, especially related to non existant groups. I will
keep my eye on there and see how it goes.

Thanks for the pointer into the right direction :-)

--
Chris.

[Xavier Roche]

On 05/17/2013 12:02 PM, sav...@savage.za.org wrote:
> It's pretty much for 99.9% of all messages from this one single peer yes
> (and it's an big full peer inclusive of binaries)

Did you check the latency between this peer and yourself ? If the peer
has a latency greatly higher that other peers, they are probably
"short-circuiting" it even in rare cases when your local INN did not
(yet) receive the (no-check) message [ie. this is a normal behavior ;
the peer is just a bit useless].

[Chris Knipe]

>Do you have this error for every single message ? It might be rejected
>messages for various reasons (unwanted group, malformed newsgroups, date
>or lines header, unapproved message in a moderated group, or simply
>duplicate due to a no-check site which sent the message in the meantime)

Ok, so no error message for every group, but I used the active file from
ftp://ftp.isc.org/pub/usenet/CONFIG as per the inn documentation.

After comparing that active file with a few other providers (Giganews,
Astraweb, etc), it's obvious the group file on isc.org is seriously
outdated. We where missing some 100K odd groups :-( I think that will
explain quite allot.

We're busy importing the missing groups now and will check how it goes
after that.

--
Chris.

[Russ Allbery]

Chris Knipe <sav...@savage.za.org> writes:

> Ok, so no error message for every group, but I used the active file from
> ftp://ftp.isc.org/pub/usenet/CONFIG as per the inn documentation.

> After comparing that active file with a few other providers (Giganews,
> Astraweb, etc), it's obvious the group file on isc.org is seriously
> outdated. We where missing some 100K odd groups :-( I think that will
> explain quite allot.

That file is certainly not *outdated* -- it was updated today. It may not
follow the group creation policy you want, but that's a different problem
than being outdated. :)

Please see:

ftp://ftp.isc.org/pub/usenet/CONFIG/README

for information about the policy used to form that active file. If you
see an indication that the file doesn't follow that policy, please let me
know; there may be a bug in the code that creates it.

--
Russ Allbery (r...@stanford.edu) <http://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<http://www.eyrie.org/~eagle/faqs/questions.html> explains why.

[Tony]

In news.software.nntp, Chris Knipe <sav...@savage.za.org> wrote:

>>Do you have this error for every single message ? It might be rejected
>>messages for various reasons (unwanted group, malformed newsgroups, date
>>or lines header, unapproved message in a moderated group, or simply
>>duplicate due to a no-check site which sent the message in the meantime)
>
>Ok, so no error message for every group, but I used the active file from
>ftp://ftp.isc.org/pub/usenet/CONFIG as per the inn documentation.
>
>After comparing that active file with a few other providers (Giganews,
>Astraweb, etc), it's obvious the group file on isc.org is seriously

Giganews appears to accept group creation control messages from
unauthorised sources in properly managed hierarchies, and they appear to
ignore properly authorised rmgroups in those hierarchies (this, I suspect,
allows them to say 'we have 90 bazillion groups, yay').

I wouldn't trust Giganews' list of 'active groups' for anything.
--
Tony Evans
Saving trees and wasting electrons since 1993
blog -> http://perceptionistruth.com/
books -> http://www.bookthing.co.uk/
[ anything below this line wasn't written by me ]

[Adam H. Kerman]

ftp.isc.org is a site for archiving and testing control messages. It's not
a site you would peer with, so it's not logical to build your active file
from there.

Various newsgroups weren't started with control messages. Entire hierarchies
aren't maintained with control messages. Various hipclones over the years
sent newgroup messages in great quantity as denial of service attacks. Some
of these got archived, but many didn't.

Lastly, Giganews and Astraweb decide for themselves how to present Usenet
to their subscribers. If they chose to create newsgroups that weren't
started with control messages, or to ignore rmgroup messages in
managed hierarchies, that's entirely up to them.

Are you supporting users? Did they have a need for all those newsgroups to
be created on your server that aren't necessarily Usenet?

If you peer with Giganews and Astraweb and you want to present their view
of Usenet to your users, then you're going to want to use their active files.