How can I debug duplicate & triplicate USENET posts?

[Danny D]

Every once in a while I get duplicate and triplicate
posts using AIOE where the only difference between the
posts is the message id and the dates are seconds apart.

How can I debug if this is happening on my end or on
the aioe server end?

All I know on my side is that some posts take up to
30 or 40 seconds to disappear from Pan, and it's those
that are often (but not always) the duplicates & triplicates.

If I'm lucky, this will be triplicate posted; but I have no
control to make it happen or not.

I'm using Pan on Centos. This only happens with aioe, and
not on any other newserver that I use.

[Paul]

But in one of your examples, the MID shows two messages
via AIOE and one message via Albasani.

http://groups.google.com/group/rec.autos.tech/msg/080f3b82ce1f0f43?hl=en-GB&dmode=source

"Yikes. Sorry for the duplicate posts.
AIOE keeps doing that!
I don't know why.

Message-ID: <kmuerm$4qu$1...@speranza.aioe.org>
Date: Tue, 14 May 2013 22:46:14 +0000 (UTC)

Message-ID: <kmuesk$63c$1...@speranza.aioe.org>
Date: Tue, 14 May 2013 22:46:44 +0000 (UTC)

Message-ID: <kmueug$t1s$1...@news.albasani.net>
Date: Tue, 14 May 2013 22:47:45 +0000 (UTC)
"

Does PAN support multiple accounts at the same time ?
Can you tell PAN to submit the same message, to multiple
servers at the same time ?

What is weird about your report, is AIOE has an
anti-spamming feature. It calculates a hash over
the body of each message. If two messages submitted,
have the same hash, the second message would be
rejected. The weird part then would be, how would
two identical messages, be accepted by AIOE ? I tried
looking at the two messages, and the body appears
identical. Now, if the albasani message was sent first,
then forwarded to AIOE, does AIOE ignore identical hashes,
if a message comes from another server with the exact same
hash ?

http://al.howardknight.net/msgid.cgi?STYPE=msgid&A=0&MSGI=%3Ckmuerm%244qu%241%40speranza.aioe.org%3E

http://al.howardknight.net/msgid.cgi?STYPE=msgid&A=0&MSGI=%3Ckmuesk%2463c%241%40speranza.aioe.org%3E

Maybe this will stop happening, when Pan is set up with only
one server in the config file ?

Paul

[Danny D]

On Wed, 15 May 2013 00:13:05 -0400, Paul wrote:

> in one of your examples, the MID shows two messages via AIOE
> and one message via Albasani.

That was my mistake for including that third message header.
What had happened is that Pan was hanging for, oh, maybe 30 seconds.
So, I killed the post - and reposted it with a different news server.
What ended up happening was that Aioe posted it twice (apparently
before I killed the post); and then Albasani sent it without problem.

But that was just my mistake in posting all three; if I had not killed
the post after about a half minute, 3 would have resulted from AIOE.

In debugging more, I noticed a LOT of consecutive messages in my Pan log
file that show reader.news4all.se:119 timing out, even though I'm not
using that newsserver for any of these messages.

I'm wondering if the numerous (scores) of reader.news4all.se:119 timeouts
is what's causing Pan to hang for 30 seconds - and - somehow - to send
the message two and three times to Aioe?

I realize this is farfetched, but I'm trying to figure out why *only*
aioe
duplicates my messages and this only happens when the Pan process hangs
for about 30 seconds - so I'm trying to figure out *why* the Pan process
of sending messages is hanging.

All I see in the log files are the scores of messages about
reader.news4all.se, which has always been a flaky server.

It's too bad Pan on Linux can't just *disable* a news server; you have
only the option of completely removing that server. So that's what I did
(but it's a pain to completely remove it).

It would be nice if I had a nntp client that simply allowed us to enable
and disable a server at will.

Anyway, I'll see if removing the flaky unrelated server eliminates the
problem of AIOE duplicating and triplicating posts.

[Danny D]

On Wed, 15 May 2013 00:13:05 -0400, Paul wrote:

> AIOE has an anti-spamming feature.

> If two messages submitted, have the same hash, the second message
> would be rejected.

If this is true (and it makes sense that it would be the case),
there is something definitely wrong with that AIOE anti-spam feature.

The next time it happens (and it will), I'll post the message IDs
and a link to the triplicate messages.

About all I can say now about when the duplicates happen is:

a) It only happens after pan "hangs" for about 20 or 30 seconds upon
message sending (normally Pan just sends the message in a second or two).

b) It has nothing do to with my network that I can tell; but it does seem
that there might be a cluttering set of connection errors from another
free news server at the same time in my Pan log file ... so I just now
deleted that news server.

c) It seems Aioe sends triplicate messages if I don't do any actions
after the SEND; but if I kill the send, AIOE still sends duplicate
messages (which are the message IDs that I had posted prior).

d) I can easily send the exact message using a different server (e.g.,
Albasani), so this is apparently only related to AIOE.

e) I post a LOT using AIOE (mostly on alt.home.repair where I am
prolific), and, often far more than the 25 per day - so Paolo Amoroso
must have a special exclusion for me. Either that or the limit of 25
messages per day for Aioe isn't real.

f) I have been in email communication with Paolo Amoroso at Aioe in the
past (regarding which newsgroups are on the banlist, for example); but he
hasn't responded yet to this duplicate message ID problem.

[Danny D]

On Wed, 15 May 2013 17:54:57 +0000, Danny D wrote:

> b) It has nothing do to with my network that I can tell; but it does
> seem that there might be a cluttering set of connection errors from
> another free news server at the same time in my Pan log file ... so I
> just now deleted that news server.

After deleting the free news server reader.news4all.se:119, I noticed in
my Pan log file another free news server that was timing out:
Error connecting to "freenews.netfront.net:119" (Connection timed out)
so, I'll delete that one also and send this message using aioe.

Note: I have only "aioe" set as my "primary" server in Pan 0.135 (all the
rest are set as "secondary" servers).

As an aside, it seems that any nntp client that allows multiple news
servers should simply have an enable/disable checkbox. That way, instead
of having to *delete* the news server, one could simply *disable* it.

OT question:
Do any (linux) nntp clients you know of have such a "disable" option?

[Jeff-Relf.Me]

If you have plain⋅text server settings, just commment them out.
Mine looks like this ( notice the "//" comments ):

Glorb NNTP newsRead.Glorb.COM:80 username password

// "⁻" tagged servers won't post HTML ( Content-Type: Text/HTML ).
Glorb⁻ NNTP newsRead.Glorb.COM:119 username password
XSU NNTP Reader.XSusenet.COM:119 username password
Alt° NNTP usNews.BlockNews.NET:119 username password
Alt¹ NNTP News.US.Usenet-News.NET:119 username password
// Indy⁻ NNTP News.Individual.NET:119 username password
// Aioe⁻ NNTP Aioe.ORG:119
// Sep⁻ NNTP News.Eternal-September.ORG:119 username password

[Danny D]

On Wed, 15 May 2013 11:17:42 -0700, Jeff-Relf.Me wrote:

> If you have plain⋅text server settings, just commment them out.
> Mine looks like this ( notice the "//" comments ):

For Pan 0.135, there is a text file in ~/.pan2 titled "servers.xml" but
it's a mess syntactically (so it would be difficult to comment out
gracefully):

<server id="12">
<host>aioe.org</host>
<port>119</port>
<username></username>
<password></password>
<expire-articles-n-days-old>31</expire-articles-n-days-old>
<connection-limit>4</connection-limit>
<newsrc>newsrc-12</newsrc>
<rank>2</rank>
</server>

[Danny D]

On Wed, 15 May 2013 00:13:05 -0400, Paul wrote:

> Maybe this will stop happening, when Pan is set up with only one server
> in the config file ?

Hi Paul,
What I did was move the ~/.pan2 directory and start anew.

It was a pain setting everything all up again, but I just finished that.
The good news is that Pan is closing and posting more gracefully, so
maybe, just maybe, something was corrupt in the ~/.pan directory.

I set all the servers up again, and noticed that some were still not
responding, but that aioe was only single posting so maybe that wasn't
the problem after all.

Here, for example, is a screenshot of my Pan 0.135 log file after
starting fresh, which necessitated getting all the groups and messages
from scratch.

http://www3.picturepush.com/photo/a/13044456/img/13044456.png

Notice that despite the error message about aioe.org in the log file, I
am posting this message through aioe.org. Albasani also works, despite
that error message.

Why they have these error messages confounds me to no end, since they
work, but they sure do complain a lot! :)

[Danny D.]

On Wed, 15 May 2013 21:03:14 +0000, Danny D wrote:

> Notice that despite the error message about aioe.org in the log file, I
> am posting this message through aioe.org. Albasani also works, despite
> that error message.

Here's a post via albasani.
Notice both aioe and albasani read and post, despite the error messages:
http://www3.picturepush.com/photo/a/13044456/img/13044456.png

[Jeff-Relf.Me]

Comment "servers.xml" like this: ＜＜

<!-- Start Comment -->

<server id="12">
<host>aioe.org</host>
<port>119</port>
<username></username>
<password></password>
<expire-articles-n-days-old>31</expire-articles-n-days-old>
<connection-limit>4</connection-limit>
<newsrc>newsrc-12</newsrc>
<rank>2</rank>
</server>

<!-- End Comment --> ＞＞

To comment⋅out the block, change the top line to: ＜＜

<!-- Start Comment !-> ＞＞

The syntax is explained here:

https://developer.mozilla.org/en-US/docs/Web/API/Comment?redirectslug=DOM%2FComment

[Paul]

How many connections are you allowing Pan to attempt to open ?

The servers each have a limit on open connections. This is because
each connection uses resources on the server.

If you hit the server limit, your attempt will be rejected. A few
seconds later, when one of the active connections drops, your next
attempt to connect will succeed. In a way, you're "hammering" the server.
And all because of the level of parallelism in Pan.

http://www.aioe.org/index.php?terms-of-use

"Each IP address has the right to establish two concurrent connections
per time with 'news://nntp.aioe.org' or four connections
with 'news://news.aioe.org' which is slower. All IP addresses
which make some attempt to establish more than some dozen of
concurrent connections are banned forever."

"Each IP address has got the right to establish about
400 connections per day even if the exact number of allowed
connections changes depending on the system load."

Paul

[Danny D]

On Wed, 15 May 2013 18:25:59 -0400, Paul wrote:

> How many connections are you allowing Pan to attempt to open ?

Good question!
I never understood that "connection" setting in Pan.
So, I mostly leave it at the default:
http://www2.picturepush.com/photo/a/13045460/img/13045460.png

All my servers are set up for "4", simply because that's the default.
Should I set it to 1?

> "Each IP address has the right to establish two concurrent
> connections per time with 'news://nntp.aioe.org' or four
> connections with 'news://news.aioe.org' which is slower.

Hmmm... I'm using "aioe.org". Which one is that?
$ ping aioe.org
==> PING aioe.org (94.75.214.39) 56(84) bytes of data.
==> 64 bytes from newsfeed.aioe.org (94.75.214.39): icmp_seq=8 ttl=53
time=160 ms

> "Each IP address has got the right to establish about
> 400 connections per day

Hmm... I thought it was 25 messages; so maybe that explains why
I can post more than 25 messages in a day.

If I have aioe.org set as a primary server, does that increase
the number of connections? (If so, I'll put it as a secondary
server because it doesn't seem to matter otherwise.)

[Danny D]

On Wed, 15 May 2013 18:25:59 -0400, Paul wrote:

> Each IP address has the right to establish two concurrent
> connections per time with news://nntp.aioe.org or four
> connections with news://news.aioe.org which is slower.

I was using "aioe.org":
http://www2.picturepush.com/photo/a/13045460/img/13045460.png

$ ping aioe.org

==> 64 bytes from newsfeed.aioe.org (94.75.214.39):

==> icmp_seq=1 ttl=53 time=161 ms

$ ping nntp.aioe.org
==> 64 bytes from nntp.aioe.org (94.75.214.90):
==> icmp_seq=1 ttl=52 time=176 ms

$ ping news.aioe.org

==> 64 bytes from newsfeed.aioe.org (94.75.214.39):

==> icmp_seq=3 ttl=53 time=160 ms

So aioe.org is the same as news.aioe.org (which is the slower one).

[Danny D]

On Wed, 15 May 2013 18:05:43 +0000, Danny D wrote:

> As an aside, it seems that any nntp client that allows multiple news
> servers should simply have an enable/disable checkbox. That way, instead
> of having to *delete* the news server, one could simply *disable* it.
>

> Do any (linux) nntp clients you know of have such a "disable" option?

The ng news.software.readers had the answer for disabling newsservers:

On Thu, 16 May 2013 00:05:58 -0500, John F. Morse wrote:
> Pan can disable any news server.
> Just set your Connection Limit in Pan to zero.