It takes too long to download multiple binary messages

[DonK]

I download a lot of movies and TV shows from a.b.movies.divx and a.b.multimedia using Agent 6.00/32,
Giganews with "Max connections = 20" and a 25 Mb/s Comcast connection.

Most of the movies are broken into 50 RAR's of about 14,649 MB each. If I select one RAR and press
"A" to download it, it takes between 7 and 12 seconds. (I'd say the average is about 8 seconds.)

When I select all 50 RAR's and press "A" it takes four times as long per RAR as it does when
selecting one individual RAR.

EXAMPLE:

I just downloaded the first 6 RAR's of a movie individually and they took from 4 to 10 seconds each
for a total of 35 secs. I downloaded the remaining 42 RAR's as a group and that took 28 mins and 35
secs or an average of 40.83 secs each.

- 42 bodies took 1,715 seconds = 40.83 secs/body

- A single body took 6 to 10 seconds each

EXAMPLE 2:

Select 1 RAR = 9 secs
Select 2 RAR's = 29 secs
Select 3 RAR's = 47 secs


Can anyone tell me why this happens and if/how I can make it faster when I download multiple
selections?

Thanks

Don

[Jaimie Vandenbergh]

[Default] On Sat, 11 May 2013 15:43:21 -0700, DonK

<don8...@comcast.net.notvalid> wrote:

>Can anyone tell me why this happens and if/how I can make it faster when I download multiple
>selections?

I'd guess that's because you have many connections working at once,
and Agent has to deal with all those data streams. It's not terribly
good at that, as the database format Agent uses was built looooooong
before fast Usenet access existed and isn't terribly good at multiple
access...

Try cutting down the number of Max Connections to say 2, or 4.

If that doesn't help, instead of pressing A, press M to mark things
for retrieval, then Action/Get bodies for marked messages.

Otherwise, get an SSD - that should speed things up.

Cheers - Jaimie
--
Sufficiently advanced incompetence is indistinguishable from malice.

[Ralph Fox]

On Sat, 11 May 2013 15:43:21 -0700, DonK wrote:

> Can anyone tell me why this happens and if/how I can make it faster when I download multiple
> selections?

See this thread from 3 months ago: http://narkive.com/aYe01f1W


--
Kind regards
Ralph

[Geoff]

On Sat, 11 May 2013 15:43:21 -0700, DonK

Make sure the max number of connections matches the max number of
connections allowed by your NNTP provider. Negotiating for more
sessions causes contention in the TCP/IP stack that can affect
bandwidth available. It's been my observation that if you set them to
match then Agent performs better.

I have a 75Mbps download capability (FiOS) and a sample about 1 hour
ago on APN only yielded 4.8Mbps download sessions from that group. I
have seen no difference in the bandwidth monitors I use when testing 1
body versus a 1900 body RAR. Remember also that RAR's are compressed
and this compression causes the effective bandwidth to be greater than
the real-time bandwidth of the NNTP channels.

My system is a Win7/x64 i7 @ 2.8GHz with 8GB RAM and a 1Gbps wired
connection to the router and it loafs doing NNTP.

It has been my experience that bandwidth utilization is highly
dependent on the load on the server and the time of day.

[DonK]

Thanks Ralph, the first section changes made no difference but the second section made a pretty
dramatic improvement. I'll do some further tweaking to see if I can optimize it.

Thank you,

Don

[DonK]

On Sat, 11 May 2013 18:13:02 -0700, Geoff <ge...@invalid.invalid> wrote:

>On Sat, 11 May 2013 15:43:21 -0700, DonK
><don8...@comcast.net.notvalid> wrote:
>
>>
>>I download a lot of movies and TV shows from a.b.movies.divx and a.b.multimedia using Agent 6.00/32,
>>Giganews with "Max connections = 20" and a 25 Mb/s Comcast connection.
>>

>>EXAMPLE 2:
>>
>>Select 1 RAR = 9 secs
>>Select 2 RAR's = 29 secs
>>Select 3 RAR's = 47 secs
>>
>>
>>Can anyone tell me why this happens and if/how I can make it faster when I download multiple
>>selections?
>>
>>Thanks
>>
>> Don
>
>Make sure the max number of connections matches the max number of
>connections allowed by your NNTP provider. Negotiating for more
>sessions causes contention in the TCP/IP stack that can affect
>bandwidth available. It's been my observation that if you set them to
>match then Agent performs better.
>
>I have a 75Mbps download capability (FiOS) and a sample about 1 hour
>ago on APN only yielded 4.8Mbps download sessions from that group. I
>have seen no difference in the bandwidth monitors I use when testing 1
>body versus a 1900 body RAR. Remember also that RAR's are compressed
>and this compression causes the effective bandwidth to be greater than
>the real-time bandwidth of the NNTP channels.
>
>My system is a Win7/x64 i7 @ 2.8GHz with 8GB RAM and a 1Gbps wired
>connection to the router and it loafs doing NNTP.
>
>It has been my experience that bandwidth utilization is highly
>dependent on the load on the server and the time of day.

Wow! 75Mbps! I'm envious.

Have you tried the second section changes from Ralph's previous thread? They've made more than a 3
fold improvement for me.

http://narkive.com/aYe01f1W

Good luck

Don

[DonK]

You're right about the number of connections and I'm sure your logic about Agent's Db is also
correct. I used to do a lot of that kind of work including some similar Db scheme's. It can work
just fine for smaller amounts of data but . . . I just checked my
"C:\Users\Don\AppData\Roaming\Forte\Agent" folder and it's 129 GB. I'd need an SSD dedicated to
Agent. Yes, I know I need to compact a lot of .dat files. :0(

These changes that Ralph suggested made a very big difference in my download efficiency:

<Ralph Fox>
1. Turn off "Automatically adjust maximum connections."
2. Try reducing "Maximum allowed connections" bit by bit,
to see if it makes any improvement.
</Ralph Fox>

Thank you for your help,

Don

[Tom Cole]

Did you also see Ralph's second post in that 3 months old thread, plus
the last post in that thread?

[Geoff]

On Sun, 12 May 2013 15:12:16 -0700, DonK
<don8...@comcast.net.notvalid> wrote:

>Wow! 75Mbps! I'm envious.
>
>Have you tried the second section changes from Ralph's previous thread? They've made more than a 3
>fold improvement for me.

I was able to get 75/35 plus TV for $5/mo. less than I was paying for
20/5 with no TV. That will change in 2 years of course. Yeah, 75Mbps
is pretty awesome. I haven't met a server yet that can fill the pipe.
My benchmark used to be an FTP server at a university near me that
used to host the Fedora Linux ISO's. It could saturate my 20Mbps FiOS
but they quit hosting big files so I have not tested it lately. Speed
tests show the system can peak at 86Mbps.

I am not a big binary downloader but I tried Agent tweaks and I set it
up for 4 sessions max with 1 reserved for priority and I get 2.5 to 5
Mbps off APN downloading a Farscape RAR. This morning during lower
activity on the server it peaked at 25Mbps and averaged about 20.

I am skeptical that getting an SSD would improve your Agent download
performance. Disk throughput is so high, 100MBps (that's BYTES per
sec, not bits), or more, which is 100 times faster than wideband
connections. You would have a hard time convincing me that it would
make a difference on a contemporary computer. Disk caching and Windows
7 NTFS performance is such that I don't think Agent on it's best days
would bottleneck on the disk.

If you really want to study it, I recommend starting "perfmon" from
the Window 7 start button and check out the performance of your system
and identify the bottlenecks. I believe it's included on all systems
now. If not, you can download it from Microsoft.