About torrents and twister 0.9.18

[Miguel Freitas]

Debugging the messages (posts) propagation issues during the last few days is being a great learning experience and insightful about the best twister strategies.

twister basically has the problem of distributing torrents (where user’s posts are stored) among peers in a way that is both fair and secure. fair: each user will contribute their small share of processing and storage (keep your twister client running!). secure: prevent denial-of-services as using too much resources.

libtorrent introduces a nice concept for this purpose called “auto managed torrents” (*). Auto managed torrent have limits on the number of simultaneous active torrents, thus keeping resource usage limited, and may decide when to pause/resume each torrent.

Unfortunately twister was not rotating torrents to alternate between active and paused ones. Last torrents to be added where kept queued during the entire execution time of twisterd. Thanks @stark and @kseistrup for the help with debugging which lead to understanding this.

I don’t need to explain that this twister bug caused it to be secure but not fair. So I’ve just committed the code to rotate torrents in queue.

A lot of posts should start to properly propagate now. Welcome to twister-core 0.9.18.

Please download and update (git pull && make).

thanks and have fun.

* btw: libtorrent’s author – Arvid Norberg – deserves a lot of twister credit as it wouldn’t be possible to develop this project in such short time without libtorrent.

[Miguel Freitas]

There is another trade-off I'd like to discuss. Or at least make people aware about it.

In recent 0.9.18 all torrents are auto-managed and made to rotate. This includes both the torrents we follow and the torrents we contribute as trackers.

The problem is that we may lose the instant notification as some torrents we follow may be paused at the time a new post is sent. This might add very long delays to your timeline that i don't find acceptable.

To overcome this I have just committed a small change to prevent the torrents you follow from being auto-managed. Users are invited to give it a try and let me know how it works.

I think the premise is good: if you want to follow more users and be instantly notified about their posts, you should also be willing to use more resources of your computer for this. The resources you contribute back to the network (to the other users) are still limited.

There are however some limits that will be reached sooner or later. If you follow too many users (how many? i don't know yet), *something* is going to be exhausted. If nothing else, at some point we should be running out of open file descriptors.

Another possible improvement: in future, the user might elect a few torrents (users) he wants the instant notification to work, and the rest of his following would be given lower priority to rotate.

regards,

Miguel

[Arco Mul]

May I wonder how long such rotations take? If someone is following 100 users, how long would it take to rotate them all? This might be depending on the bandwidth I guess, but on average?

Besides that I'm just curious about it. If it is a reasonable amount of time we could choose to show it in the UI:

Syncing - 2 minutes left...

Not really pretty of course, but as you say when someone wants to follow a lot of people it might be an idea. Since it is also confusing when you keep on getting new posts for a couple of minutes when you just started your client.

Regards, Arco

--

You received this message because you are subscribed to the Google Groups "twister-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to twister-user...@googlegroups.com.
To post to this group, send email to twiste...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Klaus Alexander Seistrup]

Is that the reason twister seems noticably more memory hungry since v0.9.18 (I'm following around 250 accounts)?

[Miguel Freitas]

---------- Forwarded message ----------
From: Miguel Freitas <mfre...@gmail.com>
Date: Tue, Feb 25, 2014 at 11:08 AM
Subject: Re: [twister-dev] Re: [twister-users] Re: About torrents and twister 0.9.18
To: twist...@googlegroups.com

On Tue, Feb 25, 2014 at 9:29 AM, Arco Mul <m...@arcomul.nl> wrote:

May I wonder how long such rotations take? If someone is following 100 users, how long would it take to rotate them all? This might be depending on the bandwidth I guess, but on average?

Sure, that's a good question! thanks for asking ;-)

Currently we rotate every 30 seconds (the first active is sent to the last of the queue). The number of active downloads is set to 20.

So it will take 100 * 30s = 50 min to rotate everything. Each torrent in rotation remains active for 20 * 30s = 10 minutes.

We may change these parameters freely. I would just avoid that one torrent stays active for too little time, as we have all the overhead of establishing connections, asking for pieces etc. So if active time is too short (how short?) it might not even let it to synchronize properly.

 

Besides that I'm just curious about it. If it is a reasonable amount of time we could choose to show it in the UI:

Syncing - 2 minutes left...

I'd rather not have any delay to update the users you follow. Unless we realize such a thing would become too costly.

Of course, we may try other strategies to rotate torrents though. For example: try to detect we have just finished sync'ing a given torrent (we have connections to several peers but no pieces have been sent/received for a while) and move it to the end of the queue. No need to wait until it moves through all the 20 download positions.

regards,

Miguel

[Miguel Freitas]

On Tue, Feb 25, 2014 at 2:10 PM, Klaus Alexander Seistrup <kl...@seistrup.dk> wrote:

Is that the reason twister seems noticably more memory hungry since v0.9.18 (I'm following around 250 accounts)?

Most likely, yes. How much memory? This is an interesting figure.

regards,

Miguel

[Klaus Alexander Seistrup]

Approximately 2-3 times the amount of previous versions.  It's hard to say in absolute numbers, since the amount seems to increase over time (and besides, I just restarted twisterd in order to reclaim some memory).

In ~/.twister/twister.conf I can specify  “maxconnections=N”, but that number will only affect the number of “bitcoin” connections.  Is there a similar value for torrent connections, or will it have to be managed automatically?

[Miguel Freitas]

On Tue, Feb 25, 2014 at 2:57 PM, Klaus Alexander Seistrup <kl...@seistrup.dk> wrote:

Approximately 2-3 times the amount of previous versions.  It's hard to say in absolute numbers, since the amount seems to increase over time (and besides, I just restarted twisterd in order to reclaim some memory).

In ~/.twister/twister.conf I can specify  “maxconnections=N”, but that number will only affect the number of “bitcoin” connections.  Is there a similar value for torrent connections, or will it have to be managed automatically?

Yes, in libtorrent/session.cpp (about the line 1238) there are some limits for auto manager: active_downloads, active_limit etc. Also connections_limit (line 1324).

These settings are currently not being read from the config file.

regards,

Miguel

[Klaus Alexander Seistrup]

Sure, I'll give it a try.