Bulk download BioTorrents .torrent files
50 views
Skip to first unread message
Mike Chelen
Apr 16, 2010, 11:26:53 PM4/16/10
to BioTorrents
What is the best way to download as many as possible of the
hosted .torrent files?
BioTorrents is quickly collecting interesting data sets, and a way to
process and share them in large number would help to keep up :)
The RSS feed seems to be the best export method currently, though it
includes only the most recent uploads.
A copy of all .torrent files that appear in the RSS is being collected
for bulk download: http://pub.opensci.info/torrent/biotorrents/
--
Subscription settings: http://groups.google.com/group/biotorrents/subscribe?hl=en
hosted .torrent files?
BioTorrents is quickly collecting interesting data sets, and a way to
process and share them in large number would help to keep up :)
The RSS feed seems to be the best export method currently, though it
includes only the most recent uploads.
A copy of all .torrent files that appear in the RSS is being collected
for bulk download: http://pub.opensci.info/torrent/biotorrents/
--
Subscription settings: http://groups.google.com/group/biotorrents/subscribe?hl=en
Morgan Langille
Apr 16, 2010, 11:44:20 PM4/16/10
to bioto...@googlegroups.com
Right now the default is that the RSS feed returns the number of
torrents based on the pager on the browse page. I can add a new option
to be able to choose the number of torrents to return in the RSS feed.
For now you can work around this by visiting the options on your profile
(click profile in the upper right while signed in). Then change the
setting for "Torrents per page" to something large like 200 and then
visit the browse page to see all of the torrents on a single browse
page. The RSS feed will now return all of the torrents.
Thanks for posting this to the google group.
Morgan
On 4/16/2010 8:26 PM, Mike Chelen wrote:
> What is the best way to download as many as possible of the
> hosted .torrent files?
>
> BioTorrents is quickly collecting interesting data sets, and a way to
> process and share them in large number would help to keep up :)
>
> The RSS feed seems to be the best export method currently, though it
> includes only the most recent uploads.
>
> A copy of all .torrent files that appear in the RSS is being collected
> for bulk download: http://pub.opensci.info/torrent/biotorrents/
>
>
>
--
Morgan Langille, PhD
http://www.morganlangille.com
torrents based on the pager on the browse page. I can add a new option
to be able to choose the number of torrents to return in the RSS feed.
For now you can work around this by visiting the options on your profile
(click profile in the upper right while signed in). Then change the
setting for "Torrents per page" to something large like 200 and then
visit the browse page to see all of the torrents on a single browse
page. The RSS feed will now return all of the torrents.
Thanks for posting this to the google group.
Morgan
On 4/16/2010 8:26 PM, Mike Chelen wrote:
> What is the best way to download as many as possible of the
> hosted .torrent files?
>
> BioTorrents is quickly collecting interesting data sets, and a way to
> process and share them in large number would help to keep up :)
>
> The RSS feed seems to be the best export method currently, though it
> includes only the most recent uploads.
>
> A copy of all .torrent files that appear in the RSS is being collected
> for bulk download: http://pub.opensci.info/torrent/biotorrents/
>
>
>
--
http://www.morganlangille.com
Michael Chelen
Apr 17, 2010, 2:00:57 AM4/17/10
to bioto...@googlegroups.com
The pager for RSS works great! With a script to go through each (only
two so far) all the .torrent files could be downloaded pretty
efficiently.
Adding an option for number of results would be handy, especially if
it were set through a URL parameter.
Changing profile settings is helpful too, if the RSS client can read cookies.
Thanks for explaining all the current options. Maybe a package with
all the .torrent files could be released periodically as well?
two so far) all the .torrent files could be downloaded pretty
efficiently.
Adding an option for number of results would be handy, especially if
it were set through a URL parameter.
Changing profile settings is helpful too, if the RSS client can read cookies.
Thanks for explaining all the current options. Maybe a package with
all the .torrent files could be released periodically as well?
Morgan Langille
Apr 20, 2010, 5:45:10 PM4/20/10
to BioTorrents
A URL parameter has been added to set the number of torrents to be
returned in the RSS feed. The URL parameter is "page_limit". For
example, if 25 torrents were to be obtained instead of the default 15
then you could get this by using the following url:
http://www.biotorrents.net/rss.php?page_limit=25
On Apr 16, 11:00 pm, Michael Chelen <michael.che...@gmail.com> wrote:
> The pager for RSS works great! With a script to go through each (only
> two so far) all the .torrent files could be downloaded pretty
> efficiently.
>
> Adding an option for number of results would be handy, especially if
> it were set through a URL parameter.
>
> Changing profile settings is helpful too, if the RSS client can read cookies.
>
> Thanks for explaining all the current options. Maybe a package with
> all the .torrent files could be released periodically as well?
>
returned in the RSS feed. The URL parameter is "page_limit". For
example, if 25 torrents were to be obtained instead of the default 15
then you could get this by using the following url:
http://www.biotorrents.net/rss.php?page_limit=25
On Apr 16, 11:00 pm, Michael Chelen <michael.che...@gmail.com> wrote:
> The pager for RSS works great! With a script to go through each (only
> two so far) all the .torrent files could be downloaded pretty
> efficiently.
>
> Adding an option for number of results would be handy, especially if
> it were set through a URL parameter.
>
> Changing profile settings is helpful too, if the RSS client can read cookies.
>
> Thanks for explaining all the current options. Maybe a package with
> all the .torrent files could be released periodically as well?
>
Mike Chelen
Apr 21, 2010, 10:55:49 PM4/21/10
to BioTorrents
Perfect, thanks! http://www.biotorrents.net/rss.php?page_limit=500
successfully includes every .torrent so far. Nice to see there are
several ways of accomplishing this now.
On Apr 20, 5:45 pm, Morgan Langille <morgan.g.i.langi...@gmail.com>
wrote:
successfully includes every .torrent so far. Nice to see there are
several ways of accomplishing this now.
On Apr 20, 5:45 pm, Morgan Langille <morgan.g.i.langi...@gmail.com>
wrote:
Egon Willighagen
Apr 22, 2010, 7:00:49 AM4/22/10
to BioTorrents
Mike and Morgan,
On Apr 22, 4:55 am, Mike Chelen <michael.che...@gmail.com> wrote:
> Perfect, thanks!http://www.biotorrents.net/rss.php?page_limit=500
> successfully includes every .torrent so far. Nice to see there are
> several ways of accomplishing this now.
do you plan to write up how people can use this feed to quickly set up
> several ways of accomplishing this now.
a 'mirror' using an open source tool like rtorrent?
Egon
Morgan Langille
Apr 22, 2010, 3:01:40 PM4/22/10
to bioto...@googlegroups.com
Yes, a tutorial would be nice. I am still trying to figure out the
easiest configuration possible for users to set up on a linux machine.
Windows and Mac is very easy and can be done all within uTorrent with a few simple steps (maybe I will do this now and post in the BioTorrents Forums).
As for doing this on a headless linux server it is a little more complex. Deluge seems to provide the best client features and has a really nice web interface. However, it takes quite a few steps to set up (and requires root access). Additionally, a second application (FlexGet or PodGet) needs to retrieve the torrents from the RSS feed and hand it to the Deluge torrent client. I have set this up, but it takes many steps. This tutorial seems fairly useful: http://ubuntuforums.org/showthread.php?t=1433783
Windows and Mac is very easy and can be done all within uTorrent with a few simple steps (maybe I will do this now and post in the BioTorrents Forums).
As for doing this on a headless linux server it is a little more complex. Deluge seems to provide the best client features and has a really nice web interface. However, it takes quite a few steps to set up (and requires root access). Additionally, a second application (FlexGet or PodGet) needs to retrieve the torrents from the RSS feed and hand it to the Deluge torrent client. I have set this up, but it takes many steps. This tutorial seems fairly useful: http://ubuntuforums.org/showthread.php?t=1433783
Egon Willighagen wrote:
Mike and Morgan, On Apr 22, 4:55 am, Mike Chelen <michael.che...@gmail.com> wrote:Perfect, thanks!http://www.biotorrents.net/rss.php?page_limit=500 successfully includes every .torrent so far. Nice to see there are several ways of accomplishing this now.do you plan to write up how people can use this feed to quickly set up a 'mirror' using an open source tool like rtorrent? Egon
-- Morgan Langille, PhD http://www.morganlangille.com/
Mike Chelen
Apr 26, 2010, 8:12:54 AM4/26/10
to BioTorrents
There are so many different BitTorrent programs and configurations :)
Table 1 covers the varying feature and OS support, "Comparison of
several popular BitTorrent software clients and their features"
http://www.plosone.org/article/slideshow.action?uri=info:doi/10.1371/journal.pone.0010071&imageURI=info:doi/10.1371/journal.pone.0010071.t001
Which one would be the best tutorial subject? Maybe it would help to
list existing guides such as http://ubuntuforums.org/showthread.php?t=1433783
for all the most popular software.
The guides will help new visitors gets started learning the
appropriate software. Here is documentation for RSS feeds in uTorrent:
http://www.utorrent.com/documentation/rss
Table 1 covers the varying feature and OS support, "Comparison of
several popular BitTorrent software clients and their features"
http://www.plosone.org/article/slideshow.action?uri=info:doi/10.1371/journal.pone.0010071&imageURI=info:doi/10.1371/journal.pone.0010071.t001
Which one would be the best tutorial subject? Maybe it would help to
list existing guides such as http://ubuntuforums.org/showthread.php?t=1433783
for all the most popular software.
The guides will help new visitors gets started learning the
appropriate software. Here is documentation for RSS feeds in uTorrent:
http://www.utorrent.com/documentation/rss
Morgan Langille
Apr 26, 2010, 3:21:23 PM4/26/10
to bioto...@googlegroups.com
I wrote a brief guide on the BioTorrents forum on how to use uTorrent
with RSS: http://www.biotorrents.net/forums.php?action=viewtopic&topicid=14
Also Mike, I made you a "Moderator" on BioTorrents. You have contributed
quite a bit already and I this will allow you to make edits to Forum posts.
Morgan
On 4/26/2010 5:12 AM, Mike Chelen wrote:
> There are so many different BitTorrent programs and configurations :)
>
> Table 1 covers the varying feature and OS support, "Comparison of
> several popular BitTorrent software clients and their features"
> http://www.plosone.org/article/slideshow.action?uri=info:doi/10.1371/journal.pone.0010071&imageURI=info:doi/10.1371/journal.pone.0010071.t001
>
> Which one would be the best tutorial subject? Maybe it would help to
> list existing guides such as http://ubuntuforums.org/showthread.php?t=1433783
> for all the most popular software.
>
> The guides will help new visitors gets started learning the
> appropriate software. Here is documentation for RSS feeds in uTorrent:
> http://www.utorrent.com/documentation/rss
>
>
>
--
with RSS: http://www.biotorrents.net/forums.php?action=viewtopic&topicid=14
Also Mike, I made you a "Moderator" on BioTorrents. You have contributed
quite a bit already and I this will allow you to make edits to Forum posts.
Morgan
On 4/26/2010 5:12 AM, Mike Chelen wrote:
> There are so many different BitTorrent programs and configurations :)
>
> Table 1 covers the varying feature and OS support, "Comparison of
> several popular BitTorrent software clients and their features"
> http://www.plosone.org/article/slideshow.action?uri=info:doi/10.1371/journal.pone.0010071&imageURI=info:doi/10.1371/journal.pone.0010071.t001
>
> Which one would be the best tutorial subject? Maybe it would help to
> list existing guides such as http://ubuntuforums.org/showthread.php?t=1433783
> for all the most popular software.
>
> The guides will help new visitors gets started learning the
> appropriate software. Here is documentation for RSS feeds in uTorrent:
> http://www.utorrent.com/documentation/rss
>
>
>
--
Mike Chelen
Apr 28, 2010, 11:25:20 AM4/28/10
to BioTorrents
That covers uTorrent, are there any other frequently used setups to
consider? Egon asked about rTorrent, which could be used with FlexGet,
or maybe the Deluge headless tutorial would be a better suggestion.
Is there a general site usage guideline? Thanks for mod powers, glad
to help.
On Apr 26, 3:21 pm, Morgan Langille <morgan.g.i.langi...@gmail.com>
wrote:
consider? Egon asked about rTorrent, which could be used with FlexGet,
or maybe the Deluge headless tutorial would be a better suggestion.
Is there a general site usage guideline? Thanks for mod powers, glad
to help.
On Apr 26, 3:21 pm, Morgan Langille <morgan.g.i.langi...@gmail.com>
wrote:
> I wrote a brief guide on the BioTorrents forum on how to use uTorrent
> with RSS:http://www.biotorrents.net/forums.php?action=viewtopic&topicid=14
>
> Also Mike, I made you a "Moderator" on BioTorrents. You have contributed
> quite a bit already and I this will allow you to make edits to Forum posts.
>
> Morgan
>
> On 4/26/2010 5:12 AM, Mike Chelen wrote:
>
>
>
> > There are so many different BitTorrent programs and configurations :)
>
> > Table 1 covers the varying feature and OS support, "Comparison of
> > several popular BitTorrent software clients and their features"
> >http://www.plosone.org/article/slideshow.action?uri=info:doi/10.1371/...
> with RSS:http://www.biotorrents.net/forums.php?action=viewtopic&topicid=14
>
> Also Mike, I made you a "Moderator" on BioTorrents. You have contributed
> quite a bit already and I this will allow you to make edits to Forum posts.
>
> Morgan
>
> On 4/26/2010 5:12 AM, Mike Chelen wrote:
>
>
>
> > There are so many different BitTorrent programs and configurations :)
>
> > Table 1 covers the varying feature and OS support, "Comparison of
> > several popular BitTorrent software clients and their features"
>
> > Which one would be the best tutorial subject? Maybe it would help to
> > list existing guides such ashttp://ubuntuforums.org/showthread.php?t=1433783
> > Which one would be the best tutorial subject? Maybe it would help to
> > for all the most popular software.
>
> > The guides will help new visitors gets started learning the
> > appropriate software. Here is documentation for RSS feeds in uTorrent:
> >http://www.utorrent.com/documentation/rss
>
> --
> Morgan Langille, PhDhttp://www.morganlangille.com
>
> > The guides will help new visitors gets started learning the
> > appropriate software. Here is documentation for RSS feeds in uTorrent:
> >http://www.utorrent.com/documentation/rss
>
> --
Morgan Langille
Apr 28, 2010, 3:24:27 PM4/28/10
to bioto...@googlegroups.com
I would like to have a tutorial using Deluge. However, I am still not
sure what the easiest set up is using deluge.
I followed the tutorial that you pointed out earlier
(http://ubuntuforums.org/showthread.php?t=1433783) and it worked fine,
but it felt quite clunky and required many root commands (due to the
making of a new "deluge" user). I think I would like to running it as a
normal user.
Morgan
--
Morgan Langille, PhD
http://www.morganlangille.com/
sure what the easiest set up is using deluge.
I followed the tutorial that you pointed out earlier
(http://ubuntuforums.org/showthread.php?t=1433783) and it worked fine,
but it felt quite clunky and required many root commands (due to the
making of a new "deluge" user). I think I would like to running it as a
normal user.
Morgan
Morgan Langille, PhD
http://www.morganlangille.com/
Morgan Langille
May 13, 2010, 6:03:37 PM5/13/10
to BioTorrents
In case people are still interested I added another tutorial on using
the BitTorrent client Deluge with FlexGet:
http://www.biotorrents.net/forums.php?action=viewtopic&topicid=17
The most useful part of the tutorial is what to put in the config file
for FlexGet.
Morgan
On Apr 28, 12:24 pm, Morgan Langille <morgan.g.i.langi...@gmail.com>
the BitTorrent client Deluge with FlexGet:
http://www.biotorrents.net/forums.php?action=viewtopic&topicid=17
The most useful part of the tutorial is what to put in the config file
for FlexGet.
Morgan
On Apr 28, 12:24 pm, Morgan Langille <morgan.g.i.langi...@gmail.com>
Reply all
Reply to author
Forward
0 new messages