Help wanted with metalink client for Wubi

[ago]

Hi all,

I am Wubi author (http://wubi-installer.org). As some of you may know,
wubi is a windows installer for Ubuntu, which will hopefully become a
official installer in next release. Wubi at the moment uses metadl as
a download manager. Unfortunately Hampus, metadl author, informed us
he will have little time to keep supporting metadl going forward. We'd
be glad if somebody could help us with metadl (in particular helping
to add segmented downloads) or suggesting another download manager.
Prerequisites for us:

1) must run on all windows versions
2) small dll
3) checksums (possibly partial checksums)
4) segmented downloads
5) support metalink files

Additional points if an nsis plugin is available (otherwise we can
adapt the one written by Hampus) and/or if a bittorrent client is
included.

Thanks in advance,

Ago

[seba...@willemijns.com]

On 11 déc, 01:23, ago <agostino.ru...@gmail.com> wrote:

> 1) must run on all windows versions
> 2) small dll
> 3) checksums (possibly partial checksums)
> 4) segmented downloads
> 5) support metalink files

@cls
@echo.
@echo please wait a while to download ubuntu on your computer
@echo.
@start aria2c.exe http://test-debit.free.fr/1024.rnd

ask to TT (which is the japanese author of aria2) to add a function to
display verbose in a lower stat (do not display
502, 404 errors in the console) because ubuntu have more than 100
mirrors and a few of it which is down is not important ;)

[ago]

I agree that aria2 is a natural candidate, in fact I tried to contact
the authors on SF forums some time ago'.
It'd be lovely if it was available as a dll/nsis plugin and if it
could be squeezed down a bit.
For instance possibly libxml2 could be replaced with expat (http://
expat.sourceforge.net/) which is far lighter.
As mentioned, for us size does matter (wubi will be also on the ubuntu
CD and there is a lot
of competition for space). Expat is what metadl uses by the way.

On Dec 11, 11:58 am, "sebast...@willemijns.com"

<sebast...@willemijns.com> wrote:
> On 11 déc, 01:23, ago <agostino.ru...@gmail.com> wrote:
>
> > 1) must run on all windows versions
> > 2) small dll
> > 3) checksums (possibly partial checksums)
> > 4) segmented downloads
> > 5) support metalink files
>
> @cls
> @echo.
> @echo please wait a while to download ubuntu on your computer
> @echo.

> @start aria2c.exehttp://test-debit.free.fr/1024.rnd

[Anthony Bryan]

On Dec 11, 2007 6:50 PM, ago <agostin...@gmail.com> wrote:
>
> I agree that aria2 is a natural candidate, in fact I tried to contact
> the authors on SF forums some time ago'.
> It'd be lovely if it was available as a dll/nsis plugin and if it
> could be squeezed down a bit.
> For instance possibly libxml2 could be replaced with expat (http://
> expat.sourceforge.net/) which is far lighter.
> As mentioned, for us size does matter (wubi will be also on the ubuntu
> CD and there is a lot
> of competition for space). Expat is what metadl uses by the way.
>
> On Dec 11, 11:58 am, "sebast...@willemijns.com"
> <sebast...@willemijns.com> wrote:
> > On 11 déc, 01:23, ago <agostino.ru...@gmail.com> wrote:
> >
> > > 1) must run on all windows versions
> > > 2) small dll
> > > 3) checksums (possibly partial checksums)
> > > 4) segmented downloads
> > > 5) support metalink files

Tatsuhiro, aria2's author is on this list & cc'ed here. you might also
want to get in touch w/ Ross Smith who does the win32 port.

aria2 meets all your needs, except maybe small size (depending on your
definition of small). it's about 1.85 MB compressed and maybe they
could get that smaller.

it also has the added benefit of being able to download from both
BitTorrent and mirrors at the same time, if you want to.

I don't know about dll/nsis plugins, or switching to expat (that would
be up to the author).

--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads

[ago]

> Tatsuhiro, aria2's author is on this list & cc'ed here. you might also
> want to get in touch w/ Ross Smith who does the win32 port.

Tatsuhiro and Ross Smith, ping

> aria2 meets all your needs, except maybe small size (depending on your
> definition of small). it's about 1.85 MB compressed and maybe they
> could get that smaller.

As mentioned that would be quite desirable, libxml2 seems like a first
good target to me.

> I don't know about dll/nsis plugins, or switching to expat (that would
> be up to the author).

I can help with the nsis plugin, Hampus provided a nice wrapper for
that

Thanks for everything by the way,

Ago

[Nicolas]

On Dec 11, 8:50 pm, ago <agostino.ru...@gmail.com> wrote:
> For instance possibly libxml2 could be replaced with expat (http://
> expat.sourceforge.net/) which is far lighter.

That would need a total rewrite of the metalink parsing code... Expat
is a streaming-push parser, libxml2 is tree-based. In human-readable
terms: expat tells you when it finds start tags, text, attributes,
etc. in the order they are found; libxml2 reads the whole file into a
memory structure and then lets you read the structure (in any order).

[Hampus Wessman]

ago skrev:

I don't know about dll/nsis plugins, or switching to expat (that would
be up to the author).
    

I can help with the nsis plugin, Hampus provided a nice wrapper for
that
  

I actually tried to make a NSIS plugin for aria2 once. The problem I ran into was that there is no way (and I mean NO way, at least AFAIK and without some really ugly hacks) to send an interrupt signal (as when the user presses Ctrl+Break) to command line apps in windows unless you run the two apps (the sender and the receiver) in the same console window (only the standard windows console works here). Right now I forget how that last thing worked, but we don't want to pop up a console window in Wubi anyway. You can force a process to terminate, but in that case aria2 doesn't save any resume data (at least not when I tried) and therefore that's not an option either.

I did some research back then and I don't remember every detail, but I think this was what was going on. In linux you could start aria2 as a child process and then send a SIGINT signal to it when you wanted it to terminate. Then aria2 would catch this and save resume data on the disk, before terminating. Unfortunately Windows doesn't support the SIGINT signal and there is no real alternative either. There's the TerminateProcess function, but it just forces the process to terminate, so that's no good.
More info here:
http://msdn2.microsoft.com/en-us/library/xdkz3x12(vs.71).aspx
http://en.wikipedia.org/wiki/SIGINT_%28POSIX%29

I never contacted Tatsuhiro or Ross Smith back then, but of course this could surely be solved somehow. One possible solution I can think of is to make it possible for the "parent" application to send a message over a pipe to standard in or something like that. All that would be needed is another way (besides SIGINT signals) to tell aria2 to exit. Perhaps Tatsuhiro and/or Ross Smith could come up with something? I don't know how hard it would be to implement something like this in aria2, because I haven't looked at the source code. Someone (me or ago, perhaps?) should probably speak to Ross Smith, because this is first of all a win32 issue... What do the rest of you think? What would be the best solution?

If nothing else I will most probably have time to finish metadl 2 next summer, if not before. Right now it's mostly in a planning stage and as ago said earlier I won't have much time to work on metadl, so other solutions are welcome. If finished, it will support segmented downloading, but not bittorrent (to begin with at least). It would be smaller than aria2, but a bit larger than the old metadl, because I'm rewriting it in C++ (and the C++ standard libs seems to be a bit larger than the ones for C). Much of the code would need to be rewritten anyway, because with segmented downloading the plugin needs to switch from the "easy" to the "multi" interface of libcurl and change much of the logic...
More info here: http://hampus.vox.nu/metadl/

I can help with making an aria2 wrapper plugin for NSIS if the issues described above is sorted out somehow. That would be the easiest solution and aria2 is a really good downloader too! Most of the plugin code is already written, as ago said, so that wouldn't take much time...

[Tatsuhiro]

aria2 uses libxml2's SAX parser which behaves like streaming push
parser.
I checked expat resource and found that callback handlers are mostly
the same with libxml2's
SAX handler, so I can reuse most of the code. I think supporting expat
is relatively easy.

[ago]

Sounds great, do you think would it also be possible to have aria2 as
a (windows) library, with callbacks for progress status and hooks for
cancelling the download? That would make the integration with nsis and
other programs much easier. For the reasons illustrated by Hampus, a
command line tool is suboptimal in our case.

[Hampus Wessman]

That would be a great solution. I think it would be useful for many
people! Right now the only C/C++ download library I know of is libcurl
and it doesn't support things like segmented downloading on its own...

ago skrev:

[Tatsuhiro]

I agree that solution is great,,but sadly though, currently aria2 is
not designed in that way,..
I think it is my long term goal.

I searched web and found that perl and ruby provide kill function/
command in their win32/win32.c
http://www.google.com/codesearch?hl=ja&q=+win32/win32.c+ruby+show:La3HalH7C3w:wRyn2Rs7zr8:l4DgcFesN4c&sa=N&cd=5&ct=rc&cs_p=cvs+:pserver:anonymous%40cvs.ruby-lang.org:/src+ruby&cs_f=win32/win32.c#first

Is there anyone who tried it?

As for expat, I successfully built aria2 with expat a yesterday.
Not yet committed to svn trunk.

[Hampus Wessman]

I kind of guessed it was that way. I'm sure there is a simpler solution
that will do for now (perhaps the one below). I looked at the src code
you linked to. They use the GenerateConsoleCtrlEvent API call to send
SIGINTs to the target process. Unfortunately that only seems to work if
the both apps are running in the same console, so it doesn't work for a
GUI app (AFAIK).
See http://msdn2.microsoft.com/en-us/library/ms683155.aspx

I found something else that is really interesting, though. I don't think
I've tried to do this, so it might work. It seems like a Ctrl+C sent to
stdin will be translated into a SIGINT signal automatically by Windows
(it remains a mystery why they didn't create a nice api function for
doing this, though). This behaviour apparently depends on the "console
mode", but the default is that it's enabled.
See here:
http://msdn2.microsoft.com/en-us/library/ms686033.aspx
http://msdn2.microsoft.com/en-us/library/ms683462.aspx

I'll put together a small test to see if this works... This would be a
really simple solution to the problem!

Tatsuhiro skrev:

[Hampus Wessman]

I made a python script that starts aria2 as a subprocess and creates a
pipe to its stdin. Nothing happens when I send Ctrl+C (i.e. ASCII code
3). Perhaps I'm doing something wrong, but otherwise that didn't work
either... Does anyone else manage to get that working?

Perhaps the Windows version of aria2 could be made to read stdin (poll
it every now and then) and translate e.g. ^C (or something else, so it
doesn't collide with Windows in any way) to an interrupt handler call?
Would that be hard to add?

There is a frontend for aria2 already, that works in Windows
(aria2::gui), right? Does anyone know how it cancels a download? After
looking at a few screenshots I get the feeling that it doesn't...

Hampus Wessman skrev:

[Sebastien WILLEMIJNS]

On Thu, 13 Dec 2007 19:49:27 +0100, "Hampus Wessman" <h...@vox.nu> said:
>
> I made a python script that starts aria2 as a subprocess and creates a
> pipe to its stdin. Nothing happens when I send Ctrl+C (i.e. ASCII code
> 3). Perhaps I'm doing something wrong, but otherwise that didn't work
> either... Does anyone else manage to get that working?

D:\aria2>d:\aria2\aria2c.exe http://test-debit.free.fr/1024.rnd
[#1 SIZE:80.0KiB/1.0MiB(7%) CN:1 SPD:491.43KiB/s ETA:01s]
2007-12-13 20:08:36 NOTICE - Shutdown sequence commencing... Press
Ctrl-C again
for emergency shutdown.
[#1 SIZE:592.0KiB/1.0MiB(57%) CN:0 SPD:0.00KiB/s]
Download Results:
(OK):download completed.(ERR):error occurred.(INPR):download
in-progress.
gid|stat|path/URI
===+====+======================================================================
1|INPR|http://test-debit.free.fr/1024.rnd

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in
help/man page
for details.
Terminer le programme de commandes (O/N) ?

> Perhaps the Windows version of aria2 could be made to read stdin (poll
> it every now and then) and translate e.g. ^C (or something else, so it
> doesn't collide with Windows in any way) to an interrupt handler call?
> Would that be hard to add?

i do not understood what do you want (bad english for me i think) but
FYI cancelling
is done in 2 steps

2007-12-13 20:08:36 NOTICE - Shutdown sequence commencing... Press
Ctrl-C again
for emergency shutdown.

if you ask how to detect a CTRL+C:
- ask errorlevel management ?
- read a variable %ERRORLEVEL ?
- easy temporary hack is to create a temporary file which contains
scancode "ctrl-c" on file content ?

[Tatsuhiro]

On 12月14日, 午前3:49, Hampus Wessman <h...@vox.nu> wrote:
> I made a python script that starts aria2 as a subprocess and creates a
> pipe to its stdin. Nothing happens when I send Ctrl+C (i.e. ASCII code
> 3). Perhaps I'm doing something wrong, but otherwise that didn't work
> either... Does anyone else manage to get that working?
>
> Perhaps the Windows version of aria2 could be made to read stdin (poll
> it every now and then) and translate e.g. ^C (or something else, so it
> doesn't collide with Windows in any way) to an interrupt handler call?
> Would that be hard to add?

OK, I created a patch to poll stdin. The patch is uploaded in this
discussion forum's file section. I tested on linux, but not on
windows.

>i do not understood what do you want (bad english for me i think) but
>FYI cancelling
>is done in 2 steps

If you send ETX twice, then it enters emergency shutdown mode.

BTW, I've done the job to support expat in aria2. Now available in svn
trunk.

[Sebastien WILLEMIJNS]

On Fri, 14 Dec 2007 07:23:34 -0800 (PST), "Tatsuhiro"
<tatsu...@gmail.com> said:
> OK, I created a patch to poll stdin.

what is this ? ;)

[Tatsuhiro]

On 12月15日, 午前5:33, "Sebastien WILLEMIJNS" <sebast...@willemijns.com>
wrote:

In windows, you can send SIGINT signal by pressing ctrl-c in the dos
prompt, but you cannot send SIGINT signal programmattically to the
aria2 process.

Hampus found the way to send ETX character(ctrl-c) to aria2's stdin,
so I wrote a patch to detect it and call signal handler as if ctrl-c
is pressed by user.

[Sebastien WILLEMIJNS]

On Fri, 14 Dec 2007 22:22:55 -0800 (PST), "Tatsuhiro"
<tatsu...@gmail.com> said:
> In windows, you can send SIGINT signal by pressing ctrl-c in the dos

> prompt, [...]

thanks of you answer,

[Rene Leonhardt]

Hello Tatsuhiro,

I am wondering why Flashgot only mentions the old Aria RPM download
client?
http://flashgot.net/
Are you in contact with the Flashgot developers about supporting Aria2
for Linux, Windows and Mac OS X?

And why does Aria RPM states that Aria development stopped in December
2002?
http://aria-rpm.sourceforge.net/features.php

[Sebastien WILLEMIJNS]

On Sat, 15 Dec 2007 01:57:55 -0800 (PST), "Rene Leonhardt"
<rene.le...@googlemail.com> said:
> I am wondering why Flashgot only mentions the old Aria RPM download
> client?
>

> And why does Aria RPM states that Aria development stopped in December
> 2002?
> http://aria-rpm.sourceforge.net/features.php

aria is stopped. aria2 not ;) it is true flashgot can purpose aria2 now
;)

[Hampus Wessman]

Tatsuhiro skrev:

> On 12月14日, 午前3:49, Hampus Wessman <h...@vox.nu> wrote:
>
>> I made a python script that starts aria2 as a subprocess and creates a
>> pipe to its stdin. Nothing happens when I send Ctrl+C (i.e. ASCII code
>> 3). Perhaps I'm doing something wrong, but otherwise that didn't work
>> either... Does anyone else manage to get that working?
>>
>> Perhaps the Windows version of aria2 could be made to read stdin (poll
>> it every now and then) and translate e.g. ^C (or something else, so it
>> doesn't collide with Windows in any way) to an interrupt handler call?
>> Would that be hard to add?
>>
>
> OK, I created a patch to poll stdin. The patch is uploaded in this
> discussion forum's file section. I tested on linux, but not on
> windows.
>

That's great! This can't fail... I haven't tried it yet, because I
haven't managed to compile it. When I have managed to get it working
(it's easier to cross compile in linux, will try that) I'll make a
simple nsis plugin for it. I'll tell you about the results later!

[Hampus Wessman]

Hampus Wessman skrev:

Was harder to compile aria2 for Windows than I thought. I don't have
time to work on it, but I think this is the best/easiest way to make a
NSIS plugin (or other Windows frontend) for aria2. Perhaps someone else
(ago?) is willing to create a plugin for the patched aria2. I won't be
working on it (for the coming weeks at least; have other things that I
need to do). It would be great if someone did. In that case, don't
forget to post something about it here, when you are finished. I'd love
to hear about it!

[ago]

Thanks to Hampus and Tatsuhiro,

Looks like very encouraging progress. I must admint I haven't looked
at the code, but I am not sure I understood why it's not possible to
split aria2 into a library + executable, that would make it more
flexible and useful and probably easier to maintain.

On our side cancelling the download is one of the issues, it's also
important to provide feedback to users for long running tasks, and
having to parse piped text is doable but less than optimal,
particularly if such strings are localized. Is all the output to
stdout/stderror "safe" for automated processing in this respect?

Re windows (cross?)compilation, I haven't tried that either, but if
Hampus finds it difficult, it's unlikely to be any easier for me. It
might be a good idea to start testing with a simple linux library
which calls aria2c as an external application via pipes and provides
appropriate hooks (to which we can attach nsis later on), if that
works well, porting to windows and writing an nsis wrapper should not
be too much trouble.


Ago

> to hear about it!- Hide quoted text -
>
> - Show quoted text -

[Tatsuhiro]

> On our side cancelling the download is one of the issues, it's also
> important to provide feedback to users for long running tasks, and
> having to parse piped text is doable but less than optimal,
> particularly if such strings are localized. Is all the output to
> stdout/stderror "safe" for automated processing in this respect?

Some notification messages are localized, but the progress indicator
is not localized, just an ascii text.
If you desire, I can add an option to hide localized messages.
I can also add console readout that is more parser friendly format.

> at the code, but I am not sure I understood why it's not possible to
> split aria2 into a library + executable, that would make it more
> flexible and useful and probably easier to maintain.

I was short of words in my previous posts.
Make a library is not just splitting up aria2 into library and
executable part.
I must choose aria2 api carefully and provide the developers header
files of aria2 api. Adding aria2's original namespace to all the
source files is also required to avoid class name collisions.
I don't say they are not possible and the idea is great, but I think
it takes time and I cannot say when it will be done.

[Sebastien WILLEMIJNS]

On Fri, 21 Dec 2007 06:19:39 -0800 (PST), "Tatsuhiro"
<tatsu...@gmail.com> said:
> I don't say they are not possible and the idea is great, but I think
> it takes time and I cannot say when it will be done.

IMHO i think curl is a well known tool thanks to the "libary mode" ;)

[ago]

> If you desire, I can add an option to hide localized messages.
> I can also add console readout that is more parser friendly format.

Yes that would be most welcome

On a side note, I have a few questions:

1) Metadl detects whether a source file is altready available in the
target directory and runs a checksum on the target, if the checksum
fails it downloads whatever it needs otherwise it skips. Is something
like that available in aria2?

2) Has anybody managed to crosscompile aria2 for windows on a linux
box?

3) Is there any build option I should consider to save space?

Thanks in advance

Ago

[Tatsuhiro]

On 1月10日, 午後10:12, ago <agostino.ru...@gmail.com> wrote:
> > If you desire, I can add an option to hide localized messages.
> > I can also add console readout that is more parser friendly format.
>
> Yes that would be most welcome
>

OK, I'll post an idea of readout format.
Do you have any idea or request for this?

> On a side note, I have a few questions:
>
> 1) Metadl detects whether a source file is altready available in the
> target directory and runs a checksum on the target, if the checksum
> fails it downloads whatever it needs otherwise it skips. Is something
> like that available in aria2?
>

Yes, but aria2 uses chunk checksum like BitTorrent, not one checksum
for 1 file.
aria2 downloads only the part of file where chunk checksum
verification fails.

> 2) Has anybody managed to crosscompile aria2 for windows on a linux
> box?
>

aria2 win32 package in sf.net is made by Ross.
http://smithii.com/aria2

> 3) Is there any build option I should consider to save space?
>

If you don't require BitTorrent functionality, you can drop that by
giving --disable-bittorrent to configure script.

[Sebastien WILLEMIJNS]

On Fri, 11 Jan 2008 19:52:04 -0800 (PST), "Tatsuhiro"
<tatsu...@gmail.com> said:

> > 2) Has anybody managed to crosscompile aria2 for windows on a linux
> > box?
>
> aria2 win32 package in sf.net is made by Ross.
> http://smithii.com/aria2

but ross can compile win32 version under a win OS ;)

>
> > 3) Is there any build option I should consider to save space?
> >
> If you don't require BitTorrent functionality, you can drop that by
> giving --disable-bittorrent to configure script.

??????? where is the gain ? in a hidden temp file or directory ?????

[Hampus Wessman]

Sebastien WILLEMIJNS skrev:

Should make the executable smaller. Not that important if you're just
going to use it yourself, but can be useful if you're going to
distribute it over the net (like Wubi). Probably makes the memory
foot-print smaller too.

[Sebastien WILLEMIJNS]

On Sat, 12 Jan 2008 09:16:29 +0100, "Hampus Wessman" <h...@vox.nu> said:

> Should make the executable smaller. Not that important if you're just
> going to use it yourself, but can be useful if you're going to
> distribute it over the net (like Wubi). Probably makes the memory
> foot-print smaller too.

hmmm.... wubi or equivalent tools will prefer static links (FTP/HTTP)
why not ? but they are too two another "concepts" (bt+http or bt+ftp)
instead of classical aria2c http/ftp/bt protocols...

[Hampus Wessman]

Sebastien WILLEMIJNS skrev:

Personally I think it would be cool if Wubi used all three. We haven't
got it working at all yet, though :)

[Sebastien WILLEMIJNS]

if you speak to stay light aria2c core in wubi you must delete some
functiond on it ;)

[Hampus Wessman]

Yeah. That's a problem. It might be worth the extra space for the added functionality, though. Right now we don't even have segmented downloading...

[Anthony Bryan]

Wubi update: it's now on the alpha release of the Ubuntu CDs. congrats guys!

2008-02-25: Wubi now ships with Ubuntu Live CD 8.04 Alpha5, also
available in stand-alone mode here:
http://people.ubuntu.com/~evand/wubi Please report any feedback in the
wubi forum ( http://ubuntuforums.org/forumdisplay.php?f=234 )

http://www.downloadsquad.com/2008/02/23/ubuntu-8-04-alpha-5-released/

--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads