how does disco take in input?

[Carlo C.]

Hi,

Can someone explain how disco takes in input? The docs are a little
vague (http://discoproject.org/doc/py/core.html#disco.core.Job). If
you specify a url as input, does the master pull that url and
distribute chunks to the workers or does each worker pull a copy of
the input? Also, what's the disco:// protocol?

[Paweł Reef]

One worker fetches the whole file from url. If You want multiple
workers to work on a single file, You need to split it by Yourself.

Currently, disco://cnode03/bigtxt/file_name is an alias for
http://cnode03:[DISCO_PORT]/bigtxt/file_name. You could use put a file
on one of the nodes and use it as input for a job on other nodes.

[Luis Sisamon]

Would it be possible to modify Disco to, for example, read compressed
files?

On Dec 27 2009, 3:19 pm, Paweł Reef <reefia...@gmail.com> wrote:
> One worker fetches the whole file from url. If You want multiple
> workers to work on a single file, You need to split it by Yourself.
>

> Currently, disco://cnode03/bigtxt/file_name is an alias forhttp://cnode03:[DISCO_PORT]/bigtxt/file_name. You could use put a file

[Paweł Reef]

You could implement a map_reader capable of reading entries in any
format (compressed too).
See map_reader parameter here:
http://discoproject.org/doc/py/core.html#disco.core.Job

[orac....@gmail.com]

Im not sure, if I understand this correctly. In word_count example, input is url: http://discoproject.org/media/text/chekhov.txt. 

So, data is read in chunks using HTTP GET and passed directly to map_reader without pushing it to DDFS. Input file size is not a problem, because file is read in chunks. This procedure reads file over HTTP, line by line, the same as it would be read from DDFS. 

Is this the way disco reads file over HTTP?

Dne sreda, 20. januar 2010 23:02:36 UTC+1 je oseba Reef napisala:

[orac....@gmail.com]

In count_words.py example Disco gets following URL on input: http://discoproject.org/media/text/chekhov.txt

If I understand this correctly, firstly, Disco makes HTTP GET request with a file URL and gets in response a chunk of a file. Chunk is passed directly to map_readers and it is not stored to DDFS. This process repeats till end of a file. Because Disco is reading file chunk by chunk, there is no upper file size limit.  

Dne sreda, 20. januar 2010 23:02:36 UTC+1 je oseba Reef napisala:

You could implement a map_reader capable of reading entries in any

[Jens Rantil]

Hi Orac,

The file is not read in chunks. It is read contiguously using a single GET request done by a single map instance.

This means that the job will have not have any concurrency. If you'd like to have concurrency you would want to give the job multiple HTTP input URLs.

I hope this answers your question,

Jens

—
Sent from Mailbox for iPhone

--
You received this message because you are subscribed to the Google Groups "Disco-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to disco-dev+...@googlegroups.com.
To post to this group, send email to disc...@googlegroups.com.
Visit this group at http://groups.google.com/group/disco-dev.
For more options, visit https://groups.google.com/groups/opt_out.