Problem when using non-standard port
huy.phan
I'm new to Disco and trying to setup a cluster with 4 machines.
Disco was installed on all the servers with command "make install
DESTDIR=/"
I also set DISCO_PORT = 6969 in /etc/disco/settings.py
Following the step "Test the system" in tutorial, I run the command
"python count_words.py http://master:6969",
I noticed that the map/reduce part was done, but when getting the
result, Disco still tried to use the standard port which is 8989.
This is a bug of Disco or I'm missing something ?
Here is the output of my wordcount:
Starting Disco job..
Go to http://master:6969 to see status of the job.
Job done. Results:
Traceback (most recent call last):
File "count_words.py", line 23, in <module>
for word, frequency in result_iterator(results):
File "/usr/lib/python2.5/site-packages/disco/core.py", line 447, in
result_iterator
res += util.parse_dir(dir_url, proxy)
File "/usr/lib/python2.5/site-packages/disco/util.py", line 93, in
parse_dir
r = download(url).splitlines()
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
31, in download
sze, fd = open_remote(url, data = data, offset = offset)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
85, in open_remote
return open_remote(url, data, ttl=ttl - 1)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
85, in open_remote
return open_remote(url, data, ttl=ttl - 1)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
85, in open_remote
return open_remote(url, data, ttl=ttl - 1)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
85, in open_remote
return open_remote(url, data, ttl=ttl - 1)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
85, in open_remote
return open_remote(url, data, ttl=ttl - 1)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
85, in open_remote
return open_remote(url, data, ttl=ttl - 1)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
85, in open_remote
return open_remote(url, data, ttl=ttl - 1)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
85, in open_remote
return open_remote(url, data, ttl=ttl - 1)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
85, in open_remote
return open_remote(url, data, ttl=ttl - 1)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
85, in open_remote
return open_remote(url, data, ttl=ttl - 1)
File "/usr/lib/python2.5/site-packages/disco/comm_httplib.py", line
81, in open_remote
(url, MAX_RETRIES, e), url)
disco.comm_httplib.CommException: HTTP exception (http://master:8989/
master/a2/wordcount@1259945135/reduce-index.txt): Downloading
http://master:8989/master/a2/wordcount@1259945135/reduce-index.txt
failed after 10 attempts: (111, 'Connection refused')
Ville Tuulos
Hi,
On Dec 4, 8:52 am, "huy.phan" <dac...@gmail.com> wrote:
> Hi all,
> I'm new to Disco and trying to setup a cluster with 4 machines.
> Disco was installed on all the servers with command "make install
> DESTDIR=/"
> I also set DISCO_PORT = 6969 in /etc/disco/settings.py
>
> Following the step "Test the system" in tutorial, I run the command
> "python count_words.pyhttp://master:6969",
> I noticed that the map/reduce part was done, but when getting the
> result, Disco still tried to use the standard port which is 8989.
> This is a bug of Disco or I'm missing something ?
Are you using Disco 0.2.3 or an older version? Apparently you have
configured your master node ("master") to act as a worker node as
well, since it tries to fetch results from http://master:8989... Prior
to 0.2.3 this was only possible if you ran two Lighttpd on the master
node on two different ports. From 0.2.3 onwards, "disco master start"
handles both the master and the worker roles and you need only a
single Lighttpd instance. In your case, Lighttpd runs on port 6969.
Since you said you have four machines in your cluster, you could try
not to use your master node as a worker node, i.e. leave it out from
the node config table. At least this might help debugging the problem.
Ville
> Here is the output of my wordcount:
>
> Starting Disco job..
> Go tohttp://master:6969to see status of the job.
> master/a2/wordcount@1259945135/reduce-index.txt): Downloadinghttp://master:8989/master/a2/wordcount@1259945135/reduce-index.txt