always ready in status?

[Christian]

Hi,

i try to adpat this nice jug functionality to something like string
comparing.
But after I execute the file the status is always on read? What might
be my mistake.

Many thanks,
Christian


Task name Waiting Ready
Finished Running
----------------------------------------------------------------------------------------
fpe_multi.join 1
0 0 0
fpe_multi.similarity 0
100000 0 0
........................................................................................
Total: 1
100000 0 0



import Levenshtein
import jug
import random
import time
from itertools import product,chain
from jug import TaskGenerator


def generate_string_simple(n):
lstring = ""
for i in range(n):
""" letters a to z are ascii 97 to 122 """
r = random.choice([random.randint(97,122)])
lstring += chr(r)
return lstring



def produce_example_combos(str_size=12,na=1000,nb=100):
a = [generate_string_simple(str_size) for x in xrange(na)]
b = [generate_string_simple(str_size) for x in xrange(nb)]
it = product(a,b)
return it



@TaskGenerator
def similarity(elements):
ratio = Levenshtein.ratio(elements[0],elements[1])
return ratio


@TaskGenerator
def join(partials):
return list(chain(*partials))


ratio_array =[]
#combos = produce_example_combos()
#print "tesdata generated"
#start_time = time.time()
for i in produce_example_combos():
elements = i
#print elements
ratio= similarity(i)
ratio_array.append([elements[0],elements[1],ratio])

fullresults = join(ratio_array)

[Luis Pedro Coelho]

Hi,

This should work on real data, but, on random input, this is actually
expected.

See the FAQ:

http://packages.python.org/Jug/faq.html#it-doesn-t-work-with-random-input

HTH
Luis
--

LxMLS 2012: Lisbon Machine Learning School
http://lxmls.it.pt

--
Luis Pedro Coelho | Institute for Molecular Medicine | http://luispedro.org

[Christian]

Uups I see. And what might be the problem when I use the backend with
redis (2.4.10) with your decrypt example.
Last but not least , do you have any hint about the time issue for
"jug status", it takes "very long" for execution in my example.
Maybe I did a suoptimal decouple, even if I change something to make
bigger tasks avoiding overhead.

Anyway great work!

Many thanks
Christian

jug execute jugfile.py --jugdir=redis://127.0.0.1 & ## Always on
ready
jug execute jugfile.py --jugdir=redis://127.0.0.1/ &

~/projekte/dev/jug/examples/decrypt/ Traceback (most recent call
last):
File "/usr/local/bin/jug", line 5, in <module>
pkg_resources.run_script('Jug==0.9-git', 'jug')
File "/usr/local/lib/python2.7/site-packages/setuptools-0.6c11-
py2.7.egg/pkg_resources.py", line 489, in run_script
File "/usr/local/lib/python2.7/site-packages/setuptools-0.6c11-
py2.7.egg/pkg_resources.py", line 1207, in run_script
File "/usr/local/lib/python2.7/site-packages/Jug-0.9_git-py2.7.egg/
EGG-INFO/scripts/jug", line 4, in <module>
jug.jug.main()
File "/usr/local/lib/python2.7/site-packages/Jug-0.9_git-py2.7.egg/
jug/jug.py", line 341, in main
execute(options)
File "/usr/local/lib/python2.7/site-packages/Jug-0.9_git-py2.7.egg/
jug/jug.py", line 189, in execute
execution_loop(tasks, options, tasks_executed, tasks_loaded)
File "/usr/local/lib/python2.7/site-packages/Jug-0.9_git-py2.7.egg/
jug/jug.py", line 129, in execution_loop
while tasks and tasks[0].can_run():
File "/usr/local/lib/python2.7/site-packages/Jug-0.9_git-py2.7.egg/
jug/task.py", line 113, in can_run
if not hasattr(dep, '_result') and not dep.can_load():
File "/usr/local/lib/python2.7/site-packages/Jug-0.9_git-py2.7.egg/
jug/task.py", line 204, in can_load
return store.can_load(self.hash())
File "/usr/local/lib/python2.7/site-packages/Jug-0.9_git-py2.7.egg/
jug/backends/redis_store.py", line 83, in can_load
return self.redis.exists(_resultname(name))
File "build/bdist.linux-i686/egg/redis/client.py", line 402, in
exists
File "build/bdist.linux-i686/egg/redis/client.py", line 278, in
execute_command
File "build/bdist.linux-i686/egg/redis/connection.py", line 258, in
send_command
File "build/bdist.linux-i686/egg/redis/connection.py", line 241, in
send_packed_command
File "build/bdist.linux-i686/egg/redis/connection.py", line 187, in
connect
File "build/bdist.linux-i686/egg/redis/connection.py", line 198, in
_connect
File "/usr/local/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
TypeError: an integer is required

[1] + exit 1 jug execute jugfile.py --jugdir=redis://127.0.0.1/

On 3 Apr., 17:51, Luis Pedro Coelho <l...@luispedro.org> wrote:
> Hi,
>
> This should work on real data, but, on random input, this is actually
> expected.
>
> See the FAQ:
>

> http://packages.python.org/Jug/faq.html#it-doesn-t-work-with-random-i...
>
> HTH
> Luis
> --
>
> LxMLS 2012: Lisbon Machine Learning Schoolhttp://lxmls.it.pt

[Luis Pedro Coelho]

On Tuesday, April 03, 2012 11:15:23 AM Christian wrote:
> Uups I see. And what might be the problem when I use the backend with
> redis (2.4.10) with your decrypt example.
> Last but not least , do you have any hint about the time issue for
> "jug status", it takes "very long" for execution in my example.
> Maybe I did a suoptimal decouple, even if I change something to make
> bigger tasks avoiding overhead.

You can try

jug status --cache

The cache will build a cache file of status results. This way, the next time
will be way faster (the first time, should be similar).