Why does pyke think there's a duplicate knowledge base?

[Bill Janssen]

I'm trying hard to "like pyke" (t-shirt idea there: "I like PyKE!"),
but I'm struggling with what it's doing to my file system. I have a
daemon trying to compile a specific knowledge base, foo.krb, which
I've placed in a newly created temporary directory, /tmp/problem-
demo-6417RnK2Bu/. But I get complaints from pyke that it's a
"duplicate knowledge base". I set the debug flags in
"pyke.knowledge_engine" and "pyke.target_pkg", trying to figure this
out; below is the output from that.

It would be really nice if something would tell me what directory
"compiled_krb" was being put in, too.

Bill


2011-05-15T12:28:48 | engine._create_target_pkg:
2011-05-15T12:28:48 | /tmp/problem-demo-6417RnK2Bu
2011-05-15T12:28:48 | _create_target_pkg path:
2011-05-15T12:28:48 | '/tmp/problem-demo-6417RnK2Bu'
2011-05-15T12:28:48 | _create_target_pkg target_package_name:
2011-05-15T12:28:48 | '.compiled_krb'
2011-05-15T12:28:48 | _create_target_pkg path to _pythonify_path:
2011-05-15T12:28:48 | '/tmp/problem-demo-6417RnK2Bu'
2011-05-15T12:28:48 | path_to_package:
2011-05-15T12:28:48 | ''
2011-05-15T12:28:48 | source_package_name:
2011-05-15T12:28:48 | ''
2011-05-15T12:28:48 | remainder_path:
2011-05-15T12:28:48 | '/tmp/problem-demo-6417RnK2Bu'
2011-05-15T12:28:48 | zip_file_flag:
2011-05-15T12:28:48 | False
2011-05-15T12:28:48 | _create_target_pkg num_dots:
2011-05-15T12:28:48 | 1
2011-05-15T12:28:48 | _create_target_pkg absolute target_package_name:
2011-05-15T12:28:48 | compiled_krb
2011-05-15T12:28:48 | _create_target_pkg target_name:
2011-05-15T12:28:48 | compiled_krb.compiled_pyke_files
2011-05-15T12:28:48 | _get_target_pkg
2011-05-15T12:28:48 | compiled_krb.compiled_pyke_files
2011-05-15T12:28:48 | import_:
2011-05-15T12:28:48 | compiled_krb.compiled_pyke_files
2011-05-15T12:28:48 | _create_target_pkg: no target module
2011-05-15T12:28:48 | import_:
2011-05-15T12:28:48 | compiled_krb
2011-05-15T12:28:48 | target_pkg: no target package
2011-05-15T12:28:48 | compiled_krb
2011-05-15T12:28:48 | target_pkg package_parent_dir:
2011-05-15T12:28:48 | target_pkg target_package_dir:
2011-05-15T12:28:48 | compiled_krb
2011-05-15T12:28:48 | target_pkg: mkdir
2011-05-15T12:28:48 | compiled_krb
2011-05-15T12:28:48 | target_pkg init_filepath:
2011-05-15T12:28:48 | compiled_krb/__init__.py
2011-05-15T12:28:48 | target_pkg: creating
2011-05-15T12:28:48 | compiled_krb/__init__.py
2011-05-15T12:28:48 | target_pkg:
2011-05-15T12:28:48 | compiled_krb
2011-05-15T12:28:48 | compiled_krb/compiled_pyke_files.py
2011-05-15T12:28:48 | target_pkg.reset
2011-05-15T12:28:48 | target_pkg.add_source_package
source_package_name:
2011-05-15T12:28:48 | ''
2011-05-15T12:28:48 | path_from_package:
2011-05-15T12:28:48 | '/tmp'
2011-05-15T12:28:48 | source_package_dir:
2011-05-15T12:28:48 | ''
2011-05-15T12:28:48 | target_pkg.add_source:
2011-05-15T12:28:48 | /tmp
2011-05-15T12:28:48 | tmpqKtHPU/foo.krb
2011-05-15T12:28:48 | rb_name:
2011-05-15T12:28:48 | foo
2011-05-15T12:28:48 | key:
2011-05-15T12:28:48 | ('', '/tmp', 'tmpqKtHPU/foo.krb')
2011-05-15T12:28:48 | tmpqKtHPU/foo.krb
2011-05-15T12:28:48 | needs to be compiled
2011-05-15T12:28:48 | target_pkg.add_source:
2011-05-15T12:28:48 | /tmp
2011-05-15T12:28:48 | tmp27aPlu/foo.krb
2011-05-15T12:28:48 | rb_name:
2011-05-15T12:28:48 | foo
2011-05-15T12:28:48 | Traceback (most recent call last):
[... 'path' is /tmp/problem-demo-6417RnK2Bu ...]
File "/usr/local/lib/python2.6/dist-packages/pyke/
knowledge_engine.py", line 103, in __init__
self._create_target_pkg(path, target_pkgs)
File "/usr/local/lib/python2.6/dist-packages/pyke/
knowledge_engine.py", line 225, in _create_target_pkg
source_package_dir)
File "/usr/local/lib/python2.6/dist-packages/pyke/target_pkg.py",
line 224, in add_source_package
os.path.getmtime(source_abspath))
File "/usr/local/lib/python2.6/dist-packages/pyke/target_pkg.py",
line 252, in add_source
raise ValueError("%s: duplicate knowledge base name" % rb_name)
ValueError: foo: duplicate knowledge base name

[Bruce Frederiksen]

Bill,

Sorry for the limited responses to your messages.  I've been extremely busy of late...

You have two knowledge bases called "foo".  One in /tmp/tmpqKtHPU and the other in /tmp/tmp27aPlu.  As it happens, these are both rule bases (.krb), but you'd get the same error if any two knowledge bases have the same name (e.g., "foo.krb" and "foo.kfb").

I'm not sure why you have a daemon process compiling the .krb files.  Pyke should do that for you automatically.

If these two "foo" files are from different runs of the deamon (such each run only has one "foo" file), you might try deleting the compiled_krb directory prior to each run.

If you see an alternate approach for Pyke to handle these files, I would be open to hearing your ideas and letting you take a shot at it.  But I don't have the time now to mess with it myself.  And, after all, what better way to proudly wear "I like PyKE!" on your T-shirt than making it do what you want yourself!

HTH,

-Bruce

--
You received this message because you are subscribed to the Google Groups "PyKE" group.
To post to this group, send email to py...@googlegroups.com.
To unsubscribe from this group, send email to pyke+uns...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pyke?hl=en.

[Bill Janssen]

> I'm not sure why you have a daemon process compiling the .krb files. Pyke
> should do that for you automatically.

It's a testing framework, and compiling the rules is part of the
framework after the
rulebase is retrieved from the database. By "compiling the rules", I
mean simply
putting the rulebase in a directory, and invoking
pyke.knowledge_engine.create()
on that directory.

> If these two "foo" files are from different runs of the deamon (such each
> run only has one "foo" file), you might try deleting the compiled_krb
> directory prior to each run.

I'd love to do that, but I don't know where it is. How does pyke
determine
where to put that? Is it in my home directory, or the current working
directory, or what? And, can I somehow direct pyke as to where it
*should*
be?

Bill

On May 15, 1:59 pm, Bruce Frederiksen <dangy...@gmail.com> wrote:
> Bill,
>
> Sorry for the limited responses to your messages.  I've been extremely busy
> of late...
>
> You have two knowledge bases called "foo".  One in /tmp/tmpqKtHPU and the
> other in /tmp/tmp27aPlu.  As it happens, these are both rule bases (.krb),
> but you'd get the same error if any two knowledge bases have the same name
> (e.g., "foo.krb" and "foo.kfb").
>
> I'm not sure why you have a daemon process compiling the .krb files.  Pyke
> should do that for you automatically.
>
> If these two "foo" files are from different runs of the deamon (such each
> run only has one "foo" file), you might try deleting the compiled_krb
> directory prior to each run.
>
> If you see an alternate approach for Pyke to handle these files, I would be
> open to hearing your ideas and letting you take a shot at it.  But I don't
> have the time now to mess with it myself.  And, after all, what better way
> to proudly wear "I like PyKE!" on your T-shirt than making it do what you
> want yourself!
>
> HTH,
>
> -Bruce
>

[Bill Janssen]

I see what's going on here, though I don't see how to fix it.

I create a new knowledge base in /tmp/X/foo.krb. I compile it, and
use it.

I then create a new knowledge base in /tmp/Y/foo.krb, and call

knowledge_engine.create("/tmp/Y")

The pyke engine starts walking /tmp (!??!), and finds both /tmp/X/
foo.krb
and /tmp/Y/foo.krb, and thinks they're dups. The bug is that it walks
/tmp.

Bill

[Bill Janssen]

Thanks for pointing out the duplicate, Bruce. I've worked around
pyke's scouring all of /tmp by passing in a pseudo-filename instead of
just a directory. Now I'm getting a different error, as shown below.

Bill


engine._create_target_pkg:
/tmp/problem-demo-8971lhxoZW/foo
_create_target_pkg path:
'/tmp/problem-demo-8971lhxoZW/foo'
_create_target_pkg target_package_name:
'.compiled_krb'
_create_target_pkg path to _pythonify_path:
'/tmp/problem-demo-8971lhxoZW/foo'
path_to_package:
''
source_package_name:
''
remainder_path:
'/tmp/problem-demo-8971lhxoZW/foo'
zip_file_flag:
False
_create_target_pkg num_dots:
1
_create_target_pkg absolute target_package_name:
compiled_krb
_create_target_pkg target_name:
compiled_krb.compiled_pyke_files
_get_target_pkg
compiled_krb.compiled_pyke_files
import_:
compiled_krb.compiled_pyke_files
_create_target_pkg: no target module
import_:
compiled_krb
target_pkg: no target package
compiled_krb
target_pkg package_parent_dir:

target_pkg target_package_dir:
compiled_krb
target_pkg init_filepath:
compiled_krb/__init__.py
target_pkg:
compiled_krb
compiled_krb/compiled_pyke_files.py
target_pkg.reset
target_pkg.add_source_package source_package_name:
''
path_from_package:
'/tmp/problem-demo-8971lhxoZW'
source_package_dir:
''
target_pkg.add_source:

/tmp/problem-demo-8971lhxoZW
foo.krb
rb_name:
foo
key:
('', '/tmp/problem-demo-8971lhxoZW', 'foo.krb')
foo.krb
needs to be compiled
target_package:
<pyke.target_pkg.target_pkg object at 0x1477750>
compiled_krb.compile:
compile_krb:
/tmp/problem-demo-8971lhxoZW/foo.krb
writing [compiled_krb]/foo_bc.py
writing [compiled_krb]/foo_plans.py
target_files:
['foo_plans.py', 'foo_bc.py']
target_pkg.write
writing [compiled_krb]/compiled_pyke_files.py
write got:
('', '/tmp/problem-demo-8971lhxoZW', 'foo.krb')
[1305500555.7290511, 'foo_plans.py', 'foo_bc.py']
writing:
('', '/tmp/problem-demo-8971lhxoZW', 'foo.krb')
[1305500555.7290511, 'foo_plans.py', 'foo_bc.py']
target_pkg.load:
{'load_fb': True, 'load_fc': True, 'load_bc': True, 'load_qb': True}
load:
foo_plans.py
load_py:
foo_plans.py
load_module:
compiled_krb.foo_plans
foo_plans.py
load:
foo_bc.py
load_py:
foo_bc.py
load_module:
compiled_krb.foo_bc
foo_bc.py
load_module: importing
import_:
compiled_krb.foo_bc

Traceback (most recent call last):

[...]
engine = knowledge_engine.engine(*newrules)
File "/usr/local/lib/python2.6/dist-packages/pyke/
knowledge_engine.py", line 109, in __init__
target_package.load(self, **kws)
File "/usr/local/lib/python2.6/dist-packages/pyke/target_pkg.py",
line 365, in load
self.do_by_ext('load', target_filename, engine, load_flags)
File "/usr/local/lib/python2.6/dist-packages/pyke/target_pkg.py",
line 264, in do_by_ext
return getattr(self, "%s_%s" % (prefix, ext))(filename, *args)
File "/usr/local/lib/python2.6/dist-packages/pyke/target_pkg.py",
line 376, in load_py
self.load_module(module_path, target_filename, engine)
File "/usr/local/lib/python2.6/dist-packages/pyke/target_pkg.py",
line 406, in load_module
module = import_(module_path)
File "/usr/local/lib/python2.6/dist-packages/pyke/target_pkg.py",
line 463, in import_
mod = __import__(modulename)
ImportError: No module named compiled_krb.foo_bc

[Bill Janssen]

Aha! Got it. Turns out that the directory pyke was putting the
compiled_krb directory into was the current working directory, which
wasn't on sys.path. So import couldn't load compiled_krb.foo_bc.

It would really be great if you could tell pyke where compiled_krb
should go.

Bill

[Lauri Kainulainen]

Hi all,

On 05/16/2011 02:18 AM, Bill Janssen wrote:
> It would really be great if you could tell pyke where compiled_krb
> should go.

+1 for this option. We're using PyKE with Django and I would love to put
the compiled_krb into /tmp instead of the working directory. The problem
with the latter is that the WSGI-process does not have write access to
the application directory and from time to time we get permission errors
as the rules change.

I know I could fix it with umask or maybe adding the compiled_krb into
our GIT, but that doesn't seem like a pretty solution.

Cheers,
Lauri Kainulainen

[Bruce Frederiksen]

Hi Lauri,

I guess you're also generating rules dynamically?

Pyke was designed thinking that the rules would be relatively static (like the source code).  And an attempt was made to allow applications to be written that could be distributed through pypi or whatever; which means that they would be installed into different directories on different systems.

This was done by relying on the Python path to locate the compiled_krb directory.  You can override the default placement of the compiled_krb directory (as well as it's name) by passing a 2-tuple rather than a string for the path to load:

My_engine = knowledge_engine.engine(
  (rules_dir, "pyke_rules"))

Then tell Python where "pyke_rules" is by putting the directory containing pyke_rules on your Python path (e.g., add a .pth file to your site-packages directory).  This can be whatever directory you like, for example: /tmp/myapp_rules.  I think that if you don't have a prefix on the target (not "foo.pyke_rules"), you'll have to create a /tmp/myapp_rules/pyke_rules directory with an __init__.py in it so that Pyke will find it.  If you said "foo.pyke_rules", then the "foo" directory would have to exist, and Pyke would create the pyke_rules directory under it.

HTH,

-Bruce

[Bill Janssen]

Lauri,

I've added this code to the main() of each of my agents:

# to use PyKE, we need to have our own private working directory
wd = tempfile.mkdtemp("", "agent-using-pyke-%s-" % os.getpid())
os.chdir(wd)
if os.path.isdir(os.path.join(wd, "compiled_krb")):
shutil.rmtree(os.path.join(wd, "compiled_krb"))
sys.path.append(wd)
atexit.register(lambda x=wd: shutil.rmtree(x))

That way I don't have two agents both trying to use the same
compiled_krb, possibly for rulebases that have the same name.

Bill


On May 15, 10:28 pm, Lauri Kainulainen <lauri.kainulai...@gmail.com>
wrote:

[Lauri Kainulainen]

Hi Bruce,

On 05/16/2011 05:23 PM, Bruce Frederiksen wrote:
> I guess you're also generating rules dynamically?

Not really. The problem just arises when I've done some changes on the
development server and updated compiled_krb as my normal user (e.g. with
Django this happens if you use the Django shell). Then if I change
something locally, updating and refreshing the server will cause the
perms error as www-data tries writing to the compiled_krb folder.

> Pyke was designed thinking that the rules would be relatively static
> (like the source code). And an attempt was made to allow applications
> to be written that could be distributed through pypi or whatever; which
> means that they would be installed into different directories on
> different systems.

Ok. I see.

> Then tell Python where "pyke_rules" is by putting the directory
> containing pyke_rules on your Python path (e.g., add a .pth file to your
> site-packages directory). This can be whatever directory you like, for
> example: /tmp/myapp_rules. I think that if you don't have a prefix on
> the target (not "foo.pyke_rules"), you'll have to create a
> /tmp/myapp_rules/pyke_rules directory with an __init__.py in it so that
> Pyke will find it. If you said "foo.pyke_rules", then the "foo"
> directory would have to exist, and Pyke would create the pyke_rules
> directory under it.

I poked around the code enough to find that option. Didn't just occur to
me to use it with .pth files. I'll try that at some point.

Thanks for the pointers!
Lauri