Worldbase and metabase
Marek Szuba
As you probably remember, one of the purposes of rewriting the download
megatest has been to have it use dedicated metabases rather than
worldbase. It turns out however the two interact with each other - or
to be exact, individual metabases appear to be tied to worldbase - in
a sometimes peculiar manner. And I'm not even talking about downloading
and building NLMSAs being controlled controlled by $WORLDBASExxx
environment variables even when individual metabases are used - it gets
more interesting...
Take a machine running the new download megatest, with my stupid
httpdPort bug already fixed. The test repeatedly and reproducibly runs
without errors. At some point, the UCLA XML-RPC server stops responding
to requests. This should have no effect on the new version of the test,
right? Except what really happens is:
- the text file is downloadad and unpacked;
- after beginning to save indexes (i.e. after it has been announced
but before the message about index 0 is displayed), a warning appears
about problems talking to the UCLA XML-RPC server;
- "No Test.NLMSA found in WORLDBASEPATH".
What do you think, guys? Is it a bug in the megatest or something in
Pygr code itself worth looking into?
Cheers,
--
MS
Christopher Lee
On Dec 9, 2009, at 6:22 PM, Marek Szuba wrote:
> - after beginning to save indexes (i.e. after it has been announced
> but before the message about index 0 is displayed), a warning appears
> about problems talking to the UCLA XML-RPC server;
> - "No Test.NLMSA found in WORLDBASEPATH".
- i can't tell from your message whether you're reporting a behavior that actually causes a problem, or merely a warning message. Please clarify.
- And you have to show us the code if you want us to understand what you mean by such details as "but before the message about index 0 is displayed".
Reading between the lines, I'm wondering how the metabase code will handle the case where the XMLRPC server goes away in the middle of a worldbase session. It may not properly handle the exception that arises in that case.
-- Chris
Marek Szuba
I've just realised I never got back to this thread and elaborated on this
issue... Here goes:
To begin with, our reference code will be downloadNLMSA_megatest.py from
the current master.
Secondly, what happens is an error, not a warning.
Thirdly, here is what exactly happens:
- good case, i.e. with WORLDBASEPATH pointing at the UCLA XML-RPC
server
[...]
INFO downloader.download_unpickler: Download done.
INFO downloader.uncompress_file: gunzipping
/var/tmp/megatestSNZtX8/dm2_multiz9way.txt.gz...
INFO nlmsa_utils.nlmsa_textdump_unpickler: Saving NLMSA indexes from
textdump: /var/tmp/megatestSNZtX8/dm2_multiz9way.txt
/result/mareks/pygr/pygr/sqlgraph.py:1111: DeprecationWarning: The cursor
argument is deprecated. Use serverInfo instead!
SQLTableBase.__init__(self, *args, **kwargs)
INFO nlmsa_utils.nlmsa_textdump_unpickler: Saving NLMSA binary index: 0
NLMSA_LPO_Internal0 165964729
...
INFO nlmsa_utils.nlmsa_textdump_unpickler: Saving NLMSA binary index: 1
NLMSA_UNION_Internal 1 1847310997
...
ok
- bad case, i.e. after setting WORLDBASEPATH so that worldbase can't find
any resources - 'export WORLDBASEPATH=/nonexistent', for instance)
[...]
INFO downloader.download_unpickler: Download done.
INFO downloader.uncompress_file: gunzipping
/var/tmp/megatestJLHQ9l/dm2_multiz9way.txt.gz...
INFO nlmsa_utils.nlmsa_textdump_unpickler: Saving NLMSA indexes from
textdump: /var/tmp/megatestJLHQ9l/dm2_multiz9way.txt
ERROR
======================================================================
ERROR: Test downloading NLMSA data
----------------------------------------------------------------------
Traceback (most recent call last):
File "downloadNLMSA_megatest.py", line 69, in test_download
msa_dl = self.mdb_dl(self.resource_name) # download and build it!
File "/result/mareks/pygr/pygr/metabase.py", line 777, in __call__
for objdata, docstr in self.find_resource(resID, download):
File "/result/mareks/pygr/pygr/metabase.py", line 1047, in find_resource
% resID)
WorldbaseNotFoundError: 'unable to find Test.NLMSA in WORLDBASEPATH'
----------------------------------------------------------------------
Finally, the issues I have got here are as follows:
- despite using an individual metabase, one still uses $WORLDBASEDOWNLOAD
and $WORLDBASEBUILDDIR to control the destination of alignment files. This
may be somewhat confusing, not to mention it is impossible to set these
individually for different metabases. Possible solution - make these
metabase-specific options which only fall back to the environment if the
local option is not set?
- the error message given in the bad case is very confusing. Test.NLMSA
is not in WORLDBASEPATH because it's not supposed to be in it, which
brings us to an important question - why has Pygr begun looking in the
worldbase even though the resource has been requested specifically from
a dedicated metabase. Possible solution: none right now, we should
however discuss whether we want metabase isolation or not, how to control
it and so on;
- another question posed by the above is what exactly happened that has
caused the downloaded Test.NLMSA not to be accessed correctly. My guess is
that with no worldbase resources available, Pygr has failed to locate the
necessary sequence data, quietly aborted the build and went on to look
for Test.NLMSA elsewhere. The problem here is the "quietly aborted" bit -
given how talkative NLMSA building normally is, at least at high verbosity
levels, it really should report such problems, even when not invoked
diretly. Possible solution: make sure errors from NLMSA building are
not suppressed by code invoking in and explain what specifically has been
wrong.
Let me know what you think, guys.
Cheers,
--
MS
Christopher Lee
On Feb 24, 2010, at 4:46 PM, Marek Szuba wrote:
> - despite using an individual metabase, one still uses $WORLDBASEDOWNLOAD and $WORLDBASEBUILDDIR to control the destination of alignment files. This may be somewhat confusing, not to mention it is impossible to set these individually for different metabases. Possible solution - make these metabase-specific options which only fall back to the environment if the local option is not set?
Off the top of my head, I can't think of a good way to make that work. Imagine the following: the Python interpreter is deep in the middle of a user's custom unpickling function and it's about to open a file for writing data it's downloading. How on earth will it know whether it was called by "an individual metabase" or "worldbase"? No such information is passed to Python custom unpickler functions. Remember that unpickling is a standardized Python process; we can't change the rules of what data the Python unpickler will pass to the user's custom unpickling function. That's why we were forced to use environment variables. I guess we could use a global variable that gets temporarily set just for the duration of a Metabase.__call__(), although in general I hate to use global variables...
>
> - the error message given in the bad case is very confusing. Test.NLMSA is not in WORLDBASEPATH because it's not supposed to be in it, which brings us to an important question - why has Pygr begun looking in the worldbase even though the resource has been requested specifically from a dedicated metabase. Possible solution: none right now, we should however discuss whether we want metabase isolation or not, how to control it and so on;
You are asking two separate questions here. I'll take them separately:
1. you don't like the error message. OK, what do you want it changed to? The simplest thing I can think of is just to print out the actual search path string instead of saying "WORLDBASEPATH".
2. you claim MetabaseList is searching the WORLDBASEPATH, but you would need to *show* that it does so, and exactly where in the code, for us to move forward with this. The metabase module does not import worldbase. MetabaseList only defaults to WORLDBASEPATH if you tell it to, e.g. by giving it a blank string. When it finds another worldbaseID in a resource it is building, it simply calls itself to fetch that resource. Have you even run this under a debugger to show that it searches the WORLDBASEPATH? If not, you need to nail that down before there is a basis for us to work on.
>
> - another question posed by the above is what exactly happened that has caused the downloaded Test.NLMSA not to be accessed correctly. My guess is that with no worldbase resources available, Pygr has failed to locate the necessary sequence data, quietly aborted the build and went on to look for Test.NLMSA elsewhere. The problem here is the "quietly aborted" bit - given how talkative NLMSA building normally is, at least at high verbosity levels, it really should report such problems, even when not invoked diretly. Possible solution: make sure errors from NLMSA building are not suppressed by code invoking in and explain what specifically has been wrong.
Actually, do you know that your Test.NLMSA download / build really should work as written? What sequence resources does the NLMSA need, and where is it supposed to get them from? Unless the test code provides completely clear answers to those questions, the test itself is buggy. We can't treat this as a Pygr bug until we have excluded that possibility.
-- Chris