searchIndex or searchTrix both fail

[Cook, Malcolm]

Hi,

I am working with in-house mirror of genome browser installed following https://github.com/maximilianh/browserInstall

I am unable to get searchIndex or searchTrix to work for trackhubs.

I have followed closely the instructions about bedToBigBed with -extraIndex fields, and ixIxxa index creation.

Is it possible that this feature is not enabled in the browser as installed, or that some extra configuration is required to make it work?

If it should work, do you have any advice on sleuthing this.

hubCheck checks out, for starters.

When I tail the apache error log and see the same message: " Sorry, couldn't locate malcolmx in genome database," (I spiked my ix file with "malcolmx")

I can provide more detail if we want to try and debug this together, but I thought I'd ask first if I should expect it to work at all.

Ideas? Next steps?

Thanks!

Malcolm Cook

[Matthew Speir]

Hi Malcolm,

Thank you for your question about using searchIndex and searchTrix for
your track hubs on a mirror. Can you test a few track hubs on your
mirror that we have created and where we know that searching works?
First, test the following two links on our public site to confirm that
the searching works for them. Then, test them on your mirror by
replacing "genome.ucsc.edu" with the URL to your mirror. Try loading
this example hub with search enabled on hg18 (simple index search, try
searching "PDE1C" and scroll down to see a result from the hub):

http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg18&hubUrl=http://hgwdev.cse.ucsc.edu/~brianlee/hubTesting/searchHubExample/hub.txt.

Next, try loading the example hub on hg19 (more complicated search with
ixx files, try searching "2053" and scroll down to see a hit on the hub,
do this first on the public site to confirm how it should look then
repeat on your mirror):

http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg19&hubUrl=http://hgwdev.cse.ucsc.edu/~brianlee/hubTesting/searchHubExample/hub.txt.

Please see this answered MLQ archive for more info about these example
hubs.
https://groups.google.com/a/soe.ucsc.edu/d/msg/genome/1ZWq30-89fw/R_Wdc0Z1-X4J.
Or see the notes in the hg19/trackDb.txt:
http://hgwdev.cse.ucsc.edu/~brianlee/hubTesting/searchHubExample/hg19/trackDb.txt.

I hope this is helpful. If you have any further questions, please reply
to gen...@soe.ucsc.edu. All messages sent to that address are archived
on a publicly-accessible Google Groups forum. If your question includes
sensitive data, you may send it instead to genom...@soe.ucsc.edu.

Matthew Speir
UCSC Genome Bioinformatics Group

[Cook, Malcolm]

Hi Matthew,

Thanks so much for crafting these tests.

Alas, my "mirror" was constructed following https://github.com/maximilianh/browserInstall which "At the end it shows instructions how to download genome assemblies to the local machine." - and I did not download human.

I am only hosting NemVec1 which your team tarballed up for me.

Is there anything other kind of test we can possibly first perform to otherwise check out my configuration, or do you think my installing human is the best way to move this issue forward?

Thankful regards,

Malcolm Cook

[Hiram Clawson]

Good Morning Malcolm:

Try using your track hubs with the nemVec1 browser on genome-test:

http://genome-test.cse.ucsc.edu/cgi-bin/hgTracks?db=nemVec1

See if the searching works there.

--Hiram

[Cook, Malcolm]

Hi,

OK - agreed- I should have tried that myself in the first place.

I have decided to focus on debugging my use of searchTrix for starters. If we get this going, perhaps we can proceed to searchIndex.

For our mutual inspection, I have now published my hub to: http://research.stowers.org/mec/hub/NemVec1_2.MOLNG-1190.STAR/hub.txt

I still find that searchTrix indexing does not work when the hub is loaded at http://genome-test.cse.ucsc.edu/cgi-bin/hgTracks?db=nemVec1

Alas, I learn "Sorry, couldn't locate elvis in genome database" when I search for "elvis".

This despite
1) the presence of "elvis NV1_2_00000001-RA,17" in my index (http://research.stowers.org/mec/hub/NemVec1_2.MOLNG-1190.STAR/nemVec1/nemVec1_2.bigbed.ix)
2) the last stanza in my trackDb.txt http://research.stowers.org/mec/hub/NemVec1_2.MOLNG-1190.STAR/nemVec1/trackDb.txt being

track nemVec1_2
bigDataUrl nemVec1_2.bigbed
shortLabel nemVec1_2
longLabel nemVec1_2
type bigBed
searchTrix nemVec1_2.bigbed.ix

I am hoping I have misunderstood the configuration instructions somehow.

Swimmingly,
Malcolm

> >> question includes sensitive data, you may send it instead to genome-
> w...@soe.ucsc.edu.

[Cook, Malcolm]

Hi,

I have not fixed this, but want to alert you that your colleague Steve did resolve for me a different issue under the subject "[genome] HOWTO: include annotated gene predictions in a track hub" where he corrected my trackDb.txt to
change to "type bigBed 12 +" to get intron/exon structure to display. FWIW, it is in fact a 14 column bedDetail converted with `bedToBigBed -tab type=bed12+2 -as=bedDetail.as ...` and the text I am indexing with ixTxx are gained via `cut -f4,13,14', which are the name,id,description of the bedDetail.

I look forward to your further assistance in resolving my current only open issue with your excellent team (and its second part, which is getting searchIndex to work for me as well, but, one thing at a time....)

Cheers,

~Malcolm

[Matthew Speir]

Hi Malcolm,

Thank you for providing a link to your track hub. The issue is that you
are defining a searchTrix file in your trackDb without defining a
corresponding searchIndex. The searchTrix file is used to map free-text
to IDs, which are then searched for in the searchIndex of the
corresponding data file. Having a searchTrix file without the
corresponding searchIndex will not do anything.

We will expand the documentation to make this clearer. We may also add
something to hubCheck to report searchTrix entries without a matching
searchIndex entry in the future.

I hope this is helpful. If you have any further questions, please reply
to gen...@soe.ucsc.edu. All messages sent to that address are archived
on a publicly-accessible Google Groups forum. If your question includes

sensitive data, you may send it instead to genom...@soe.ucsc.edu.

Matthew Speir
UCSC Genome Bioinformatics Group

[Hiram Clawson]

Good Afternoon Malcolm:

This three-assembly hub:

http://genome-test.cse.ucsc.edu/~hiram/hubs/Plants/hub.txt

which can be found on our public list of hubs. It has a searchTrix
setup on the Ensembl gene track for the gene names:

searchIndex name
searchTrix ensGene.araTha1.ix

See if that index works in your browser, use a name from the Ensembl
gene track.

--Hiram

[Cook, Malcolm]

Matthew,

> Thank you for providing a link to your track hub. The issue is that you are
> defining a searchTrix file in your trackDb without defining a corresponding
> searchIndex. The searchTrix file is used to map free-text to IDs, which are
> then searched for in the searchIndex of the corresponding data file. Having a
> searchTrix file without the corresponding searchIndex will not do anything.

Aha! Now I see it was a uninformed mistake for me to simply my problem by trying to debug the searchTrix in the absence of a searchIndex.

(sounds of forehead slapping and murmurs of "doh" echo throughout the chambers)

So, while restoring my own "searchIndex name" in my trackdb.txt, I realize that before I had been placing it on the wrong stanzas! I had it put the searchIndex on all my bigWigs!

("doh"s are now joined by sotto voce whispers of "dumb twit"...)

So, I've fixed all that.

And, of course, I've remembered to re-enable my `-extraIndex=name` in my call to bedToBigBed

And, woo-hoo,

ELVIS IS IN THE GENOME

> We will expand the documentation to make this clearer.

Sounds good.

> We may also add
> something to hubCheck to report searchTrix entries without a matching
> searchIndex entry in the future.

And possibly to report
searchIndex on track types that don't take them.??
searchIndex referring to bed's that are not indexed ( you catch this during run-time now with a nice warning , but hubCheck could also screen on this)

Finally, I do note that my search for elvis is returning two hits to the same locus. This must be an error.

Thank you all so much for your assistance in sleuthing this.

Cheers,

~Malcolm

[Cook, Malcolm]

Hiram,

Thanks for this example. I did confirm that the hub worked on my mirror, and traced backward the reasoning how the search worked. Doing this was instrumental in helping me understand Matthew's remark that I should not have expected searchTrix to work in the absence of a searchIndex.

Case closed,

!Malcolm