IGV server not rendering reads or variants

Mark Cowley

unread,

Sep 19, 2016, 7:18:03 PM9/19/16

to igv-...@googlegroups.com

Hi,

Can you please help me debug some challenges that i’m having with configuring a local IGV server? As detailed below, I can see reads and variants if I load data via URLs, but not via a custom IGV server, so the problem likely lies in the way i’ve written the XML files.

We use DNAnexus, which for the purposes of accessing data in IGV standalone, you need to generate a secure URL for both the BAM, and BAI file. i.e. BAI url is not just BAM url+”.bai”. I’ve made some scripts to generate an XML representation of a DNAnexus project, creating secure download URLs, and a registry to point to these xml files. Here’s one such project (data is ok to be shared).

<?xml version="1.0" encoding="utf-8"?>
<Global genome="1kg_v37" name="X10 TruSeq Nano V2.5 NA12878 (Public)" version=“1">
   <Resource index="https://dl.dnanex.us/F/D/6kbP94910JYbF6g7b6vVYFYgBFpVPpBQJV6YPGx9/NA12878-Vial1.dedup.realigned.recalibrated.bam.bai" name="NA12878-Vial1.dedup.realigned.recalibrated.bam" path="https://dl.dnanex.us/F/D/Q4V427fk05j8yj6B5PgVK403xv4Kbv505zGPV0zy/NA12878-Vial1.dedup.realigned.recalibrated.bam"/>
   <Resource index="https://dl.dnanex.us/F/D/B6BjXp023F2xy411QFFJv3YvZJXJq7vqQjFfXgjQ/NA12878-Vial1.hc.vqsr.vcf.gz.tbi" name="NA12878-Vial1.hc.vqsr.vcf.gz" path="https://dl.dnanex.us/F/D/21kV32x5z1Bk5FYfpXbbb1JbgXFbqBpZ9FBxjyx7/NA12878-Vial1.hc.vqsr.vcf.gz"/>
</Global>

I’m getting two different error modes when using three different versions of IGV:

1) IGV Snapshot last night

Error encountered querying alignments: htsjdk.samtools.util.RuntimeIOException: Invalid file header in BAM index https://dl.dnanex.us/F/D/Q4V427fk05j8yj6B5PgVK403xv4Kbv505zGPV0zy/NA12878-Vial1.dedup.realigned.recalibrated.bam.bai: �

(yes with the � char at the end)

2) IGV 2.3.81 or IGV 2.3.55

File > Load from Server

This loads only the BAM file track, and when you zoom in to an exon, there’s no reads displayed. The courage depth drops from default of [0,60] to [0,10] and no reads are shown.

Debugging

* wget on each of the URLs works fine

* Thanks to your recent changes (prompted by DNAnexus I believe), we can load from URL, specifying both the BAM and BAI URL. IGV 2.3.81 loads both the BAM and vcf.gz file using the URLs as above.

* IGV Snapshot is not loading BAM or VCF.gz from URLs. I get empty tracks with no reads.

* tried nesting these Resources into a Category node, but no change.

* removed genome from the Global element, but no change

My best guess is there’s something not quite right in the XML manifest. Do you know what’s going wrong here?

cheers,

Mark

igv.log for 2.3.81 when Load from Server

INFO [2016-09-20 09:13:13,258] [Main.java:133] Startup IGV Version 2.3.81 (127)08/30/2016 02:20 PM

INFO [2016-09-20 09:13:13,258] [Main.java:134] Java 1.8.0_101

INFO [2016-09-20 09:13:13,258] [DirectoryManager.java:72] Fetching user directory...

INFO [2016-09-20 09:13:13,693] [Main.java:135] Default User Directory: /Users/marcow

INFO [2016-09-20 09:13:13,694] [Main.java:136] OS: Mac OS X

INFO [2016-09-20 09:13:18,969] [GenomeManager.java:145] Loading genome: /Users/marcow/igv/genomes/1kg_v37.genome

INFO [2016-09-20 09:13:22,191] [GenomeManager.java:192] Genome loaded. id= 1kg_v37

INFO [2016-09-20 09:13:22,249] [CommandListener.java:106] Listening on port 60151

INFO [2016-09-20 09:13:29,368] [IGV.java:1700] Loading 2 resources.

INFO [2016-09-20 09:13:29,369] [TrackLoader.java:120] Loading resource, path https://dl.dnanex.us/F/D/Q4V427fk05j8yj6B5PgVK403xv4Kbv505zGPV0zy/NA12878-Vial1.dedup.realigned.recalibrated.bam

INFO [2016-09-20 09:13:36,006] [HttpUtils.java:832] Range-byte request succeeded

INFO [2016-09-20 09:13:38,036] [TrackLoader.java:120] Loading resource, path https://dl.dnanex.us/F/D/21kV32x5z1Bk5FYfpXbbb1JbgXFbqBpZ9FBxjyx7/NA12878-Vial1.hc.vqsr.vcf.gz

INFO [2016-09-20 09:13:48,709] [HttpUtils.java:832] Range-byte request succeeded

INFO [2016-09-20 09:14:04,010] [ShutdownThread.java:51] Shutting down

INFO [2016-09-20 09:14:04,016] [ShutdownThread.java:51] Shutting down

ERROR [2016-09-20 09:14:04,018] [DefaultExceptionHandler.java:49] Unhandled exception

java.lang.NullPointerException

at org.broad.igv.sam.AlignmentTrack.dispose(AlignmentTrack.java:1029)

at org.broad.igv.ui.ShutdownThread.runS(ShutdownThread.java:57)

at org.broad.igv.ui.ShutdownThread.run(ShutdownThread.java:64)

Mark Cowley

unread,

Sep 19, 2016, 8:08:07 PM9/19/16

to igv-help

slight correction in the debugging. I just had some internet lag - reads do load fine in IGV Snapshot

* IGV Snapshot is not loading BAM or VCF.gz from URLs. I get empty tracks with no reads.

best, mark

James Robinson

unread,

Sep 19, 2016, 9:07:27 PM9/19/16

to igv-help

Hi Mark, thanks for the report. I am traveling this week and won't be able to look at this in detail until I return. It appears there are 2 problems, a possible XML problem and VCF bug in the snapshot. Could you attach one of your XML files?

Actually, it would be helpful if you could open a git issue (or 2) with the XML attached.

Jim

--

---
You received this message because you are subscribed to the Google Groups "igv-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to igv-help+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/fedb2d3e-271f-4603-8c5e-19ebf4bde41a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Mark Cowley

unread,

Oct 4, 2016, 9:16:03 AM10/4/16

to igv-...@googlegroups.com

Hi Jim,

Have you had a chance to look into this issue? Cracking this would really facilitate us to interact with large amounts of data sitting on the cloud.

best regards,

Mark

On 20 Sep 2016, at 12:30 PM, Mark Cowley <m.co...@garvan.org.au> wrote:

Thanks Jim,
https://github.com/igvteam/igv/issues/317

All the best,
Mark

To unsubscribe from this group and stop receiving emails from it, send an email to igv-help+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/CACOP%2Bpuj2jOGChfzLNTEJufDeKG6dqSBhra0a_3a7tvGNzNgGQ%40mail.gmail.com.

James Robinson

unread,

Oct 5, 2016, 5:16:07 PM10/5/16

to igv-help

I'm looking into it now. The first problem is "index" is not supported in session xml files, I've added support for it.

The second problem is that your server is returning "200" codes for files that do not exist. There are some optional files for bams and vcf that get loaded if they are there, specifically a coverage file (.tdf) for bams and a sample->bam mapping file for VCFs. To see if these exist a head (or get if head fails) request is made and the http return code checked. Your server seems to return 200 for any path sent it, see the screenshot for example. Any idea why this is?

https://dl.dnanex.us/F/D/X7317f2jjKpX6Gyf6pZffYzPFPpjbQXVGqv286PQ/NA12878-Vial1.hc.vqsr.vcf.gz.mapping

=> return code 200

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/CACOP%2Bpuj2jOGChfzLNTEJufDeKG6dqSBhra0a_3a7tvGNzNgGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "igv-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to igv-help+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/A9B18350-F0FA-4842-A4E0-4B3418D31171%40gmail.com.

James Robinson

unread,

Oct 5, 2016, 5:23:43 PM10/5/16

to igv-help

OK, quite strange, https://dl.dnanex.us/F/D/X7317f2jjKpX6Gyf6pZffYzPFPpjbQXVGqv286PQ/NA12878-Vial1.hc.vqsr.vcf.gz.mapping returns the content of https://dl.dnanex.us/F/D/X7317f2jjKpX6Gyf6pZffYzPFPpjbQXVGqv286PQ/NA12878-Vial1.hc.vqsr.vcf.gz. I'm sure the same is probably true for <path to bam>.tdf (it returns the content of the bam). This needs fixed on your server. I will check in support for index in server xml shortly, when the server is fixed it should work for you.

Jim

Mark Cowley

unread,

Oct 5, 2016, 10:42:33 PM10/5/16

to igv-...@googlegroups.com

Hi Jim,

This is because the actual file is specified by the URL stem (up to the final ‘/‘) and the part following the ‘/‘ represents the name that you would like the final file to be. Thus changing that end of the URL will always return the same file. This same behaviour relates to some code that you merged a while back to allow an explicit bai URL to be specified. We’ve discussed changing this behaviour, but that’s beyond our control, as the platform is commercial.

Would it be feasible to put an explicit mapping=None, tdf=None in the <Resource> element to indicate to not automatically pursue these links?

cheers,

Mark

On 6 Oct 2016, at 8:23 AM, James Robinson <jrob...@broadinstitute.org> wrote:

OK, quite strange, https://dl.dnanex.us/F/D/X7317f2jjKpX6Gyf6pZffYzPFPpjbQXVGqv286PQ/NA12878-Vial1.hc.vqsr.vcf.gz.mapping returns the content of https://dl.dnanex.us/F/D/X7317f2jjKpX6Gyf6pZffYzPFPpjbQXVGqv286PQ/NA12878-Vial1.hc.vqsr.vcf.gz. I'm sure the same is probably true for <path to bam>.tdf (it returns the content of the bam). This needs fixed on your server. I will check in support for index in server xml shortly, when the server is fixed it should work for you.

Jim

On Wed, Oct 5, 2016 at 2:16 PM, James Robinson <jrob...@broadinstitute.org> wrote:

I'm looking into it now. The first problem is "index" is not supported in session xml files, I've added support for it.

The second problem is that your server is returning "200" codes for files that do not exist. There are some optional files for bams and vcf that get loaded if they are there, specifically a coverage file (.tdf) for bams and a sample->bam mapping file for VCFs. To see if these exist a head (or get if head fails) request is made and the http return code checked. Your server seems to return 200 for any path sent it, see the screenshot for example. Any idea why this is?

https://dl.dnanex.us/F/D/X7317f2jjKpX6Gyf6pZffYzPFPpjbQXVGqv286PQ/NA12878-Vial1.hc.vqsr.vcf.gz.mapping

=> return code 200

<Screen Shot 2016-10-05 at 2.12.02 PM.png>

To unsubscribe from this group and stop receiving emails from it, send an email to igv-help+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/CACOP%2BptHkFdupFiPR76PpXLOWMyx3o-3CPdGb9XPCiRhBTXmxg%40mail.gmail.com.

James Robinson

unread,

Oct 5, 2016, 11:18:17 PM10/5/16

to igv-help

Yes, that would be the only solution, and makes sense in any event. I feel like that platform is breaking expected rules for paths in a fundamental way, but it is what it is. The change to allow explicit bai, and other indexes, is to handle cases where the path do not follow standard convention. For example s3 urls.

Mark Cowley

unread,

Oct 5, 2016, 11:30:46 PM10/5/16

to igv-...@googlegroups.com

agreed - thanks for considering these changes though!

cheers,
Mark

--

---
You received this message because you are subscribed to the Google Groups "igv-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to igv-help+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/CACOP%2BpuM%3DObjkN9Yj3eXx7qQrcEyc86PaOHbDMOuWe%3DMS_Dj8Q%40mail.gmail.com.

Reply all

Reply to author

Forward