Loading BAM file from HTML link problem

[ersen kavak]

Hello, 

I am using the IGV 2.3.34(41)

I have a 544 KB bam file at 

www.genomize.com/genera/deneme.bam 

and 

www.genomize.com/genera/deneme.bai and www.genomize.com/genera/deneme.bam.bai 

index files are also there... 

I am trying to load the file by:

http://localhost:60151/load?file=http://www.genomize.com/genera/deneme.bam&locus=MEFV&genome=hg19

And, without any errors it says loading for hours. When I try to download the file by using the link, it takes only seconds to download it. 

I have read the related section in the manual:

http://www.broadinstitute.org/software/igv/book/export/html/189

this thread, and actually many others, but could not understand what I am doing wrong.

https://groups.google.com/forum/#!msg/igv-help/BL4Qu41rjTk/Ggf-JglcRjkJ

What might I be doing wrong ?

Cheers

Ersen

[James Robinson]

Does the sever hosting the bam support range byte requests?  

--

---
You received this message because you are subscribed to the Google Groups "igv-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to igv-help+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/41a5e84e-b525-4ebe-8103-645691d97acf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jim Robinson]

Hi,

There's something different about your server.  I was able to reproduce the problem with your links below,  then put the same files on our server and it worked fine (link below).   Of course you have to start IGV before trying this link.

http://localhost:60151/load?file=http://www.broadinstitute.org/igvdata/test/deneme/deneme.bam&locus=MEFV&genome=hg19

I'll see what I can discover in the debugger and pass on any information I find here. 

Jim

[Jim Robinson]

OK,  I found the problem.  IGV checks for the existence of a coverage file with the path  <path to bam file>.tdf.   If it exists it is automatically loaded with the bam.   The test for existence is to do a head request on the path and check the response code.   You server is returning a response code of 200 for the path http://www.genomize.com/genera/deneme.bam.tdf.   If the file does not exist you should set a return status code of 404,  although any code other than 2XX will be interpreted as the file is not reachable.    This should fix your problem.
 
Other suggestions if you are implementing a file server for use with IGV:

Implement support for the range byte headers (I think your server is already doing this)
Set the content-length in the response


Jim

[ersen kavak]

Jim,

Thank you very much for your answers. 

This was a test case to test HTML based usage of IGV for our software. The real usage is on a separate dedicated CentOS server. I think the byte range is supported in that server (please see the end of this message). I am getting a GZIP header warning

But, with your suggestions in mind, the tdf search is now raising a 404 with Python on Django. 

Now, when I try to load the file from 

http://seqtest.genomize.com/seqpipeline/download/fmf/MID-1/1/MID-1.bam

by using the URL 

http://localhost:60151/load?file=http://seqtest.genomize.com/seqpipeline/download/fmf/MID-1/1/MID-1.bam&locus=MEFV&genome=hg19

I am getting an error of invalid GZIP header in IGV v2.34. 

I tried downloading both the bam and the .bai files and loading manually to the IGV. This works just fine.

The Django code which prepares the response upon requesting the above URL is exactly as follows:

@login_required(login_url = loginurl)

def download(request, runname, samplename, analysisid, param):

    from django.utils.encoding import smart_str

    from django.http import Http404

    import mimetypes

    from django.core.servers.basehttp import FileWrapper

    from accounts.models import Site

    import socket

    sitename = Site.objects.get(userprofile = request.user).name

    datapath = "/mnt/genomize/seqdata/"

    if param.endswith( ".bam" ) or param.endswith(".bai"):

        ext = "bam.bai" if param.endswith(".bai") else "bam"

        import os

        resultingfilepath =  os.path.join(datapath, sitename, runname, samplename, analysisid, '%s.sorted.%s' % (samplename, ext))

        print resultingfilepath

        resultingfilename = "%s-%s-%s.%s" % (runname, samplename, analysisid, ext)

    elif param.endswith(".tdf"):

        raise Http404

    else:

        raise UserWarning, 'this is not supported yet...'

    file_wrapper = FileWrapper(file(resultingfilepath, 'rb'))

    file_mimetype = mimetypes.guess_type(resultingfilepath)

    #response = HttpResponse(mimetype='application/force-download')

    response = HttpResponse(file_wrapper, content_type = file_mimetype )

    print smart_str(resultingfilepath)

    response['Content-Length'] = os.path.getsize( resultingfilepath )

    response['X-Sendfile'] = resultingfilepath

    response['Content-Disposition'] = 'attachment; filename=%s' % smart_str(resultingfilename)

    # It's usually a good idea to set the 'Content-Length' header too.

    # You can also set any other required headers: Cache-Control, etc.

    return response

ABOUT THE BYTE RANGE SUPPORT

 

Using curl as suggested in this thread ( http://superuser.com/questions/451445/how-can-i-test-if-a-server-supports-byte-serving ). I am getting a found response.

myhost:Desktop ersenkavak$ curl -H Range:bytes=16- -I http://seqtest.genomize.com/seqpipeline/download/fmf/MID-1/1/MID-1.bam

HTTP/1.1 302 FOUND

Date: Sat, 26 Jul 2014 20:30:27 GMT

Server: Apache/2.2.15 (CentOS)

Vary: Accept-Language,Cookie

X-Frame-Options: SAMEORIGIN

Content-Language: en

Location: http://seqtest.genomize.com/accounts/xlogin/?next=/seqpipeline/download/fmf/MID-1/1/MID-1.bam

Connection: close

Content-Type: text/html; charset=utf-8

--

---
You received this message because you are subscribed to a topic in the Google Groups "igv-help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/igv-help/dp6-FUjdX4s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to igv-help+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/233887c0-bbe0-4221-8771-35973fc4e759%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Erşen Kavak, PhD

CEO, Genomize  Inc.

E: er...@genomize.com 

T: +90 212 970 06 88

F: +90 850 220 04 51

M: +90 534 075 14 90

-----------------

Esentepe Mah. Büyükdere Cad. 

Oyal İş Merkezi, 108/1, Kat 9

34394 Şişli İstanbul Turkey

http://www.genomize.com

[Jim Robinson]

Hi,

Sorry I don't know anything about django and am not in a position to
debug your server. As you have an apache server available I suggest
you get it working with that first.

Jim

[ersen kavak]

Hi, 

I understand that. I do not have control over the apache server (Django runs with Apache as well, by the way.) So, I cannot change the 404 response property there. But, for the dedicated server, i have root access. 

I pasted in the code to simply to show I am setting the content-length in the response. And, the server supports byte range. And, may be, people here came across a similar problem and come up with an idea (i am not sure if others read these messages now). 

Anyways, thank you very much for the responses Jim. I will paste the solution here, just so we have it recorded, if i can solve the issue in Django. 

Just for the record, Django, because it is coded with Python, may be in common use with IGV, because bioinformaticians like Django. 

Regards

Ersen

--

--- You received this message because you are subscribed to a topic in the Google Groups "igv-help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/igv-help/dp6-FUjdX4s/unsubscribe.

To unsubscribe from this group and all its topics, send an email to igv-help+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/53D464C3.9010900%40broadinstitute.org.

[Jim Robinson]

Hi Ersen,

Yes, please post any solutions you find, that would be helpful. Have you
verified that your server is returning the correct range of data for a
range-byte request? Is that something you are coding yourself, by
seeking to the position in the file and returning the bytes requested?
If you are coding that yourself I would suspect it first and verifiy
it. Its easy to write such a test, just create a file with a known
sequence of bytes then make a range byte request in the middle of the
file somewhere and verify that you are getting the bytes requested.

Also, be careful that your server does not gzip the response, and that
no gzip content encodings are specified in the response header.

Finally, I did try your url
http://seqtest.genomize.com/seqpipeline/download/fmf/MID-1/1/MID-1.bam
by using "File > Load from URL". This is a more direct way to test than
the localhost url. It reveals another error with your server (below)

ERROR [2014-07-27 09:53:10,641] [HttpUtils.java:218] [pool-1-thread-6]
Byte range requested, but no Content-Range header in response

See section 14.16 of this document for an explanation:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html. As an
example, if range 500-599 of a file of size 1234 bytes is requested the
content-range header would be

Content-Range: bytes 500-599/1234

I don't think this is causing an actual problem with IGV, but should
be fixed. It is causing a lot of log messages.

- Jim

[ersen kavak]

Hi Jim, 

Have been working on this. Unfortunately, I still do not have a django based solution. I will try some more by keeping in mind your suggestions, and definitely will let you know upon a solution. 

To summarize where I am, 

1) About the CENTOS server, i think the settings are generally ok, because, i have another non-django based website there, and when I load the URL http://content.genomize.com/MID-3.sorted.bam then it is just fine, even though there is no TDF file there, just the bam.bai file.

2) As answer to your questions:

Have you verified that your server is returning the correct range of data for a range-byte request?   Is that something you are coding yourself, by seeking to the position in the file and returning the bytes requested?

I am not coding the byte-range function myself. In django, one writes a Python function (so called the views.py in the Models-Views-Processors web application setup) which takes in a request and then returns a response. 

So, after I load a functioning URL to IGV via the loadURL menu function, i looked at the Request to find something about the byte-range.. 

This is what the request dictionary looks like, so i could add your suggestions to the django code. But, I could not identify any byte range related requests in it. 

<WSGIRequest

path:/seqpipeline/download/demo/run1/MID-4/1/MID-4.bam,

GET:<QueryDict: {}>,

POST:<QueryDict: {}>,

COOKIES:{},

META:{'Apple_PubSub_Socket_Render': '/tmp/launch-PWXSLR/Render',

 'CELERY_LOADER': 'djcelery.loaders.DjangoLoader',

 'CONTENT_LENGTH': '',

 'CONTENT_TYPE': 'text/plain',

 u'CSRF_COOKIE': u'6AxID7no76hUcflPGOcR2wMWslxPosRK',

 'DJANGO_SETTINGS_MODULE': 'genoseq.settings_development',

 'GATEWAY_INTERFACE': 'CGI/1.1',

 'HOME': '/Users/ersenkavak',

 'HTTP_ACCEPT': 'text/plain',

 'HTTP_CACHE_CONTROL': 'no-cache',

 'HTTP_CONNECTION': 'keep-alive',

 'HTTP_HOST': 'localhost:8001',

 'HTTP_PRAGMA': 'no-cache',

 'HTTP_USER_AGENT': 'IGV Version 2.3.34 (41)06/19/2014 10:41 PM',

 'JAVA_HOME': '/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home',

 'LC_CTYPE': 'en_US.UTF-8',

 'LOGNAME': 'ersenkavak',

 'PATH': '/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/local/mysql/bin',

 'PATH_INFO': u'/seqpipeline/download/demo/run1/MID-4/1/MID-4.bam',

 'PYCHARM_HOSTED': '1',

 'PYTHONIOENCODING': 'UTF-8',

 'PYTHONPATH': '/Users/ersenkavak/Sites/genomize/genoseq:/Users/ersenkavak/Documents/genomize/seq/code',

 'PYTHONUNBUFFERED': '1',

 'QUERY_STRING': '',

 'REMOTE_ADDR': '127.0.0.1',

 'REMOTE_HOST': '',

 'REQUEST_METHOD': 'GET',

 'RUN_MAIN': 'true',

 'SCRIPT_NAME': u'',

 'SERVER_NAME': '1.0.0.127.in-addr.arpa',

 'SERVER_PORT': '8001',

 'SERVER_PROTOCOL': 'HTTP/1.1',

 'SERVER_SOFTWARE': 'WSGIServer/0.1 Python/2.7.7',

 'SHELL': '/bin/bash',

 'SSH_AUTH_SOCK': '/tmp/launch-syn4Ze/Listeners',

 'TMPDIR': '/var/folders/j6/1bdk77s155z01tzgdj018np80000gn/T/',

 'TZ': 'UTC',

 'USER': 'ersenkavak',

 'VERSIONER_PYTHON_PREFER_32_BIT': 'no',

 'VERSIONER_PYTHON_VERSION': '2.7',

 '__CF_USER_TEXT_ENCODING': '0x1F5:0:0',

 '__CHECKFIX1436934': '1',

 '_system_arch': 'x86_64',

 '_system_name': 'OSX',

 '_system_type': 'Darwin',

 '_system_version': '10.9',

 'rvm_bin_path': '/Users/ersenkavak/.rvm/bin',

 'rvm_delete_flag': '0',

 'rvm_path': '/Users/ersenkavak/.rvm',

 'rvm_prefix': '/Users/ersenkavak',

 'rvm_ruby_string': 'system',

 'rvm_version': '1.25.25 (stable)',

 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x10e7ce1e0>,

 'wsgi.file_wrapper': <class wsgiref.util.FileWrapper at 0x10fae4870>,

 'wsgi.input': <socket._fileobject object at 0x1102728d0>,

 'wsgi.multiprocess': False,

 'wsgi.multithread': True,

 'wsgi.run_once': False,

 'wsgi.url_scheme': 'http',

 'wsgi.version': (1, 0)}>

Lastly, my django code which is evoked by loading the URL is as follows,

def download(request, sitename, runname, samplename, analysisid, param):

    

    ....... Some prelimiary code.......

    file_wrapper = FileWrapper(file(resultingfilepath, 'rb'))

    file_mimetype = mimetypes.guess_type(resultingfilepath)

    response = HttpResponse(file_wrapper, mimetype='application/force-download')

    #response = HttpResponse(file_wrapper, content_type = file_mimetype ) commented out

    print smart_str(resultingfilepath)

    response['Content-Length'] = os.path.getsize( resultingfilepath )

    response['X-Sendfile'] = resultingfilepath

    #response['Content-Disposition'] = 'attachment; filename=%s' % smart_str(resultingfilename)

    #response['Accept-Encoding'] = 'gzip, deflate'

    response['Accept-Ranges'] = 'bytesi

    return response

Regards

Ersen

--

--- You received this message because you are subscribed to a topic in the Google Groups "igv-help" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/igv-help/dp6-FUjdX4s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to igv-help+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/53D50753.4030403%40broadinstitute.org.

For more options, visit https://groups.google.com/d/optout.

[Jim Robinson]

That's the request to IGV.  IGV itself will then make a series of requests to the server hosting your bam (specified in the path to the bam).  Some of these requests will have range headers, specifying a start and end byte position.    You will need to seek to the start position of the file, then read and return the bytes requested in the response.

Jim

...

To unsubscribe from this group and all its topics, send an email to igv-help+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/53D50753.4030403%40broadinstitute.org.

For more options, visit https://groups.google.com/d/optout.

--

Erşen Kavak, PhD

CEO, Genomize  Inc.

E: er...@genomize.com 

T: +90 212 970 06 88

F: +90 850 220 04 51

M: +90 534 075 14 90

-----------------

Esentepe Mah. Büyükdere Cad. 

Oyal İş Merkezi, 108/1, Kat 9

34394 Şişli İstanbul Turkey

http://www.genomize.com

--

---

You received this message because you are subscribed to the Google Groups "igv-help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to igv-help+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/CALtNb1HzLW1xpacRa3PG1tbRLLBWWApTSMRELUKphA%2Bzi1LZNw%40mail.gmail.com.

[ersen kavak]

Hi Jim, 

I got it working now, found a related GIST and adapted stuff from our discussions and some trials. wala..

Thanks again for your support, and hopefully, we getting this working at Django will help some other people in the future.

I want to summarize the key points, which has been a hassle for me to get this work:

1) IGV will look for a <BAMFILEPATH>.bai file, so make sure this is ready

2) IGV will look for a <BAMFILEPATH>.tdf file, this does not have to be there, but you need to return a HTTP404 if it is not there... 

3) The tuff part is to get the streaming working. For this, I started with the code snippet  at from https://gist.github.com/dcwatson/cb5d8157a8fa5a4a046e.

Very importantly, one must add the Content-Disposition html header to the response.. Otherwise, IGV will keep searching for something, without any log warning...

I want to send the final working code, so we have a working example..

This code resides in views.py (a note for django folks)

from django.http import StreamingHttpResponse

from wsgiref.util import FileWrapper

import mimetypes

import os

import re

class RangeFileWrapper (object):

    def __init__(self, filelike, blksize=8192, offset=0, length=None):

        self.filelike = filelike

        self.filelike.seek(offset, os.SEEK_SET)

        self.remaining = length

        self.blksize = blksize

    def close(self):

        if hasattr(self.filelike, 'close'):

            self.filelike.close()

    def __iter__(self):

        return self

    def next(self):

        if self.remaining is None:

            # If remaining is None, we're reading the entire file.

            data = self.filelike.read(self.blksize)

            if data:

                return data

            raise StopIteration()

        else:

            if self.remaining <= 0:

                raise StopIteration()

            data = self.filelike.read(min(self.remaining, self.blksize))

            if not data:

                raise StopIteration()

            self.remaining -= len(data)

            return data

def download(request, sitename, runname, samplename, analysisid, param):

    #all of the parameters except request are specific to my application. You just need a path

    from django.http import Http404

    import socket

    datapath = "PATH TO YOUR BAM FILES"

    if param.endswith(".bam") or param.endswith(".bai"):

        ext = "bam.bai" if param.endswith(".bai") else "bam"

        import os

        path =  os.path.join(datapath, sitename, runname, samplename, analysisid, '%s.sorted.%s' % (samplename, ext))

        resultingfilename = "%s-%s-%s.%s" % (runname, samplename, analysisid, ext)

    elif param.endswith(".tdf"):

        raise Http404

    else:

        raise UserWarning, 'this is not supported yet...'

    range_re = re.compile(r'bytes\s*=\s*(\d+)\s*-\s*(\d*)', re.I)

    range_header = request.META.get('HTTP_RANGE', '').strip()

    range_match = range_re.match(range_header)

    size = os.path.getsize(path)

    content_type, encoding = mimetypes.guess_type(path)

    content_type = content_type or 'application/octet-stream'

    if range_match:

        first_byte, last_byte = range_match.groups()

        first_byte = int(first_byte) if first_byte else 0

        last_byte = int(last_byte) if last_byte else size - 1

        if last_byte >= size:

            last_byte = size - 1

        length = last_byte - first_byte + 1

        resp = StreamingHttpResponse(RangeFileWrapper(open(path, 'rb'), offset=first_byte, length=length), status=206, content_type=content_type)

        resp['Content-Length'] = str(length)

        resp['Content-Range'] = 'bytes %s-%s/%s' % (first_byte, last_byte, size)

    else:

        resp = StreamingHttpResponse(FileWrapper(open(path, 'rb')), content_type=content_type)

        resp['Content-Length'] = str(size)

    resp['Accept-Ranges'] = 'bytes'

    resp['Content-Disposition'] = 'attachment; filename=%s' % param

    return resp

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/53D6A4C7.6070202%40broadinstitute.org.

For more options, visit https://groups.google.com/d/optout.

[Bettina Harr]

Hello,

I am new to IGV. I have a set of bam and bai files on a server. I started the software locally on my Mac typing: sh igv.command

The software then starts up correctly, but when I try and load my files (manually) I get the following error:

INFO [2015-04-30 15:22:34,641]  [DirectoryManager.java:149] [main]  IGV Directory: /Users/harrb1/igv
INFO [2015-04-30 15:22:34,641] [DirectoryManager.java:149]  IGV Directory: /Users/harrb1/igv
INFO [2015-04-30 15:22:34,645]  [Main.java:85] [main]  Startup  IGV Version 2.3.51 (66)04/21/2015 11:11 AM
INFO [2015-04-30 15:22:34,645]  [Main.java:86] [main]  Java 1.7.0_67
INFO [2015-04-30 15:22:34,645]  [DirectoryManager.java:58] [main]  Fetching user directory...
INFO [2015-04-30 15:22:34,808]  [Main.java:87] [main]  Default User Directory: /Users/harrb1
INFO [2015-04-30 15:22:34,808]  [Main.java:88] [main]  OS: Mac OS X
INFO [2015-04-30 15:22:44,774]  [GenomeManager.java:130] [main]  Loading genome: /Users/harrb1/igv/genomes/mm10.genome
INFO [2015-04-30 15:22:45,890]  [GenomeManager.java:177] [main]  Genome loaded.  id= mm10
INFO [2015-04-30 15:22:46,017]  [CommandListener.java:88] [Thread-6]  Listening on port 60151
INFO [2015-04-30 15:23:14,293]  [IGV.java:1639] [pool-1-thread-6]  Loading 1 resources.
INFO [2015-04-30 15:23:14,295]  [MessageUtils.java:60] [pool-1-thread-6]  File not found: localhost:60151/load?file=http://http://wwwuser.gwdg.de/~evolbio/evolgen/wildmouse/m_m_domesticus/genomes/bam/14.recal.bam&genome=mm10/14.recal.bam

INFO [2015-04-30 15:23:48,202]  [IGV.java:1639] [pool-1-thread-7]  Loading 1 resources.
INFO [2015-04-30 15:23:48,203]  [TrackLoader.java:112] [pool-1-thread-7]  Loading resource, path http://wwwuser.gwdg.de/~evolbio/evolgen/wildmouse/m_m_domesticus/genomes/bam/14.recal.bam
INFO [2015-04-30 15:24:05,577]  [HttpUtils.java:855] [pool-1-thread-7]  Range-byte request succeeded
ERROR [2015-04-30 15:24:07,400]  [HttpUtils.java:217] [pool-1-thread-7]  Byte range requested, but no Content-Range header in response
ERROR [2015-04-30 15:24:08,493]  [HttpUtils.java:217] [pool-1-thread-7]  Byte range requested, but no Content-Range header in response


I have read this thread several times, but cannot understand what the solution of this problem can be.

Thank you very much for your help!

Tina

On Friday, July 25, 2014 at 7:49:11 PM UTC-5, Jim Robinson wrote:

Does the sever hosting the bam support range byte requests?  

On Friday, July 25, 2014, ersen kavak <ersen...@gmail.com> wrote:

Hello, 

I am using the IGV 2.3.34(41)

I have a 544 KB bam file at 

www.genomize.com/genera/deneme.bam 

and 

www.genomize.com/genera/deneme.bai and www.genomize.com/genera/deneme.bam.bai 

index files are also there... 

I am trying to load the file by:

http://localhost:60151/load?file=http://www.genomize.com/genera/deneme.bam&locus=MEFV&genome=hg19

And, without any errors it says loading for hours. When I try to download the file by using the link, it takes only seconds to download it. 

I have read the related section in the manual:

http://www.broadinstitute.org/software/igv/book/export/html/189

this thread, and actually many others, but could not understand what I am doing wrong.

https://groups.google.com/forum/#!msg/igv-help/BL4Qu41rjTk/Ggf-JglcRjkJ

What might I be doing wrong ?

Cheers

Ersen

--

---
You received this message because you are subscribed to the Google Groups "igv-help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to igv-help+unsubscribe@googlegroups.com.

[Jim Robinson]

Hi,

There are 2 errors here

>
> INFO [2015-04-30 15:23:14,295] [MessageUtils.java:60]
> [pool-1-thread-6] File not found:
> localhost:60151/load?file=http://http://wwwuser.gwdg.de/~evolbio/evolgen/wildmouse/m_m_domesticus/genomes/bam/14.recal.bam&genome=mm10/14.recal.bam
>

This is an error in the link, notice the double http://http://

> INFO [2015-04-30 15:23:48,202] [IGV.java:1639] [pool-1-thread-7]
> Loading 1 resources.
> INFO [2015-04-30 15:23:48,203] [TrackLoader.java:112]
> [pool-1-thread-7] Loading resource, path
> http://wwwuser.gwdg.de/~evolbio/evolgen/wildmouse/m_m_domesticus/genomes/bam/14.recal.bam
> INFO [2015-04-30 15:24:05,577] [HttpUtils.java:855]
> [pool-1-thread-7] Range-byte request succeeded
> ERROR [2015-04-30 15:24:07,400] [HttpUtils.java:217]
> [pool-1-thread-7] Byte range requested, but no Content-Range header
> in response
> ERROR [2015-04-30 15:24:08,493] [HttpUtils.java:217]
> [pool-1-thread-7] Byte range requested, but no Content-Range header
> in response

This indicates your sever does not support "range byte" headers, which
are required to load BAM and other indexed files via http. What type of
server is this?

Jim

[Gmail]

Hi Jim,

yes, I know the first, sorry, I should have not included that. The second command was the correct one.

Unfortunately, I don't know what operating system this server is using. I'll have to ask our system administrator. Is there anything IO could tell them to install in order for this software to work?

Thanks!
Tina

> --
>
> --- You received this message because you are subscribed to a topic in the Google Groups "igv-help" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/igv-help/dp6-FUjdX4s/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to igv-help+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/55429770.9080903%40broadinstitute.org.

[igv-help]

Hi Tina,

The important question is does the server support range byte requests?  This is a requirement for serving bam and other indexed files over http,  not just for IGV but any genome browser.   If supported it should return a Content-Range header in the response, along with a staus code of "206" for partial content.   Note this is a question about the web server  (apache, etc),  not the operating system.

Jim

[Bettina Harr]

Hi Jim,

our system administrators are working on it, at this point they don't know what the problem might be. This is what they sent me:

curl -I http://wwwuser.gwdg.de/~evolbio/evolgen/wildmouse/m_m_domesticus/genomes/bam/14.recal.bam
 
HTTP/1.1 200 OK
Date: Mon, 04 May 2015 09:49:03 GMT
Server: Apache
Last-Modified: Tue, 18 Mar 2014 02:44:20 GMT
ETag: "50dd0c19-18624510d6-4f4d8834fc900"
Accept-Ranges: bytes
Content-Length: 104727908566
Content-Type: text/plain; charset=ISO-8859-1
X-Pad: avoid browser bug

Does that narrow down the problem somewhat?

Thanks in advance!

Tina

[Jim Robinson]

Hi Tina,

Could you try adding a range byte header in the request, like this?

curl -H Range:bytes=16-32 -I http://wwwuser.gwdg.de/~evolbio/evolgen/wildmouse/m_m_domesticus/genomes/bam/14.recal.bam

Try that and tell me what it is printed out.   

Thanks

Jim

--

---

You received this message because you are subscribed to the Google Groups "igv-help" group.

To unsubscribe from this group and stop receiving emails from it, send an email to igv-help+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/0b65767e-3e25-4ea0-ae20-7124c4b798f1%40googlegroups.com.

[Bettina Harr]

Hi Jim,

here is the output:

curl -H Range:bytes=16-32 -I http://wwwuser.gwdg.de/~evolbio/evolgen/wildmouse/m_m_domesticus/genomes/bam/14.recal.bam

HTTP/1.1 206 Partial Content
Date: Mon, 04 May 2015 13:44:29 GMT

Server: Apache
Last-Modified: Tue, 18 Mar 2014 02:44:20 GMT
ETag: "50dd0c19-18624510d6-4f4d8834fc900"
Accept-Ranges: bytes

Content-Length: 17
Content-Range: bytes 16-32/104727908566
Content-Type: text/plain; charset=ISO-8859-1


THANKS!

[Jim Robinson]

That looks fine, which makes me curious about your original problem.
Could you try this again, from IGV, then close igv immediately and send
me your igv.log file?

Jim

[Bettina Harr]

Dear Jim,

When I start IGV and then Select: "File -> Load from URL" it just hangs in the loading forever. At the bottom left of the IGV screen it says "Loading...". In the terminal console where I started IGV I am getting

Tina-Imac:IGV_2.3.51 harrb1$ sh igv.command
INFO [2015-05-04 10:31:41,515]  [DirectoryManager.java:149] [main]  IGV Directory: /Users/harrb1/igv
INFO [2015-05-04 10:31:41,515] [DirectoryManager.java:149]  IGV Directory: /Users/harrb1/igv
INFO [2015-05-04 10:31:41,518]  [Main.java:85] [main]  Startup  IGV Version 2.3.51 (66)04/21/2015 11:11 AM
INFO [2015-05-04 10:31:41,518]  [Main.java:86] [main]  Java 1.7.0_67
INFO [2015-05-04 10:31:41,519]  [DirectoryManager.java:58] [main]  Fetching user directory...
INFO [2015-05-04 10:31:41,656]  [Main.java:87] [main]  Default User Directory: /Users/harrb1
INFO [2015-05-04 10:31:41,656]  [Main.java:88] [main]  OS: Mac OS X
INFO [2015-05-04 10:31:51,560]  [GenomeManager.java:130] [main]  Loading genome: /Users/harrb1/igv/genomes/mm10.genome
INFO [2015-05-04 10:31:52,585]  [GenomeManager.java:177] [main]  Genome loaded.  id= mm10
INFO [2015-05-04 10:31:52,704]  [CommandListener.java:88] [Thread-6]  Listening on port 60151
INFO [2015-05-04 10:32:41,884]  [IGV.java:1639] [pool-1-thread-6]  Loading 1 resources.
INFO [2015-05-04 10:32:41,885]  [TrackLoader.java:112] [pool-1-thread-6]  Loading resource, path http://wwwuser.gwdg.de/~evolbio/evolgen/wildmouse/m_m_domesticus/genomes/bam/14.recal.bam
INFO [2015-05-04 10:32:50,553]  [HttpUtils.java:855] [pool-1-thread-6]  Range-byte request succeeded
ERROR [2015-05-04 10:32:52,393]  [HttpUtils.java:217] [pool-1-thread-6]  Byte range requested, but no Content-Range header in response
ERROR [2015-05-04 10:32:53,707]  [HttpUtils.java:217] [pool-1-thread-6]  Byte range requested, but no Content-Range header in response

I am not getting my prompt back, it just hangs there.

Then I quit with control C. Attache the log file.

Does this make sense?

THANKS

Tina

[Jim Robinson]

Are you running the curl command from the same machine as IGV?

--

---
You received this message because you are subscribed to the Google Groups "igv-help" group.
To unsubscribe from this group and stop receiving emails from it, send an email to igv-help+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/cc209080-d356-4825-9d4c-08613a8f3cf6%40googlegroups.com.

[Bettina Harr]

no, the curl command were executed by our sys admins in Germany (servers are located in Germany (http://www.gwdg.de/index.php?id=home&no_cache=1&L=1)). I don't know from which machine they executed the curl command from. I am myself located in the US right now and trying to access the files from my local Mac OSX computer.

[Bettina Harr]

btw, the sys admins also installed IGV locally (though I am not sure on exactly which machine) and had the same error as I have.

[Jim Robinson]

Hi,

Thanks for your patience,  I was able to reproduce this locally.  The cause is subtle, and not what I originally thought.   IGV checks for the existence of certain optional files, in this case a coverage "tdf",  by performing a "head" request on the resource.    For you case that would http://wwwuser.gwdg.de/~evolbio/evolgen/wildmouse/m_m_domesticus/genomes/bam/14.recal.bam.tdf.   If the resource doesn't exist it expects to get an error, usually a 404 not found,  but the server above is returning a redirect (code 302) to http://www.gwdg.de/index.php?id=580.   This is causing igv to hang because it assumes the redirect is to the resource requested (in this case a "tdf" file) and instead is to something else.

The 404 code is the standard status code to indicate a file was not found.   The 302 code tells the client (in this case IGV) that the resource was found but in a different location.   I will have to think about how to handle this in a better way from our side,  we haven't encountered it before.

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/14c9d164-9ca2-4c73-b28e-96669534290b%40googlegroups.com.

[Jim Robinson]

Hi,

I think I have a fix for this,  could you try this development build of IGV?   Let me know if it works.

http://www.broadinstitute.org/igv/projects/downloads/snapshot/IGV_Snapshot.app.zip

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/igv-help/14c9d164-9ca2-4c73-b28e-96669534290b%40googlegroups.com.

[Bettina Harr]

brilliant! this works! I am not yet familiar with the software, but it looks just like I was hoping it would!

Thank you very much!

[Gmail]

Dear Jim,

When I start IGV and then Select: "File -> Load from URL" it just hangs in the loading forever. At the bottom left of the IGV screen it says "Loading...". In the terminal console where I started IGV I am getting 

Tina-Imac:IGV_2.3.51 harrb1$ sh igv.command 

INFO [2015-05-04 10:31:41,515]  [DirectoryManager.java:149] [main]  IGV Directory: /Users/harrb1/igv

INFO [2015-05-04 10:31:41,515] [DirectoryManager.java:149]  IGV Directory: /Users/harrb1/igv

INFO [2015-05-04 10:31:41,518]  [Main.java:85] [main]  Startup  IGV Version 2.3.51 (66)04/21/2015 11:11 AM

INFO [2015-05-04 10:31:41,518]  [Main.java:86] [main]  Java 1.7.0_67

INFO [2015-05-04 10:31:41,519]  [DirectoryManager.java:58] [main]  Fetching user directory... 

INFO [2015-05-04 10:31:41,656]  [Main.java:87] [main]  Default User Directory: /Users/harrb1

INFO [2015-05-04 10:31:41,656]  [Main.java:88] [main]  OS: Mac OS X

INFO [2015-05-04 10:31:51,560]  [GenomeManager.java:130] [main]  Loading genome: /Users/harrb1/igv/genomes/mm10.genome

INFO [2015-05-04 10:31:52,585]  [GenomeManager.java:177] [main]  Genome loaded.  id= mm10

INFO [2015-05-04 10:31:52,704]  [CommandListener.java:88] [Thread-6]  Listening on port 60151

INFO [2015-05-04 10:32:41,884]  [IGV.java:1639] [pool-1-thread-6]  Loading 1 resources.

INFO [2015-05-04 10:32:41,885]  [TrackLoader.java:112] [pool-1-thread-6]  Loading resource, path http://wwwuser.gwdg.de/~evolbio/evolgen/wildmouse/m_m_domesticus/genomes/bam/14.recal.bam

INFO [2015-05-04 10:32:50,553]  [HttpUtils.java:855] [pool-1-thread-6]  Range-byte request succeeded

ERROR [2015-05-04 10:32:52,393]  [HttpUtils.java:217] [pool-1-thread-6]  Byte range requested, but no Content-Range header in response

ERROR [2015-05-04 10:32:53,707]  [HttpUtils.java:217] [pool-1-thread-6]  Byte range requested, but no Content-Range header in response

I am not getting my prompt back, it just hangs there.