RE: UCSC Genome Browser Docker

[Uriel Rosen]

Hi,

 

Thanks for your reply , as of now no genome is working , I downloaded data of mm10 and hg38 to a shared network path,

Mounted it to the docker container and linked it to /var/lib/mysql and /gbdb/  .

I get the below error  , cannot run gbicSetup.sh in the docker as suggested , I would be happy to get online help from you such

As a zoom meeting so I can show you my configuration and you can consult me what to do to fix this .  

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Wednesday, March 3, 2021 9:23 AM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Subject: Re: UCSC Genome Browser Docker

 

Sorry I’m on vacation so better email the main support address we have.

 

Ok so hg38 works ?

 

For downloading entire genomes there is a browserSetup command in the gbic documentation. Have you read the gbic.html documentation on our website ? You can run gbicSetup.sh via “docker run”. have you used Docker before this ?

 

 

 

 

 

On Wed 3 Mar 2021 at 07:32, Uriel Rosen <uriel...@weizmann.ac.il> wrote:

More information about this issue ,

I need display genomes of mm10 and hg38 ,

When looking on the data base records I see this :

 

show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| customTrash        |

| hg18               |

| hg38               |

| hgFixed            |

| hgTemp             |

| hgcentral          |

| mm10               |

| mysql              |

| performance_schema |

| sys                |

+--------------------+

 

But when test access to the genomes on the web browser I get the following error :

 

How do I link the data to the databse ?

What am I missing ?

 

Thanks .

 

 

 

Thanks,

Uriel.

 

 

From: Uriel Rosen
Sent: Tuesday, March 2, 2021 9:02 AM
To: 'Maximilian Haeussler' <mhae...@ucsc.edu>
Subject: RE: UCSC Genome Browser Docker

 

Yes ,

 

The web page is up but I don’t know how to check if the system is connected to the data I downloaded , would be happy for a little guidance with this issue .

It would be best to have a zoom meeting if possible .

 

Thanks .

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Monday, March 1, 2021 5:46 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Subject: Re: UCSC Genome Browser Docker

 

Hi sorry for the delay, you need to pass through port 80 to the container I think it’s -p 80:8080 or something like that, then connect to your web browser to localhost:8080. this is all Standard docker stuff which you all probably know ?

 

 

On Mon 1 Mar 2021 at 14:05, Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Hi ,

 

Sorry to bother you again ,

Can you please if there is relevant documentation of using the docker of UCSC genome browser so I can check if my system is indeed working properly ?

 

 

 

Thanks,

Uriel.

 

 

From: Uriel Rosen
Sent: Sunday, February 28, 2021 9:09 AM
To: 'Maximilian Haeussler' <mhae...@ucsc.edu>
Subject: RE: UCSC Genome Browser Docker

 

Thanks

 

Is this it ?

How can I test if is working properly ?

 

If you are willing to have remote zoom session with me to check if I did everything correctly I would very much appreciate it .  

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Wednesday, February 24, 2021 11:56 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Subject: Re: UCSC Genome Browser Docker

 

Yes /gbdb

 

And it’s either /data/mysql or /var/lib/mysql I forgot ... one of these two...

 

On Wed 24 Feb 2021 at 22:27, Uriel Rosen <uriel...@weizmann.ac.il> wrote:

You mean /etc/mysql and /gbdb ?

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Wednesday, February 24, 2021 11:17 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Subject: Re: UCSC Genome Browser Docker

 

There are two data directories: MySQL and /gbdb. You can symlink your downloaded data there. 

 

 

 

On Wed 24 Feb 2021 at 22:10, Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Thank you very much  for your help ,
My UCSC genome browser mirror is almost ready (I think)  ,  I download genome files of hg38 an mm10 to a network storage and mounted it to the container ,
What I am missing now is how to link the data to the system , can you help with this  ?
I am an it guy(not scientific staff) and I don't really know exactly how this system is supposed to perform so I am really clueless about this .

Thanks,
Uriel.



-----Original Message-----
From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Wednesday, February 24, 2021 7:33 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Subject: Re: UCSC Genome Browser Docker

You can also pull it now:
https://hub.docker.com/repository/docker/genomebrowser/gbib

On Wed, Feb 24, 2021 at 6:04 PM Maximilian Haeussler <mhae...@ucsc.edu> wrote:
>
> It's working!!
>
> I'll try to upload the image... one sec... my new fiber connection is
> awesome for that...
>
> On Wed, Feb 24, 2021 at 5:51 PM Maximilian Haeussler <mhae...@ucsc.edu> wrote:
> >
> > I changed the first line of the dockerfile to:
> >
> > FROM phusion/baseimage:0.10.1
> >
> > It looks like Docker phusion doesn't accept the "latest" keyword
> > anymore, one has to supply the version number now.
> >
> > It started to build, now I'm testing it...
> >
> >
> > On Wed, Feb 24, 2021 at 1:17 PM Uriel Rosen <uriel...@weizmann.ac.il> wrote:
> > >
> > > Starnge I had another error.
> > >
> > >
> > > Thanks,
> > > Uriel.
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Maximilian Haeussler <mhae...@ucsc.edu>
> > > Sent: Wednesday, February 24, 2021 1:48 PM
> > > To: Uriel Rosen <uriel...@weizmann.ac.il>
> > > Subject: Re: UCSC Genome Browser Docker
> > >
> > > Hi Uriel, sorry for the delay... looking now...
> > >
> > > On Wed, Feb 24, 2021 at 8:50 AM Uriel Rosen <uriel...@weizmann.ac.il> wrote:
> > > >
> > > > Hi ,
> > > >
> > > > Any chance you can fix this error ?
> > > > Your help will be very appreciated .
> > > >
> > > >
> > > > Thanks,
> > > > Uriel.
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Uriel Rosen
> > > > Sent: Monday, February 22, 2021 7:02 PM
> > > > To: 'Maximilian Haeussler' <mhae...@ucsc.edu>
> > > > Subject: RE: UCSC Genome Browser Docker
> > > >
> > > > Hi ,
> > > >
> > > > Thanks  ,
> > > > Now I get the below error , also , what do you suggest about the genome hg38 download ?
> > > >
> > > > Reading package lists...
> > > > Building dependency tree...
> > > > Reading state information...
> > > > Package python-mysqldb is not available, but is referred to by another package.
> > > > This may mean that the package is missing, has been obsoleted,
> > > > or is only available from another source
> > > >
> > > > E: Package 'python-mysqldb' has no installation candidate The
> > > > command '/bin/sh -c apt-get install -yq wget rsync && sed -i
> > > > 's/101/0/g' /usr/sbin/policy-rc.d && apt-get install udev || sed
> > > > -i '2iexit 0' /etc/init.d/udev && service udev start && chmod
> > > > a+x /root/browserSetup.sh && /root/browserSetup.sh -b install &&
> > > > apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
> > > > && chmod a+x /etc/service/mysqld/run /etc/service/apache/run'
> > > > returned a non-zero code: 100
> > > >
> > > >
> > > > Thanks,
> > > > Uriel.
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Maximilian Haeussler <mhae...@ucsc.edu>
> > > > Sent: Monday, February 22, 2021 4:12 PM
> > > > To: Uriel Rosen <uriel...@weizmann.ac.il>
> > > > Subject: Re: UCSC Genome Browser Docker
> > > >
> > > > Sorry for the missing documentation.
> > > >
> > > > It's here now:
> > > > https://github.com/ucscGenomeBrowser/kent/blob/master/src/produc
> > > > t/installer/docker/README.md
> > > >
> > > > On Mon, Feb 22, 2021 at 3:08 PM Maximilian Haeussler <mhae...@ucsc.edu> wrote:
> > > > >
> > > > > Yes, you need to download the three files:
> > > > > https://github.com/ucscGenomeBrowser/kent/tree/master/src/prod
> > > > > uct/inst aller/docker Sorry I thought this was obvious, I'll
> > > > > document it now.
> > > > >
> > > > > On Mon, Feb 22, 2021 at 2:40 PM Uriel Rosen <uriel...@weizmann.ac.il> wrote:
> > > > > >
> > > > > > Tried to build from your docker file and got this error :
> > > > > >
> > > > > > Step 4/10 : ADD mysql.service /etc/service/mysqld/run ADD failed:
> > > > > > stat
> > > > > > /var/lib/docker/tmp/docker-builder055748002/mysql.service:
> > > > > > no such file or directory
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Uriel.
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Uriel Rosen
> > > > > > Sent: Monday, February 22, 2021 3:21 PM
> > > > > > To: 'Maximilian Haeussler' <mhae...@ucsc.edu>
> > > > > > Subject: RE: UCSC Genome Browser Docker
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Haven't tried it yet , do you recommend using it instead ?
> > > > > > Regarding my question ,do you  how do I link my local mirror docker to the local data and where do I download it from ?
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Uriel.
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Maximilian Haeussler <mhae...@ucsc.edu>
> > > > > > Sent: Monday, February 22, 2021 3:04 PM
> > > > > > To: Uriel Rosen <uriel...@weizmann.ac.il>
> > > > > > Subject: Re: UCSC Genome Browser Docker
> > > > > >
> > > > > > Hi Uriel,
> > > > > >
> > > > > > Icebert's docker container is not ours. I cannot answer questions on it.
> > > > > >
> > > > > > Daniel from support also sent you a Dockerfile that we support. Have you tried it?
> > > > > >
> > > > > > best
> > > > > > Max
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 22, 2021 at 12:37 PM Uriel Rosen <uriel...@weizmann.ac.il> wrote:
> > > > > > >
> > > > > > > Hi thanks for your reply,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I followed the below  instructions to get It installed , I
> > > > > > > get the docker up and running but the scientist who
> > > > > > > checked it says that the links aren’t working and
> > > > > > >
> > > > > > > He needs the hg38 genome data ,  can you explain how to download  this data into my docker image  ?
> > > > > > >
> > > > > > > If you have instructions with examples I would appreciate if you could send it to me .
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Download
> > > > > > >
> > > > > > > docker pull icebert/ucsc_genome_browser
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > docker pull icebert/ucsc_genome_browser_db
> > > > > > >
> > > > > > > Demo Run
> > > > > > >
> > > > > > > docker run -d --name gbdb -p 3338:3306
> > > > > > > icebert/ucsc_genome_browser_db
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > docker run -d --link gbdb:gbdb -p 8038:80
> > > > > > > icebert/ucsc_genome_browser
> > > > > > >
> > > > > > > Run with local data files
> > > > > > >
> > > > > > > Assume local data is going to be stored in /my/data/path
> > > > > > >
> > > > > > > First copy the basic database files into /my/data/path
> > > > > > > from docker
> > > > > > >
> > > > > > > docker run -d --name gbdb -p 3338:3306
> > > > > > > icebert/ucsc_genome_browser_db
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > cd /my/data/path && docker cp gbdb:/data ./ && mv data/*
> > > > > > > ./ && rm -rf data
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > docker stop gbdb
> > > > > > >
> > > > > > > Then put database files into /my/data/path. For example,
> > > > > > > mirror all the tracks of hg38 from ucsc genome browser
> > > > > > >
> > > > > > > rm -rf /my/data/path/hg38
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > rsync -avP --delete --max-delete=20
> > > > > > > rsync://hgdownload.soe.ucsc.edu/mysql/hg38 /my/data/path/
> > > > > > >
> > > > > > > Finally start the database server and genome browser
> > > > > > > server
> > > > > > >
> > > > > > > docker run -d --name gbdb -p 3338:3306 -v
> > > > > > > /my/data/path:/data icebert/ucsc_genome_browser_db
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > docker run -d --link gbdb:gbdb -p 8038:80
> > > > > > > icebert/ucsc_genome_browser
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Uriel.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: Maximilian Haeussler <mhae...@ucsc.edu>
> > > > > > > Sent: Monday, February 22, 2021 12:13 PM
> > > > > > > To: Uriel Rosen <uriel...@weizmann.ac.il>
> > > > > > > Subject: UCSC Genome Browser Docker
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi Uriel,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I'm the engineer who wrote the Docker support for the UCSC Genome Browser.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Please let me know if the Dockerfile works for you. I've tested it a few times, but two years ago.
> > > > > > >
> > > > > > > If it works, I'd like to announce it officially, so people don't find the icebert container anymore.
> > > > > > >
> > > > > > > If it doesn't work, I will fix it.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > thank you!
> > > > > > >
> > > > > > > Max

[Daniel Schmelter]

Hello Uriel,

Thanks for contacting the Genome Browser team about GBIC setup through Docker.

I have some ideas as to what may fix that error and we are happy to help. I will repeat that we do not yet have an official Docker version that is documented or supported, so I cannot speak with certainty. This is something we are increasingly looking into, so we appreciate your testing and input.

The first thing to check with this error is that you actually have the gbicSetup.sh script. It may not have been included in the Dockerfile properly. It can be downloaded for free for any non-profit institution from our Genome-Browser store at the bottom of this page:

https://genome-store.ucsc.edu

You may also have to enable that file execute permission or else you will get a "Permission denied" error. You can enable this with chmod like so.

chmod +x browserSetup.sh

Relevant instructions for GBIC can be found here:

https://genome.ucsc.edu/goldenpath/help/gbic.html#quick-start-instructions

I hope this was helpful. If you have any more questions, please reply-all to gen...@soe.ucsc.edu. All messages sent to that address are publicly archived. If your question includes sensitive data, please reply-all to genom...@soe.ucsc.edu.

All the best,

Daniel Schmelter
UCSC Genome Browser

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/6f3fc46182494b3cb8faff62dbf2adcb%40weizmann.ac.il.

[Uriel Rosen]

Hi ,

 

Running the install script gave me the following message :

 

| error: the file /usr/local/apache/cgiInstallComplete.flag exists. It seems that you have installed the browser already.

 

So according to the script test it seems that everything was installed  , I followed your documentation and changed the state to offline state in

The hg.conf , the docker container is mounted with a folder that I predownloaded the data of mm10 and hg38 to.

As explained by Maximilian I created symbolic links as following :

 

ln -s /shareDB/UCSC/mm10/ /var/lib/mysql

ln -s /shareDB/UCSC/hg38/ /var/lib/mysql

 

ln -s /shareDB/UCSC/mm10/ /gbdb/

ln -s /shareDB/UCSC/hg38/ /gbdb/

what else am I missing here ?

I would really appreciate if I could share my screen with you(with zoom or webex or any other software like this) so that you can speed up the process of troubleshooting for this issue because a few scientists are waiting to use it .

[Uriel Rosen]

Hi ,

 

I installed the genome browser using virtualbox today after my vmware guy didn’t succeed .

I am now on the stage you see below in the screenshots .

From what I understood in your documentation I am supposed to see the genome browser now but only see the folders shown in the image .

Can you help ?

 

 

 

 

 

   

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Thursday, March 11, 2021 11:53 AM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Cc: daniella apelblat <daniella....@gmail.com>; Christopher Lee <chm...@ucsc.edu>; Daniel Schmelter <dsch...@ucsc.edu>
Subject: Re: UCSC Genome Browser Docker

 

We know from http://genomewiki.ucsc.edu/index.php/GBiB:_Deploying_on_VMWare_ESXi that it works on Vmware, but I admit that it could be easier.

 

If your vmware guy has trouble, I'm sure we can find a VmWare license around here and get it to work. We've had this question come up a few times now.

 

On Thu, Mar 11, 2021 at 10:47 AM Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Hi ,

 

My vmware guy didn’t answer me yet regarding this , will try your suggestion also with virtual box and which option is best for us.

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Thursday, March 11, 2021 11:38 AM
To: daniella apelblat <daniella....@gmail.com>
Cc: Uriel Rosen <uriel...@weizmann.ac.il>; Christopher Lee <chm...@ucsc.edu>; Daniel Schmelter <dsch...@ucsc.edu>
Subject: Re: UCSC Genome Browser Docker

 

I assume these are patient/clinically protected samples?

 

Great, yes, this is why we created the GBIB genome browser virtual machine: it runs a full genome browser, gets updated remotely every two weeks and hopefully (though I can't check this), since we have a mirror in the EU, is fast enough from Israel without downloading terrabytes of data.

Hi Uriel, did running GBIB on your Vmware systems work, after the changes pointed out in the wiki entry? If not, we can probably even buy a VMWare license and try to make it work on Vmware.

As a temporary solution, could you install VirtualBox on one of your servers? Then running GBIB would be easy: https://genome-test.gi.ucsc.edu/goldenPath/help/gbib.html

 

You can then load tracks simply from a local directory on the server, by pasting the BAM/VCF/bigBed/etc. file names into our custom track box.

 

Please note that the best way to load tracks into your own genome browser would be track hubs. You can write little text files that load the tracks, and specify colors, order, labels, etc.

 

We have a brand new feature where your track hubs could be auto-connected in your browser, so every time someone goes to your own browser, they will see your tracks as your own track group. Please contact us for more information, the feature is not documented yet.

 

On Thu, Mar 11, 2021 at 10:32 AM daniella apelblat <daniella....@gmail.com> wrote:

Hey Uriel, sure I can explain. 

We in the lab would like to use the UCSC browser such that we can store our bigwig files in a permanent place and view them using the browser. 

Our understanding is that this can be done by storing the files on our internal server and viewing them via a UCSC mirror. 

Due to security reasons, I cannot put these files on an external ftp server such that they are constant there and linked to the global browser. 

This is a problem not just for our lab, but actually to a number of labs we spoke with in the institute, and a main reason why some people are actually not regularly working with this browser (though they would otherwise like to). 

I hope this helps, let me know if you have any other questions. 

Best, 

 

Daniella

 

On Wed, 10 Mar 2021 at 16:05, Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Hi ,

 

I am cc’ing  Daniella to this thread so she can describe her needs and usage so you can better understand why we need a local mirror and maybe you can offer us a good solution to our needs .

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Wednesday, March 10, 2021 3:28 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Cc: Christopher Lee <chm...@ucsc.edu>; Daniel Schmelter <dsch...@ucsc.edu>
Subject: Re: UCSC Genome Browser Docker

 

OK, sorry, this again is something we didn't test on VmWare ESX, just on virtualbox. However other people have done this and yes, apparently you need to make a few edits, sorry for bringing this up only now:

http://genomewiki.ucsc.edu/index.php/GBiB:_Deploying_on_VMWare_ESXi

 

That being said, we really want to understand why you want to setup a mirror. We cannot support all VM environments.  we're putting all our efforts into track hubs. We'd really like to know why you think you cannot use track hubs...

 

On Wed, Mar 10, 2021 at 11:47 AM Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Hi ,

 

Our vcenter guy tried to install your image ,

The error we got is below .

Any suggestions ?

 

 

 

 

Thanks,

Uriel.

 

 

From: Christopher Lee <chm...@ucsc.edu>
Sent: Tuesday, March 9, 2021 5:42 PM
To: Maximilian Haeussler <mhae...@ucsc.edu>
Cc: Uriel Rosen <uriel...@weizmann.ac.il>; Daniel Schmelter <dsch...@ucsc.edu>
Subject: Re: UCSC Genome Browser Docker

 

Hi Uriel,

 

Quite a while ago I created a .ova, you can download it from here:

https://genome-test.gi.ucsc.edu/~chmalee/gbibVmWare/browserbox.ova

 

I have no idea if it will work for you now as it's fairly old. Unfortunately I believe I also couldn't even really test it because at the time I wasn't even able to install the necessary vmware software on my machine due to licensing constraints.

 

Let me know if it works for you and we can start recommending it more.

 

On Tue, Mar 9, 2021 at 6:33 AM Maximilian Haeussler <mhae...@ucsc.edu> wrote:

Oh yes!! A lot easier. 

 

Because Docker is not a full linux, it's hard to use things like cronjobs there, for regular updates. We could try to convert our VM into a container, but I'm unsure how to do that. 

 

For now, yes, it would be quite a bit easier to convert the Virtualbox virtual machine to VMWare. I've CC'ed Chris, who I think made an .ova virtual appliance a while ago, which could be useful here.

 

best

Max

 

On Tue, Mar 9, 2021 at 3:17 PM Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Hi ,

 

My boss is Vadim Malkin , unfortunately he is away due to some illness for the next month or so ,

Anyway , do you recommend to try to install a mirror on our vmware vcenter , is the process easier to install ?

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Tuesday, March 9, 2021 3:48 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>; Daniel Schmelter <dsch...@ucsc.edu>
Subject: Re: UCSC Genome Browser Docker

 

Hi Uriel,

Running, updating a mirror can be a lot of work. Can you tell me who your boss is? We know a lot of people at Weizman, maybe your boss? I ask because I wonder why your boss told you to setup a mirror. Like your last mirror, it's possible that this mirror again will soon stop working or at least differ a lot in features from the public site and which point, someone will have to setup updates again.

 

Our full GBIB virtual machine was meant to solve this use case. Is there a reason why you don't like the VM ? It works on Vmware, Virtualbox and all other hosting environments. 

 

But really, ideally, you would avoid setting up a full mirror. track hubs and assembly hubs require only some webspace and will continue to work for many years, without any changes. Are you sure you need a mirror?

 

best

Max

 

On Tue, Mar 9, 2021 at 1:25 PM Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Hi ,

 

How are you ?

Thanks for your reply .

we have a cluster of linux servers , my boss asked me to check the option of

Using your software using docker because we had some issues using the linux installation ,

What is the meaning of a full mirror? I downloaded only 2 sets of genomes mm10 and hg38 ,

Is the gbib easier  to install on a linux server ?

Anyway , I hope that we are close to solving this issue, I sent my current status to the support and am waiting for a response .

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Monday, March 8, 2021 4:32 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Subject: Re: UCSC Genome Browser Docker

 

Hi Uriel,

 

back finally. Why are you using docker at all? It seems you're not used to Docker or the Unix command line. Wouldn't GBIB be the better solution for you?

Do you have a linux server where you're trying to install this? Are you sure it's a good idea to install a full mirror?

See: https://genome.ucsc.edu/goldenPath/help/mirror.html

 

On Thu, Mar 4, 2021 at 3:48 PM Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Hi ,

 

Sorry to bother your on vacation , I emailed gen...@soe.ucsc.edu  just cc’ed  you .

I was referring to the error from the previous email ,

Is gen...@soe.ucsc.edu  the support email or is it another address ?

When sshing into the docker and trying to run gbicSetup.sh  I get the error “command not found” ,

I couldn’t find documentation for the docker , can you forward the email to the support or write the corret email ?

 

Thanks  ,

Enjoy your vacation.  

 

 

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Thursday, March 4, 2021 4:28 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Subject: Re: UCSC Genome Browser Docker

 

> I get the below error

which one?

 

> cannot run gbicSetup.sh in the docker as suggested

why? It's in the docker container. Did you ssh into it and run it? 

 

If it doesn't work, what is the error message?

 

I'm still on vacation, and for the full week, as I said. Please email the support email address if you need support.

 

 

 

--

Daniella Apelblat

Spiegel Lab, Neurobiology Department

Weizmann Institute of Science

[Brian Lee]

Dear Uriel,

Thank you for sharing this message with us. An engineer suggests the following solution, run "updateBrowser.sh notSelf" as root in the /root directory.

We are working to update the file at our https://genome-store.ucsc.edu/ location, but the above commands should allow you to solve this problem without re-downloading a new virtual machine image.

All the best,

Brian

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.

To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/59c3abd4e5cc4fb598448c01f5b6fa15%40weizmann.ac.il.

--

Brian Lee

UCSC Genome Browser  

UC Santa Cruz Genomics Institute |  Revealing life’s code.

Google Scholar | Twitter | Facebook | YouTube 

[Uriel Rosen]

The reference to local data is automatic or do I have to do some action for that to happen ?

 

Also , if I choose in the future to download the data somewhere else(like network path) how do I refer it to the new path ?

 

Couldn’t find this in the documentation so can you please tell me where this information is located .

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>

Sent: Thursday, March 18, 2021 4:44 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>

Cc: Brian Lee <bria...@soe.ucsc.edu>; gen...@soe.ucsc.edu; daniella apelblat <daniella....@gmail.com>; Christopher Lee <chm...@ucsc.edu>; Daniel Schmelter <dsch...@ucsc.edu>
Subject: Re: [genome] RE: UCSC Genome Browser Docker

 

The data is already viewable in the browser, it is loaded remotely through the internet which is a bit slower. After the rsync completes, the data display should be a lot faster, as the data is local now.

 

On Thu, Mar 18, 2021 at 3:42 PM Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Hi ,

 

Your suggestion worked , thanks ,

The browser in now accessible  ,

I am now running rsync (command below) for local data download of mm10 for Daniella , it is taking hours , after the download completes  how do I sync the data to be viewable in the browser ?

 

rsync hgdownload.soe.ucsc.edu::mysql/mm10/* /data/mysql/mm10/.

 

 

 

Thanks,

Uriel.

[Maximilian Haeussler]

I don't think there is anything you need to do. You can wait for the rsync to complete and then the GBIB should be faster

(is it too slow right now from Israel? We can't check this from where we are).

Once it is faster you can load your local data from /folder as described in the GBIB manual

https://genome-test.gi.ucsc.edu/goldenPath/help/gbib.html#YourTracks

[Maximilian Haeussler]

The data is already viewable in the browser, it is loaded remotely through the internet which is a bit slower. After the rsync completes, the data display should be a lot faster, as the data is local now.

On Thu, Mar 18, 2021 at 3:42 PM Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Hi ,

 

Your suggestion worked , thanks ,

The browser in now accessible  ,

I am now running rsync (command below) for local data download of mm10 for Daniella , it is taking hours , after the download completes  how do I sync the data to be viewable in the browser ?

 

rsync hgdownload.soe.ucsc.edu::mysql/mm10/* /data/mysql/mm10/.

 

 

 

Thanks,

Uriel.

 

 

[Uriel Rosen]

Hi ,

 

Your suggestion worked , thanks ,

The browser in now accessible  ,

I am now running rsync (command below) for local data download of mm10 for Daniella , it is taking hours , after the download completes  how do I sync the data to be viewable in the browser ?

 

rsync hgdownload.soe.ucsc.edu::mysql/mm10/* /data/mysql/mm10/.

 

 

 

Thanks,

Uriel.

 

 

[Uriel Rosen]

Hi ,

 

After the rsync is complete I see the below error when I try to display the mm10 data :

What do you recommend to do now ?

 

 

 

Thanks,

Uriel.

[Maximilian Haeussler]

Hi Uriel,

this looks like a different and unrelated problem, the VM cannot establish a connection to genome-euro. I just checked, genome-euro is working fine for me. 

Could this be related to a firewall, did you change something on the VM or your network between a few days ago and now?

best

Max

[Uriel Rosen]

Hi ,

 

Could you please write me the address of the euro server and port and I will check from the VM if it can access  it with telnet check .  

[Uriel Rosen]

I just tested like this :

telnet genome-euro-mysql.soe.ucsc.edu 3306

 

 

please tell me if it is correct .

if the test is correct I will check with my security IT department.

[Maximilian Haeussler]

Yes, if I use your command, I'm seeing the MariaDB version and it's asking for a password, but of course that won't work as the the session is using the binary MySql protocol.

In your case, I imagine it doesn't show anything, I assume that the port is blocked in your firewall.

[Uriel Rosen]

Hi ,

 

From a user’s testing on my new mirror it seems performance is rather slow when browsing within the data  ,

I followed your documentation regarding the virtualbox resources and assigned 45 gigs of ram with 6 cpus to the machine but this didn’t make a difference .

I suspect that the local data is not being used and all operations approach your remote database rather than the local database .

Attached my configuration file to this mail so you can tell me if it is configured to reach the local data correctly .

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Wednesday, March 24, 2021 3:09 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Cc: daniella apelblat <daniella....@gmail.com>; Brian Lee <bria...@soe.ucsc.edu>; gen...@soe.ucsc.edu; Christopher Lee <chm...@ucsc.edu>; Daniel Schmelter <dsch...@ucsc.edu>
Subject: Re: [genome] RE: UCSC Genome Browser Docker

 

Hi Uriel,

 

Well... you set up this genome browser because you wanted an internal website for your own protected data. You then sent us your internal website to test it. However, as it is an internal website by design, the website is not accessible for us.

 

Hope this makes sense,

Best

Max

 

 

On Wed, Mar 24, 2021 at 1:40 PM Uriel Rosen <uriel...@weizmann.ac.il> wrote:

Hi Daniella ,

 

After opening the port in the firewall as suggested by Maximilian it looks like the browser is now working , thanks Maximilian ,

I would like you to test the system now and tell us if everything is working as expected.

 

Please use this url :

 

http://ibaucscbwv01.wexac.weizmann.ac.il:1234

[Uriel Rosen]

Hi ,

 

Thanks for your reply ,

My end result is to have the data on an external mount outside the virtual machine  , I checked

If I see the /media/sf_UCSC/hg19/hg19.2bit but it doesn’t exist there (but I do have /media/sf_UCSC/hg19 folder with content ), it is present at /data/mysql/hg19/hg19.2bit   , can you guide me on how I can sync the data correctly to be at  /media/sf_UCSC/    ?

Can I use your script to download the data to   /media/sf_UCSC/    ?

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Tuesday, April 6, 2021 1:16 PM
To: Uriel Rosen <uriel...@weizmann.ac.il>
Cc: Brian Lee <bria...@soe.ucsc.edu>; gen...@soe.ucsc.edu; Christopher Lee <chm...@ucsc.edu>; Daniel Schmelter <dsch...@ucsc.edu>; daniella apelblat <daniella....@gmail.com>
Subject: Re: [genome] RE: UCSC Genome Browser Docker

 

Hi Uriel,

 

From your hg.conf, it seems that you rsync'ed the gbdb files to /media/sf_UCSC/ and not to /data/gbdb:

 

# redirect /gbdb access to local directory first and fallback to UCSC
# add the trailing slash
#gbdbLoc1=/data/gbdb/
gbdbLoc1=/media/sf_UCSC/
gbdbLoc2=http://hgdownload.soe.ucsc.edu/gbdb/

 

That should be fine, as long as the gbdb files are really there now, for example, the file /media/sf_UCSC/hg19/hg19.2bit must exist. If the files are not where they should be, they won't be picked up.

 

You kept the existing mysql server that was setup during gbic installation:

 

# read mysql data first from localhost and fallback to UCSC
db.host=localhost
db.user=root
db.password=browser
db.socket=/var/run/mysqld/mysqld.sock

 

Is the additional data that you downloaded really under /data/mysql (or /var/lib/mysql) ? And accessible to the mysql server?

 

the gbic browserSetup script has options to download data into the correct directories. Can you remind me why you wanted to do the rsync yourself and not use the options we provide in the script? 

[Maximilian Haeussler]

Hi Uriel,

From your hg.conf, it seems that you rsync'ed the gbdb files to /media/sf_UCSC/ and not to /data/gbdb:

# redirect /gbdb access to local directory first and fallback to UCSC
# add the trailing slash
#gbdbLoc1=/data/gbdb/
gbdbLoc1=/media/sf_UCSC/
gbdbLoc2=http://hgdownload.soe.ucsc.edu/gbdb/

That should be fine, as long as the gbdb files are really there now, for example, the file /media/sf_UCSC/hg19/hg19.2bit must exist. If the files are not where they should be, they won't be picked up.

You kept the existing mysql server that was setup during gbic installation:

# read mysql data first from localhost and fallback to UCSC
db.host=localhost
db.user=root
db.password=browser
db.socket=/var/run/mysqld/mysqld.sock

Is the additional data that you downloaded really under /data/mysql (or /var/lib/mysql) ? And accessible to the mysql server?

the gbic browserSetup script has options to download data into the correct directories. Can you remind me why you wanted to do the rsync yourself and not use the options we provide in the script? 

[Maximilian Haeussler]

Hi Uriel,

it sounds like you confused the target directories and you copied /gbdb/hg19 data into /data/mysql/hg19/. .MYI and .frm files are from mysql. files like .2bit belong into the /gbdb directory. You can try to move the files around now, if you can still pull them apart. 

You could also move away the directories and use the "browserSetup.sh mirror" command to mirror the data, it has a "minimal" option to skip the ENCODE and a few other very big tracks. 

But a download as big as this may take a while from Israel, it may be quicker to move the files around. 

[Uriel Rosen]

Will try to move the files as you suggested and let you how this goes.

[Uriel Rosen]

Hi ,

 

After moving the files to /media/sf_UCSC/   the performance stays the same , how can I check if I am working locally or reaching out to your external mirror  ?

[Maximilian Haeussler]

Did you check that the file that I mentioned, /media/sf_UCSC/hg19/hg19.2bit now exists? There should be many other files now in this directory, mostly .bb and .bw.

Are they accessible for your apache user (often www-data) ?

To debug further, you can look at the apache log file. Depending on your linux distro, it's under /var/log/apache. Every gbdb remote access should be noted there with the reason why.

 

[Uriel Rosen]

Yes ,

The file exists , when I look at the  /var/log/apache2/access.log  I see access to a local ip address 10.0.2.2 probably the virtual machine .

But I see the files ownership is not the same it was , do I need to change the ownership of the files to  www-data   ?

[Maximilian Haeussler]

1) yes, the file has to be accessible for the www-data user for reading. You can see any Unix tutorial on chmod for more information. You probably want to change permissions of all files under this gbdb directory.

2) sorry I mean the error.log, not the access.log What does the error.log say about this file?

[Uriel Rosen]

Attached the error log file ,

I tried to change the setting back to  gbdbLoc1=/data/gbdb/ and I still see in the log that the server is reaching out and not working locally ,

Can you please tell me how can I set this up correctly so that this will work properly with my external mount to the VM .

If it is possible to get support with a live session with your support I would really appreciate this .

[Maximilian Haeussler]

The logfile suggests that most of the data is still downloaded from
UCSC. Most of the delay actually comes from mysql connections to UCSC.
Did you also rsync the mysql databases in /data/mysql ?

If you use GBIB (which we recommend for your case, as it's pre-configured):

There are various commands to show what was loaded, see our
documentation on the topic:
https://genome-test.gi.ucsc.edu/goldenPath/help/gbib.html#Commands

You can use GBIB a little first.

Then, to show all data that had to be loaded for Mysql:
gbibUcscTablesLog

To show all gbdb files that had to be loaded from gbdb:
gbibUcscGbdbLog

[Uriel Rosen]

Hi ,

 

Thanks for your suggestions and all the help , It seems now that we are working ok with offline data , there is an issue that Daniella encountered when

Trying to work in the system , she is trying to access genome browser>> reset all user settings and getting the bellow page , can you tell me what is missing for this to work as expected ?

 

 

 

[daniella apelblat]

Hi Uriel and all, 

Just to clarify- the issue is not in the reset user settings, but in the fact that I don't have the option to choose a genome, and therefore cannot really work with the system. 

Best, 

Daniella

[daniella apelblat]

Hey again, 

Uriel suggested to enter the browser directly from the Genomes pull-down menu, and this works. 

I selected the mm10 genome, and wish to view a publicly-accessible track, for example:

http://genome.ucsc.edu/cgi-bin/hgTracks?db=mm9&lastVirtModeType=default&lastVirtModeExtraState=&virtModeType=default&virtMode=0&nonVirtPosition=&position=chr10%3A87234285%2D87402644&hgsid=1092132331_ru4DwI5Zc5KAq9rkL7tmaEwFTBP8

Which looks as such:

To do this I pasted the URL starting from "htTracks" to the URL of our internal site, as such:

http://ibaucscbwv01.wexac.weizmann.ac.il:1234/cgi-bin/hgTracks?db=mm9&lastVirtModeType=default&lastVirtModeExtraState=&virtModeType=default&virtMode=0&nonVirtPosition=&position=chr10%3A87234285%2D87402644&hgsid=1092132331_ru4DwI5Zc5KAq9rkL7tmaEwFTBP8

However,  once the page loads I see I reached a different URL and the view is indeed different than expected:

What may be the reason for this?

And, following that- could we schedule an online session to go over the installed system with you guys, Uriel and I to see that all functionalities work as expected? I think it would be much easier, and surely faster for all of us. 

Best wishes, 

Daniella

[Maximilian Haeussler]

Hi Daniella,

sorry that part of the menu is missing. We'll look into that. Are you using the GBIB system still for this example? If yes, how much did you mirror: mm10 and hg19?

As for your data: the view you see on our public site has been configured by you and/or data has uploaded on the public site. You cannot take a URL, replace the server and expect it to work. Your internal site does not have the configurations that you made on the public site. All current track configurations together are what we call a "session". The settings are stored on our server and the long number after "hgsid" points to your personal "session" on our server.

What you can do is to go the public site, use the My Data > My Sessions menu, look for the option "Save current settings to a local file:", save the text file to disk, then go to your internal site, go again to My Data > My Sessions and look for the "Restore sessions from a local file" option, select your file, and you should see the same view (unless the view includes uploaded custom track data).

As for a phone call, I'll defer to our support group who are CC'ed here.

best

Max

[daniella apelblat]

Hi Max, 

The URL I sent is not of my session but rather, one that was shared with me by another user.  

Does this mean that in the internal site I don't have direct access to any publicly-accessible tracks? I need to first save them to a local file and then restore as you explain above?

Thanks a lot for the quick reply :)

Daniella

[Maximilian Haeussler]

Hi Daniella,

you should be able to see all public tracks, depending on how Uriel set up the GBIB (online data loading or not). But you cannot play with the URL and expact to see the same tracks, the URL itself does not contain the information which track is active or not. You will need to select the tracks again on your internal server or download/upload the session.

If a track is missing from your internal version, it's probably not been downloaded by Uriel. Putting into GBIB into "online" mode should activate all tracks, at the price of slower load times.

best

Max 

[Uriel Rosen]

Hi ,

 

For downloading I used this option of mm10 :

 

Default tracks with conservation tables, but no alignments (defaultConsTables): 158 tables, 9.2 GB + 34 gbdb files, 45.9 GB, downloaded

 

Is this related to the issue ? should I select other tracks to download ?

 

 

 

 

 

Thanks,

Uriel.

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Tuesday, April 20, 2021 4:38 PM
To: daniella apelblat <daniella....@gmail.com>
Cc: Uriel Rosen <uriel...@weizmann.ac.il>; Brian Lee <bria...@soe.ucsc.edu>; gen...@soe.ucsc.edu; Christopher Lee <chm...@ucsc.edu>; Daniel Schmelter <dsch...@ucsc.edu>
Subject: Re: [genome] RE: UCSC Genome Browser Docker

 

Hi Daniella,

[Maximilian Haeussler]

This option looks good. 

Otherwise: I can't see on Daniella's screenshot if there is a track missing from your internal server. 

[daniella apelblat]

I understand. Ok, so I guess the rest of my questions now are for the cc'd support group members. 

If you could just refer me to the relevant people in the cc'd list (I don't know who does what :) ) , I will be in direct contact with them. 

Thanks a lot!

Daniella

[Maximilian Haeussler]

I think that would be Brian Lee.

[Brian Lee]

Dear Daniella,

Thank you for working through these issues. I can help you with any further questions.

It may worth noting that this email thread goes back to March and currently includes our gen...@soe.ucsc.edu mailing list. All messages sent to that address are archived on a publicly accessible forum. When questions include sensitive data, we also have the genom...@soe.ucsc.edu mailing list, where the internal team can still assist. So at any time when you wish to contact the team without including the public list, the genom...@soe.ucsc.edu mailing list is available. There is also a publicly accessible forum called genome...@soe.ucsc.edu dedicated to mirror issues.

It looks like you have a session with data on the main site, genome.ucsc.edu, and you wish to port that session and view over to a locally run mirror version of the site. The best step would be to transfer down a session file of the contents (My Data menu -> My Sessions) and then upload that file on your mirror. You may have custom tracks on the main site, genome.ucsc.edu, in which case, you will also need to download those and attach them back up on the mirror.

Here is an example archived answer about transporting sessions between mirrors when custom tracks are involved:
https://groups.google.com/a/soe.ucsc.edu/g/genome/c/YASZICBGxWA/m/id1yKm-DBgAJ

Please feel free to contact me with further questions, and it is OK if you want to open a new thread to any of our other lists ( genom...@soe.ucsc.edu or genome...@soe.ucsc.edu).

All the best,