Re: [genome] hd19 data download

14 views
Skip to first unread message

Chris Villarreal

unread,
Apr 18, 2017, 4:06:45 PM4/18/17
to Mike Gavrielides, gen...@soe.ucsc.edu, genome...@soe.ucsc.edu, m...@soe.ucsc.edu

Dear Mike Gavrielides,

Thank you for your question about the UCSC Genome Browser. Unfortunately, we do not currently have a downloads server in Europe. We do offer UDR (UDT Enabled Rsync), https://github.com/LabAdvComp/UDR, a solution for slow download speeds. More information on UDR can be found on our downloads Page: http://hgdownload.soe.ucsc.edu/downloads.html, under the header "Source & Utilities Downloads."

I hope this is helpful. If you have any further questions, please reply to gen...@soe.ucsc.edu. All messages sent to that address are archived on a publicly-accessible Google Groups forum. If your question includes sensitive data, you may send it instead to genom...@soe.ucsc.edu.

-Chris V
UCSC Genome Browser


On Tue, Apr 18, 2017 at 9:26 AM, Mike Gavrielides <mike.gav...@crick.ac.uk> wrote:

Hi all,

 

I am a software developer at Francis Crick Institute, London. I am writing with regards to a full mirror genome browser setup and more specifically about the hg19 data transfer.

 

We have put the /mysql/hg19 and /gbdb/hg19 to download (see rsync commands below). The data transfer process is quite slow as for the last four days it just got to 1.3TB which means it will take another ~18 days to download the complete 7.2TB set.

 

Rsync command to download the data:

rsync -avzP --delete --max-delete=20 rsync://hgdownload.cse.ucsc.edu/mysql/hg19 /camp/stp/aim/working/ucsc_gb_hg19/mysql

rsync -avzP --delete --max-delete=20 rsync://hgdownload.cse.ucsc.edu/gbdb/hg19 /camp/stp/aim/working/ucsc_gb_hg19/gbdb

 

I was wondering if there is an alternative method to get the data. Is there any other location closer to Europe that will speed up the transfer?

 

Kind regards,

 

  Mike Gavrielides

Application Software Developer

Scientific Computing . AIM Team

The Francis Crick Institute

1 Midland Road

London

NW11AT

                     

www.crick.ac.uk

 

 

 

 

           

The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 1 Midland Road London NW1 1AT

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.
To post to this group, send email to gen...@soe.ucsc.edu.
Visit this group at https://groups.google.com/a/soe.ucsc.edu/group/genome/.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/B5456D42-4568-4B39-9DCB-D9B4F01AF797%40crick.ac.uk.
For more options, visit https://groups.google.com/a/soe.ucsc.edu/d/optout.

Christopher Lee

unread,
May 17, 2017, 2:25:14 PM5/17/17
to Mike Gavrielides, gen...@soe.ucsc.edu, genome...@soe.ucsc.edu, m...@soe.ucsc.edu, Chris Villarreal

Hi Mike,

Thank you for your question about getting human to show up in the species tree on your mirror. There are a few troubleshooting steps you can try to get to the bottom of what's going on.

1. What is the defaultDb for human?

- Please try this mysql command while connected to the hgcentral database:

select * from defaultDb where genome = "Human";

- If this returns hg38 like so:

+--------+------+
| genome | name |
+--------+------+
| Human  | hg38 |
+--------+------+

Then you will want to update the defaultDb for Human like so (you will need the appropriate MySQL update permissions for this):
update defaultDb set name = "hg19" where genome = "Human";

2. If the defaultDb for Human is already set to hg19, then you will want to make sure that your hg.conf file contains the line hgGateway.dbDbTaxonomy=/path/to/a/file, and that the file pointed to by hgGateway.dbDbTaxonomy contains "Human":
$ grep "hgGateway.dbDbTaxonomy" /usr/local/apache/cgi-bin/hg.conf
hgGateway.dbDbTaxonomy=../js/dbDbTaxonomy.js
$ grep "Human" /usr/local/apache/htdocs/js/dbDbTaxonomy.js 
                                                            [ ["Human", 9606, "Homo sapiens" ] ] ],

3. Lastly you will want to check if the dbDb table in the hgcentral database contains Human:
select * from dbDb where name='hg19';
+------+-------------------------+------------+----------+-----------------------------+--------+----------+--------+----------------+----------------------------------+----------+--------+-------------------------------------------------------------------------+-------+
| name | description             | nibPath    | organism | defaultPos                  | active | orderKey | genome | scientificName | htmlPath                         | hgNearOk | hgPbOk | sourceName                                                              | taxId |
+------+-------------------------+------------+----------+-----------------------------+--------+----------+--------+----------------+----------------------------------+----------+--------+-------------------------------------------------------------------------+-------+
| hg19 | Feb. 2009 (GRCh37/hg19) | /gbdb/hg19 | Human    | chr21:33,031,597-33,041,570 |      1 |       60 | Human  | Homo sapiens   | /gbdb/hg19/html/description.html |        1 |      0 | GRCh37 Genome Reference Consortium Human Reference 37 (GCA_000001405.1) |  9606 |
+------+-------------------------+------------+----------+-----------------------------+--------+----------+--------+----------------+----------------------------------+----------+--------+-------------------------------------------------------------------------+-------+

Please let us know if we can help you any further!

Thank you again for your inquiry and using the UCSC Genome Browser. If


you have any further questions, please reply to gen...@soe.ucsc.edu.
All messages sent to that address are archived on a

publicly-accessible forum. If your question includes sensitive data,


you may send it instead to genom...@soe.ucsc.edu.

Christopher Lee
UCSC Genomics Institute


On Wed, May 17, 2017 at 6:26 AM, Mike Gavrielides <mike.gav...@crick.ac.uk> wrote:

Hi,

 

Thank you for providing with the download information. I have it downloaded now, but I have a small issue.

 

I have installed the mysql database for hg19 on the genome browser instance we have, but there is no human selection available on the homepage. See screenshot attached.

 

I did exactly the same for S. cerevisiae (sacCer3) and it show on the home page and works fine.

 

Is there anything extra that needs to be done specifically for hg19?

 

Thank you in advance.

 

Kind regards,

Mike

. Mike Gavrielides

Mike Gavrielides

unread,
May 17, 2017, 4:40:20 PM5/17/17
to gen...@soe.ucsc.edu, genome...@soe.ucsc.edu, m...@soe.ucsc.edu, Chris Villarreal

Hi,

 

Thank you for providing with the download information. I have it downloaded now, but I have a small issue.

 

I have installed the mysql database for hg19 on the genome browser instance we have, but there is no human selection available on the homepage. See screenshot attached.

 

I did exactly the same for S. cerevisiae (sacCer3) and it show on the home page and works fine.

 

Is there anything extra that needs to be done specifically for hg19?

 

Thank you in advance.

 

Kind regards,

Mike

. Mike Gavrielides

Application Software Developer

Scientific Computing . AIM Team

The Francis Crick Institute

1 Midland Road

London

NW11AT

                     

www.crick.ac.uk

 

 

 

From: Chris Villarreal <cjvi...@ucsc.edu>


Date: Tuesday, 18 April 2017 at 21:06
To: Mike Gavrielides <mike.gav...@crick.ac.uk>
Cc: "gen...@soe.ucsc.edu" <gen...@soe.ucsc.edu>, "genome...@soe.ucsc.edu" <genome...@soe.ucsc.edu>, "m...@soe.ucsc.edu" <m...@soe.ucsc.edu>
Subject: Re: [genome] hd19 data download

 

Dear Mike Gavrielides,

Screen Shot 2017-05-17 at 14.24.13.png

Mike Gavrielides

unread,
May 18, 2017, 11:41:06 AM5/18/17
to Christopher Lee, gen...@soe.ucsc.edu, genome...@soe.ucsc.edu, m...@soe.ucsc.edu, Chris Villarreal

Hi Christopher,

 

Thanks a lot for your help. It was the default human genome that needed to be setup so I updated it to hg19.

 

Thanks again. I appreciate it a lot.

 

All best,

Reply all
Reply to author
Forward
0 new messages