Connection timing out when downloading hg19, mm10

[Alexander Stuy]

Hi,

  We are trying to setup a mirror of the UCSC browser.  The install went fine but when trying to download data we keep getting:

 root@genome:/usr/install# bash browserSetup.sh mirror hg19 mm10
|
| Downloading databases hg19 mm10 plus hgFixed/proteome/go from the UCSC download server
|
| Determining download file size... please wait...
| Downloading hg19 mm10 plus hgFixed proteome go hgFixed
| Required space in /var/lib/mysql: 1294.41 GB
rsync: [Receiver] failed to connect to hgdownload.soe.ucsc.edu (128.114.119.163): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(139) [Receiver=3.2.7]
| Required space in /gbdb: 0 GB
Exit error 1 occurred on line 1


It actually started downloading on one attempt of running the download command.

We have tried the rsync portion from numerous different computers, including one in NY (we are in Florida).  They all time out:

FL:
sysalex@omega:~ $ rsync -avn --stats --progress hgdownload-sd.sdsc.edu::mysql/hg19/ /users/sysalex/hg19/
rsync: [Receiver] failed to connect to hgdownload-sd.sdsc.edu (132.249.245.69): Operation timed out (60)
rsync error: error in socket IO (code 10) at clientserver.c(138) [Receiver=3.2.6]

NY:
root@NoxBox:~# rsync -avn --stats --progress hgdownload-sd.sdsc.edu::mysql/hg19/ . rsync: failed to connect to hgdownload-sd.sdsc.edu (132.249.245.69): Connection timed out (110) rsync error: error in socket IO (code 10) at clientserver.c(128) [Receiver=3.1.3]


Also we are getting a timeout in the other direction when trying to load maize genome files on the UCSC browser.

TCP non-blocking connect() to www.bio.fsu.edu IP 128.186.176.104 timed-out in select() after 10000 milliseconds. A connection timeout means that either the server is offline or its firewall, the UCSC firewall or any router between the two blocks the connection. netConnect() failed Couldn't open https://www.bio.fsu.edu/~bass/hwbtrax/mgusa/genomes.txt

We have tried opening https://www.bio.fsu.edu/~bass/hwbtrax/mgusa/genomes.txt from multiple other locations and don't see any issue, it opens or downloads ok.

Is there a network issue or some way to fix this?

Thanks,
 Alex

[Gerardo Perez]

Hello, Alex.

Could you provide us with more details, such as the OS and what kind of machine you are using?

Please include gen...@soe.ucsc.edu in any replies to ensure visibility by the team.

Gerardo Perez
UCSC Genomics Institute

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/a45df432-e83c-4940-ab0b-5101ae0e3c7c%40bio.fsu.edu.

[Maximilian Haeussler]

Hi Alex,

can you let us know where you got this script? We provide it via our genome-store.soe.ucsc.edu website or alternatively on our github repo. 

The hgdownload-sd.sdsc.edu server is not online anymore, you can only get this download address if you run the script with -a. To avoid confusion, I've removed this option now. 

thanks,

Max

You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Mirror-Specific Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome-mirro...@soe.ucsc.edu.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAH94JhUEPrG4cH8VeQvEnjFCpBE%3DGcr7pHL%2B%2B1Rc99fOamxNJA%40mail.gmail.com.

[Alexander Stuy]

Hi Max,
 
  The github version of the script is working great, thanks.  Data downloads are now consistently working and much faster.  It also updated the browser code (bash browserSetup.sh cgiUpdate).

   We got the older version from the from https://genome-store.soe.ucsc.edu by adding it to cart and then download from https://genome-store.soe.ucsc.edu/products/#pgbic.     This version is different from the github version.

    We are setting up a genome browser mirror on a Dell PowerEdge R740XD, dual Xeon Silver 4215, 256gb RAM, 8TB disk array for /var/libmysql, 30TB NFS mount for /gbdb, running Ubuntu 22.04.   It's URL will be https://genome.bio.fsu.edu, right now it's only accessible from inside FSU.  When we get done setting it up I'll open it up from all locations, hopefully mostly for the Maize genome data generated by Dr. Bass.

  Let me know if you have any questions, advice, etc.

Thanks!
  Alex

On 9/21/2023 4:17 AM, Maximilian Haeussler wrote:

Hi, 

here is the current script:

https://github.com/ucscGenomeBrowser/kent/blob/master/src/product/installer/browserSetup.sh

can you download with 

wget https://raw.githubusercontent.com/ucscGenomeBrowser/kent/master/src/product/installer/browserSetup.sh

and try again? I am surprised how you got this hgdownload-sd server, you must have been using a very very old version of the script...

best

Max

[Maximilian Haeussler]

This is great to know! Please keep up posted. We can also reference the website from ours.

Are you sure that you need the website, though? Couldn't you use an assembly hub?

[Alexander Stuy]

Hi Max,

     I got a mirror of the genome browser going using the browserSetup.sh script.   I used "bash browserSetup.sh install" and then "bash browserSetup.sh mirror hg19 hg38", etc to download the genome assembly datasets.    All seemed good.  However after running "bash browserSetup.sh update" the site started showing the following error whenever trying to load the hg19 or hg38 genomes:
     
Warning/Error(s):

• table omimAvSnp doesn't exist in hg38 database, or hFindTableInfoDb failed
• table omimGene2 doesn't exist in hg38 database, or hFindTableInfoDb failed

Do you know where I may have gone wrong and or a possible fix?   The website is https://genome.bio.fsu.edu

Thanks,

 Alex

[Maximilian Haeussler]

Yes, we're not allowed to ship the OMIM genes tracks, as per licensing restrictions. However these tracks are default tracks now, this is why the error is coming up.

If you need these tracks, let us know, OMIM has a special system in place for them. 

You can work around the problem by simply hiding these tracks. We'll send you a command very soon that does that in your mirror. 

[Maximilian Haeussler]

This is the code in the browserSetup script that should handle this, I'm looking into why it wasn't run:

DBS=hg38 # I assume you're using hg38 here

hideTracks='intronEst cons100way cons46way ucscRetroAli5 mrna omimGene2 omimAvSnp'

notSearchTables='wgEncodeGencodeBasicV19 wgEncodeGencodeCompV17 wgEncodeGencodeBasicV14 wgEncodeGencodeBasicV17 wgEncode GencodeCompV14 mgcFullMrna wgEncodeGencodeBasicV7 orfeomeMrna wgEncodeGencodePseudoGeneV14 wgEncodeGencodePseudoGeneV17 wgEncodeGencodePseudoGeneV19 wgEncodeGencodeCompV7 knownGeneOld6 geneReviews transMapAlnSplicedEst gbCdnaInfo oreganno vegaPseudoGene transMapAlnMRna ucscGenePfam qPcrPrimers transMapAlnUcscGenes transMapAlnRefSeq genscan bacEndPairs fosEndPairs'

for db in $DBS; do
       echo $db
       for track in $hideTracks; do
            mysql $db -e 'UPDATE trackDb set visibility=0 WHERE tableName="'$track'"'
        done

       for track in $notSearchTables; do
            mysql $db -e 'DELETE from hgFindSpec WHERE searchTable="'$track'"'
        done
    done

[Alexander Stuy]

Hi Max,

     The command to hide the tracks would suffice I think.    I don't think we need those tracks, I'll check with professors just in case.
     
      However in case we can add the tracks I did go ahead and fill out the OMIM data license request form at https://www.omim.org/downloads and got the Data Account Activation email in reply.

OMIM - Data Account Activation

This email is to confirm that your data account has been activated.


You can get each file you have access to using the URLs listed below:

    https://urldefense.com/v3/__https://omim.org/static/omim/data/mim2gene.txt__;!!PhOWcWs!wTTslQ7PUtZp4bQ8nFzlZ863ocrFrQ8StrKHZMs9uXpKO9qz9qsWng0ddh3Bcvo8uZA-OFdv8Ps-RZjB6cXkCh6WG1HLrEIi$ 
    https://urldefense.com/v3/__https://data.omim.org/downloads/8hzvxR02QpGvHfYpLKpKLg/mimTitles.txt__;!!PhOWcWs!wTTslQ7PUtZp4bQ8nFzlZ863ocrFrQ8StrKHZMs9uXpKO9qz9qsWng0ddh3Bcvo8uZA-OFdv8Ps-RZjB6cXkCh6WGxTugrN1$ 
    https://urldefense.com/v3/__https://data.omim.org/downloads/8hzvxR02QpGvHfYpLKpKLg/genemap2.txt__;!!PhOWcWs!wTTslQ7PUtZp4bQ8nFzlZ863ocrFrQ8StrKHZMs9uXpKO9qz9qsWng0ddh3Bcvo8uZA-OFdv8Ps-RZjB6cXkCh6WG2UB-zC9$ 
    https://urldefense.com/v3/__https://data.omim.org/downloads/8hzvxR02QpGvHfYpLKpKLg/morbidmap.txt__;!!PhOWcWs!wTTslQ7PUtZp4bQ8nFzlZ863ocrFrQ8StrKHZMs9uXpKO9qz9qsWng0ddh3Bcvo8uZA-OFdv8Ps-RZjB6cXkCh6WG3qk-f66$ 


You can view a list of all the data you have access with the following URL:

        https://urldefense.com/v3/__https://omim.org/downloads/8hzvxR02QpGvHfYpLKpKLg__;!!PhOWcWs!wTTslQ7PUtZp4bQ8nFzlZ863ocrFrQ8StrKHZMs9uXpKO9qz9qsWng0ddh3Bcvo8uZA-OFdv8Ps-RZjB6cXkCh6WG09SFY8g$ 

If you have additional OMIM data needs, please submit a detailed request at https://urldefense.com/v3/__https://omim.org/contact__;!!PhOWcWs!wTTslQ7PUtZp4bQ8nFzlZ863ocrFrQ8StrKHZMs9uXpKO9qz9qsWng0ddh3Bcvo8uZA-OFdv8Ps-RZjB6cXkCh6WG1Y5E3AD$ . Access to some OMIM data may require a license.

Please note that this data access account will expire on November 16, 2024.

     


Thanks,
Alex

[Gerardo Perez]

Hello, Alex.

We wanted to follow up and let you know that we have updated the browserSetup.sh script to hide the OMIM track from the default track set. You should no longer get the errors with this browserSetup.sh update.

You can re-download the script:

wget https://raw.githubusercontent.com/ucscGenomeBrowser/kent/master/src/product/installer/browserSetup.sh

Then run the update procedure:

bash browserSetup.sh update

Please let us know if you run into any issues.

I hope this is helpful. If you have any further questions, please reply to gen...@soe.ucsc.edu. All messages sent to that address are archived on a publicly-accessible Google Groups forum. If your question includes sensitive data, you may send it instead to genom...@soe.ucsc.edu.

Gerardo Perez
UCSC Genomics Institute

[Hank W Bass]

Thank you, Max.

 

I agree, best plan to hide the tracks w/ incoming code.

Thanks for your help.

 

How cumbersome is the process of getting permissions to display it later if we choose?

Best regards

 

Hank Bass

=============================================

Henry (Hank) W. Bass, Ph.D.

Professor of Biological Science

Florida State Univ., Tallahassee, FL, USA

=============================================

Dept. Biological Science

Current Faculty Positions | Graduate Programs | FL Outdoors

============================================= 

[Hank W Bass]

Oh, great, read this thread of emails in the wrong direction –

Nevermind may prev. email just now,

 

Glad to have the tracks show.

 

Thanks all.

 

FYI, we are setting up a maize site (already have it, genomaize.org, but not w / latest version of the ucsc browser installment). We want to use the new HiC contact maps display features.

 

 

Hank Bass

=============================================

Henry (Hank) W. Bass, Ph.D.

Professor of Biological Science

Florida State Univ., Tallahassee, FL, USA

=============================================

Dept. Biological Science

Current Faculty Positions | Graduate Programs | FL Outdoors

============================================= 

 

 

From: Gerardo Perez <gpe...@ucsc.edu>
Date: Friday, December 1, 2023 at 6:24 PM
To: Alexander Stuy <as...@bio.fsu.edu>
Cc: Hank W Bass <ba...@bio.fsu.edu>, biosci-dl-computersupport <computer...@bio.fsu.edu>, gen...@soe.ucsc.edu <gen...@soe.ucsc.edu>, genome-mirror <genome...@soe.ucsc.edu>
Subject: Re: [genome] Connection timing out when downloading hg19, mm10

[Maximilian Haeussler]

How cumbersome is the process of getting permissions to display it later if we choose?

Easy: You just make an account on OMIM.org and let them know by email that you want access to the UCSC mysql table files.  

best

Max

[Alexander Stuy]

Hi Gerardo,

    I tried to mirror hs1 today and go the following error.

          Hiding some tracks by default and removing some tracks from
searches
          hs1
          ERROR 1146 (42S02) at line 1: Table 'hs1.trackDb' doesn't exist

   The mirror commands for hg38, hg9, mm39, mm10 completed w/o error.

    Update generates the same error at end

| Hiding some tracks by default and removing some tracks from searches
go
hg18
hg19
hg38
hgFixed
hs1
ERROR 1146 (42S02) at line 1: Table 'hs1.trackDb' doesn't exist



This is transcript of the mirror command:

 root@genome:/install# bash ./browserSetup.sh mirror hs1
|
| Downloading databases hs1 plus hgFixed/proteome/go from the UCSC

download server
|
| Determining download file size... please wait...

| Downloading hs1 plus GenBank and RefSeq tables
| Required space in /var/lib/mysql: 0.000639439 GB
| Required space in /var/lib/mysql/hgFixed: 42.101 GB
| Required space in /gbdb: 629.379 GB
|
| Currently available disk space on this system:
|
| Filesystem                  Size  Used Avail Use% Mounted on
| tmpfs                        26G  2.8M   26G   1% /run
| /dev/sda2                    55G   17G   36G  32% /
| tmpfs                       126G     0  126G   0% /dev/shm
| tmpfs                       5.0M     0  5.0M   0% /run/lock
| efivarfs                    304K  234K   66K  79%
/sys/firmware/efi/efivars
| /dev/sda8                    55G  2.0M   52G   1% /tmp
| /dev/sda9                   183G   28K  173G   1% /stmp
| /dev/sda1                   975M  6.1M  968M   1% /boot/efi
| /dev/sda5                    92G  4.5G   82G   6% /var
| /dev/sda6                    11G  671M  9.7G   7% /var/log
| /dev/sda7                    55G   24M   52G   1% /home
| tank                        5.7T  128K  5.7T   1% /tank
| tank/reservation            5.7T  128K  5.7T   1% /tank/reservation
| tmpfs                        26G   80K   26G   1% /run/user/128
| 128.186.176.79:/users/gbdb   30T   14T   17T  47% /gbdb
| tmpfs                        26G   64K   26G   1% /run/user/1000
| tank/mysql                  6.9T  1.2T  5.7T  18% /var/lib/mysql
| /dev/sda3                   183G   14G  160G   9% /usr/local/apache
| /dev/sda4                   128G  4.5G  117G   4% /usr/local/apache/trash
|
| If your current disk space is not sufficient, you can press ctrl-c now and
| use a network storage server volume (e.g. NFS) or add cloud provider
storage
| (e.g. Openstack Cinder Volumes, Amazon EBS, Azure Storage)
| Please refer to the documentation of your cloud provider or storage
| system on how to add more storage to this machine.
|
| When you are done with the mount:
| Move the contents of /gbdb and /var/lib/mysql onto these volumes and
| symlink /gbdb and /var/lib/mysql to the new locations. You might have
to stop
| Mysql temporarily to do this.
| Example commands:
|     mv /gbdb /bigData/gbdb && ln -s /bigData/gbdb /gbdb
|     mv /var/lib/mysql /bigData/mysql && ln -s /bigData/mysql
/var/lib/mysql
|
| You can interrupt this script with CTRL-C now, add more space and
rerun the
| script later with the same parameters to start the download.
|
|
| Press any key to continue or CTRL-C to abort.

|
| Downloading Mysql files for mysql database hs1
receiving incremental file list
created directory /var/lib/mysql/hs1
./
hgFindSpec_justpushed.MYD
          3,216 100%    3.07MB/s    0:00:00 (xfr#1, to-chk=8/10)
hgFindSpec_justpushed.MYI
          4,096 100%    3.91MB/s    0:00:00 (xfr#2, to-chk=7/10)
hgFindSpec_justpushed.frm
          3,569 100%    3.40MB/s    0:00:00 (xfr#3, to-chk=6/10)
tableDescriptions.MYD
          2,492 100%    2.38MB/s    0:00:00 (xfr#4, to-chk=5/10)
tableDescriptions.MYI
          4,096 100%   68.97kB/s    0:00:00 (xfr#5, to-chk=4/10)
tableDescriptions.frm
          1,506 100%   25.36kB/s    0:00:00 (xfr#6, to-chk=3/10)
trackDb_justpushed.MYD
        584,912 100%    1.92MB/s    0:00:00 (xfr#7, to-chk=2/10)
trackDb_justpushed.MYI
         32,768 100%  109.97kB/s    0:00:00 (xfr#8, to-chk=1/10)
trackDb_justpushed.frm
          2,784 100%    9.31kB/s    0:00:00 (xfr#9, to-chk=0/10)

sent 202 bytes  received 640,268 bytes  426,980.00 bytes/sec
total size is 639,439  speedup is 1.00
| Downloading hgFixed tables
receiving incremental file list

sent 37 bytes  received 109 bytes  97.33 bytes/sec
total size is 57,041,318  speedup is 390,693.96
receiving incremental file list

sent 37 bytes  received 106 bytes  95.33 bytes/sec
total size is 732,946  speedup is 5,125.50
receiving incremental file list

sent 37 bytes  received 116 bytes  102.00 bytes/sec
total size is 3,466,825,074  speedup is 22,658,987.41
receiving incremental file list

sent 37 bytes  received 111 bytes  98.67 bytes/sec
total size is 1,008,806  speedup is 6,816.26
receiving incremental file list

sent 37 bytes  received 115 bytes  304.00 bytes/sec
total size is 24,055,174,021  speedup is 158,257,723.82
receiving incremental file list

sent 37 bytes  received 111 bytes  296.00 bytes/sec
total size is 113,450  speedup is 766.55
receiving incremental file list

sent 37 bytes  received 110 bytes  294.00 bytes/sec
total size is 82,528  speedup is 561.41
receiving incremental file list

sent 37 bytes  received 110 bytes  98.00 bytes/sec
total size is 1,701,281  speedup is 11,573.34
receiving incremental file list

sent 37 bytes  received 110 bytes  98.00 bytes/sec
total size is 10,306,735,369  speedup is 70,113,846.05
receiving incremental file list

sent 37 bytes  received 109 bytes  97.33 bytes/sec
total size is 1,771,444  speedup is 12,133.18
receiving incremental file list

sent 37 bytes  received 108 bytes  96.67 bytes/sec
total size is 31,449,290  speedup is 216,891.66
receiving incremental file list

sent 37 bytes  received 114 bytes  100.67 bytes/sec
total size is 481,451,164  speedup is 3,188,418.30
receiving incremental file list

sent 37 bytes  received 108 bytes  96.67 bytes/sec
total size is 306,244,534  speedup is 2,112,031.27
receiving incremental file list

sent 37 bytes  received 109 bytes  97.33 bytes/sec
total size is 4,630,966  speedup is 31,718.95
receiving incremental file list

sent 37 bytes  received 111 bytes  98.67 bytes/sec
total size is 3,120,368,306  speedup is 21,083,569.64
receiving incremental file list

sent 37 bytes  received 110 bytes  98.00 bytes/sec
total size is 21,242,618  speedup is 144,507.61
receiving incremental file list

sent 37 bytes  received 113 bytes  100.00 bytes/sec
total size is 62,516,130  speedup is 416,774.20
receiving incremental file list

sent 37 bytes  received 111 bytes  296.00 bytes/sec
total size is 73,568,441  speedup is 497,084.06
receiving incremental file list

sent 37 bytes  received 116 bytes  306.00 bytes/sec
total size is 16,107,823  speedup is 105,279.89
receiving incremental file list

sent 37 bytes  received 116 bytes  102.00 bytes/sec
total size is 69,264,624  speedup is 452,709.96
receiving incremental file list

sent 37 bytes  received 104 bytes  94.00 bytes/sec
total size is 25,998  speedup is 184.38
receiving incremental file list

sent 37 bytes  received 106 bytes  95.33 bytes/sec
total size is 21,765,802  speedup is 152,208.41
receiving incremental file list

sent 37 bytes  received 106 bytes  95.33 bytes/sec
total size is 1,152,710  speedup is 8,060.91
| Downloading hgFixed.refLink, required for all RefSeq tracks
receiving incremental file list

sent 37 bytes  received 111 bytes  98.67 bytes/sec
total size is 73,568,441  speedup is 497,084.06
| Downloading /gbdb files for assembly hs1
receiving incremental file list

sent 64 bytes  received 25,101 bytes  16,776.67 bytes/sec
total size is 629,378,999,937  speedup is 25,010,093.38
| Checking all mysql tables after the download to make sure that they
are closed
hgFixed.encodeExp
Warning  : Triggers for table `hgFixed`.`encodeExp` have no creation context
| Hiding some tracks by default and removing some tracks from searches
hs1
ERROR 1146 (42S02) at line 1: Table 'hs1.trackDb' doesn't exist

Exit error 1 occurred on line 1

The UCSC Genome Browser installation script exited with an error.
Please contact us at genome...@soe.ucsc.edu and send us an output log
of the command prefixed with "bash -x", e.g.
bash -x browserSetup.sh install 2>&1 > install.log



Do I need to add a table to hs1 or such?

Thanks,
 Alex

[Jairo Navarro Gonzalez]

Hello,

Thank you for using the UCSC Genome Browser and sending your inquiry.

The hs1 assembly is a curated assembly hub, so the data is stored as a hub instead of inside a MariaDB table. Unfortunately, this means that we need to add special code to the browserSetup.sh script to download the necessary files when using the mirror command, which will take a bit of time to implement and test.

Until then, you can download the hs1 data hub files from the hgdownload server:

https://hgdownload.soe.ucsc.edu/goldenPath/hs1/
https://hgdownload.soe.ucsc.edu/gbdb/hs1/

The hub.txt file for the hs1 assembly can be found here:

https://hgdownload.soe.ucsc.edu/gbdb/hs1/hubs/public/hub.txt

Thank you again for bringing this to our attention, and we appreciate your patience while we add hs1 support to the mirror command.

I hope this is helpful. If you have any further questions, please reply to gen...@soe.ucsc.edu.

All messages sent to that address are archived on a publicly accessible Google Groups forum.

If your question includes sensitive data, you may send it instead to genom...@soe.ucsc.edu.

Jairo Navarro
UCSC Genome Browser

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Mirror-Specific Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome-mirro...@soe.ucsc.edu.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/9dbd9da2-3229-49e3-be11-531f9a590b96%40bio.fsu.edu.