Missing tables after UCSC Genome Browser Upgrade

[McCully, Dwayne (NIH/NIAMS) [C]]

 

Hello Support,

 

Just completed an upgrade using the browserSetup.sh script and during testing I got the following error.

I have confirmed the tables are not in the database hg38 or hg19.

 

Dwayne

 

Dwayne McCully

NIAMS, NIH

Contractor

 

 

[Robert Kuhn]

Hello, Dwayne,

I am no longer on the Browser staff, but I still lurk on the mailing list as bit.  I can tell

you OMIM does not permit any third parties to download their data.  You need to go to

omim.org and download them there.

best wishes,

     --b0b kuhn

Robert Kuhn, PhD

Associate Director UCSC Genome Browser (retired)

dba Robert Kuhn Consulting -- bioinformatics training and consulting

On Thu, Nov 30, 2023 at 9:18 AM 'McCully, Dwayne (NIH/NIAMS) [C]' via UCSC Genome Browser Public Support <gen...@soe.ucsc.edu> wrote:

Hi Support,

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/DM8PR09MB61042F72CED3FEF324813B3FE782A%40DM8PR09MB6104.namprd09.prod.outlook.com.

[Robert Kuhn]

Hello, Dwayne,

15 years ago we set up a system where we uploaded MySQL-formatted tables back to OMIM for mirrors to download from them. I believe they now do that themselves.

They require reqistration via:

https://omim.org/downloads

Please contact them there.  

I will let the Browser staff address the question about the script. Is it possible you are updating an installation that loaded the OMIM tables separately? 

Best regards,

   ..b0b

On Thu, Nov 30, 2023, 5:06 PM McCully, Dwayne (NIH/NIAMS) [C] <dmcc...@mail.nih.gov> wrote:

Thank you Robert for answering my e-mail!!

 

The UCSC Genome Browser group allowed an upgrade that required mirror managers to download

data from OMIM?   The error is indicating the data needs to be in a MariaDB table format for multiple genomes.   Will the omim.org site have this?

Will the message that your seeing in the below screen shot will affect the operations of the mirror site until fixed?

 

Sorry for all the questions but need to get the problem resolved.

 

Dwayne

 

Dwayne McCully

NIAMS, NIH

Contractor

 

From: Robert Kuhn <ku...@soe.ucsc.edu>

Sent: Thursday, November 30, 2023 7:38 PM
To: McCully, Dwayne (NIH/NIAMS) [C] <dmcc...@mail.nih.gov>
Cc: gen...@soe.ucsc.edu; genome-www <genom...@soe.ucsc.edu>; Mullacheri, Anilkumar (NIH/NIAMS) [C] <anilkumar....@nih.gov>; Winchester, Christine (NIH/NIAMS) [E] <christine....@nih.gov>; genome...@soe.ucsc.edu; Robert Kuhn <ku...@soe.ucsc.edu>
Subject: [EXTERNAL] Re: [genome] Missing tables after UCSC Genome Browser Upgrade

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.

 

[McCully, Dwayne (NIH/NIAMS) [C]]

Thank you Robert for answering my e-mail!!

 

The UCSC Genome Browser group allowed an upgrade that required mirror managers to download

data from OMIM?   The error is indicating the data needs to be in a MariaDB table format for multiple genomes.   Will the omim.org site have this?

Will the message that your seeing in the below screen shot will affect the operations of the mirror site until fixed?

 

Sorry for all the questions but need to get the problem resolved.

 

Dwayne

 

Dwayne McCully

NIAMS, NIH

Contractor

 

From: Robert Kuhn <ku...@soe.ucsc.edu>

Sent: Thursday, November 30, 2023 7:38 PM
To: McCully, Dwayne (NIH/NIAMS) [C] <dmcc...@mail.nih.gov>
Cc: gen...@soe.ucsc.edu; genome-www <genom...@soe.ucsc.edu>; Mullacheri, Anilkumar (NIH/NIAMS) [C] <anilkumar....@nih.gov>; Winchester, Christine (NIH/NIAMS) [E] <christine....@nih.gov>; genome...@soe.ucsc.edu; Robert Kuhn <ku...@soe.ucsc.edu>
Subject: [EXTERNAL] Re: [genome] Missing tables after UCSC Genome Browser Upgrade

 

Hello, Dwayne,

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and are confident the content is safe.

 

[McCully, Dwayne (NIH/NIAMS) [C]]

Thank you Max for the update and new instructions!

 

After running the new browserSetup.sh it ran smoothly up to this point.

 

---------------------------------------------------------------------------------------------------------------------

sent 24 bytes  received 1,798 bytes  1,214.67 bytes/sec

total size is 46,015,696,612  speedup is 25,255,596.38

| Hiding some tracks by default and removing some tracks from searches

go

ERROR 1146 (42S02) at line 1: Table 'go.trackDb' doesn't exist

Exit error 1 occurred on line 1

The UCSC Genome Browser installation script exited with an error.

Please contact us at genome...@soe.ucsc.edu and send us an output log

of the command prefixed with "bash -x", e.g.

bash -x browserSetup.sh install 2>&1 > install.log

-------------------------------------------------------------------------------------------------------------------------

 

What would be my next step to fix?

 

Dwayne

 

 

From: Maximilian Haeussler <mhae...@ucsc.edu>
Sent: Friday, December 1, 2023 5:26 AM
To: McCully, Dwayne (NIH/NIAMS) [C] <dmcc...@mail.nih.gov>

Cc: Robert Kuhn <ku...@soe.ucsc.edu>; gen...@soe.ucsc.edu; genome-www <genom...@soe.ucsc.edu>; Mullacheri, Anilkumar (NIH/NIAMS) [C] <anilkumar....@nih.gov>; Winchester, Christine (NIH/NIAMS) [E] <christine....@nih.gov>; genome...@soe.ucsc.edu
Subject: Re: [genome-www] RE: [EXTERNAL] Re: [genome] Missing tables after UCSC Genome Browser Upgrade

 

Hi Dwayne, 

 

OMIM tracks are not available on mirrors, you'll need to reach out to OMIM directly, they have a mysql dump in exactly the format for your UCSC mirror and can activate this special download once you have an OMIM username and contact them with a request for the UCSC sql files.

 

But this error is our fault, we made OMIM a default track without thinking of mirrors that don't have this track yet.

 

I now changed the update script to hide the OMIM track from the default track set, so this error shouldn't appear anymore.

Can you re-download the script and then run the update procedure again?

 

The commands should be something like the following:

 

wget https://raw.githubusercontent.com/ucscGenomeBrowser/kent/master/src/product/installer/browserSetup.sh

bash browserSetup.sh update

 

This should not only upgrade your CGIs but also hide the OMIM tracks so this error message won't appear anymore.

 

If this does not work or you have other questions, please do not hesitate to reach out to us. 

I agree that this error is a problem for mirrors and we will get your mirror working again ASAP.

 

best

Max

 

[McCully, Dwayne (NIH/NIAMS) [C]]

Hi Robert,

 

We did not load the OMIM tables separately as this is the first time I’ve seen anything related to OMIM.

I will ask our federal staff about the registration with OMIM but this will take time due to the review of licensing requirements.

 

At the moment, Max is trying to resolve the OMIM issue short term with a new browserSetup.sh script.    

 

Dwayne  

[Maximilian Haeussler]

Hi Dwayne,

oh, you have a full set of databases, including "go", sorry I didn't test with a full mirror. I changed the script now to handle this case. You can re-download and I hope it's working now.

wget https://raw.githubusercontent.com/ucscGenomeBrowser/kent/master/src/product/installer/browserSetup.sh
bash browserSetup.sh update

Otherwise you can also run the SQL commands that fix this problem yourself as a temporary solution until the next update:

mysql hg38 -e 'UPDATE trackDb set visibility=0 WHERE tableName="omimGene2"
mysql hg38 -e 'UPDATE trackDb set visibility=0 WHERE tableName="omimAvSnp"

Thanks for this feedback, I'm realizing that we never completely tested all the options of this script and so far this hadn't been a problem. This is the first time that we have OMIM as the default track.

best

Max

[McCully, Dwayne (NIH/NIAMS) [C]]

Hi Max,

 

Downloaded and ran the updated script but it stopped with another issue.

 

 

sent 24 bytes  received 1,798 bytes  1,214.67 bytes/sec

total size is 46,015,696,612  speedup is 25,255,596.38

| Hiding some tracks by default and removing some tracks from searches

go

hg18

hg19

hg38

hgFixed

ERROR 1146 (42S02) at line 1: Table 'hgFixed.trackDb' doesn't exist

Exit error 1 occurred on line 1

The UCSC Genome Browser installation script exited with an error.

Please contact us at genome...@soe.ucsc.edu and send us an output log

of the command prefixed with "bash -x", e.g.

bash -x browserSetup.sh install 2>&1 > install.log

[root@niamsapp01 ~]#

 

 

What would you like me to do?

[McCully, Dwayne (NIH/NIAMS) [C]]

Hi Max,

 

One good thing is the error/warning message has been resolved with the new browserSetup.sh script. 😊

This missing table 'hgFixed.trackDb' is a new issue. ☹

 

Dwayne

[Maximilian Haeussler]

Ah. hgFixed. Sorry. Fixed that in the script now. If your problem is resolved now, you can re-download the script in around 10-15 minutes and I think we're done with this problem.

Thanks! And sorry that this took a few iterations!

[McCully, Dwayne (NIH/NIAMS) [C]]

Hi Max,

 

Another issue came up with the new script.

 

go

hg18

hg19

hg38

hgFixed

mm10

mm9

proteome

ERROR 1146 (42S02) at line 1: Table 'proteome.trackDb' doesn't exist

Exit error 1 occurred on line 1

The UCSC Genome Browser installation script exited with an error.

Please contact us at genome...@soe.ucsc.edu and send us an output log

of the command prefixed with "bash -x", e.g.

bash -x browserSetup.sh install 2>&1 > install.log

 

Dwayne

[McCully, Dwayne (NIH/NIAMS) [C]]

Hi Max,

 

Did you get my last e-mail issue?

 

Dwayne

 

From: McCully, Dwayne (NIH/NIAMS) [C]
Sent: Friday, December 1, 2023 11:43 AM
To: Maximilian Haeussler <mhae...@ucsc.edu>

[Christopher Lee]

Hi Dwayne,

Can you re-download the script in 10-15 minutes? We should have fixed this now.

Thanks,

You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Mirror-Specific Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome-mirro...@soe.ucsc.edu.
To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/DM8PR09MB6104E33B17052B9FE39F6382E781A%40DM8PR09MB6104.namprd09.prod.outlook.com.

[Christopher Lee]

Hi Dwayne,

Yes I think you're done. I have no clue why you are receiving that message. You'd have to follow the instructions to run with -x so we can see what the real error was and the command that generated the error.

On Fri, Dec 1, 2023 at 2:15 PM McCully, Dwayne (NIH/NIAMS) [C] <dmcc...@mail.nih.gov> wrote:

Hi Chris,

 

I downloaded the latest browerUpdate.sh script.  Here is the output from the run but mentions line 2070.

Are we finished?

 

----------------------------------------------------------------------------------------------------------------------------------

| Hiding some tracks by default and removing some tracks from searches

go

hg18

hg19

hg38

hgFixed

mm10

mm9

proteome

sacCer3

uniProt

| update finished

browserSetup.sh: line 2070: receiving: command not found

| Error: the UCSC Genome Browser installation script failed with an error

| You can run it again with "bash -x browserSetup.sh" to see what failed.

| You can then send us an email with the error message.

 

 

Dwayne

To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAC7r7-K0cHNg3HGvpQp7QzZ4GTWD61L_tErMJTgHEOdh5M_9dQ%40mail.gmail.com.

[Christopher Lee]

No, these changes were to remove the OMIM tables so that your genome browser can work without them

On Fri, Dec 1, 2023 at 2:25 PM McCully, Dwayne (NIH/NIAMS) [C] <dmcc...@mail.nih.gov> wrote:

Hi Chris,

 

Will try with the -x option over the weekend and report back what I find.

Thanks for you and Max’s support in fixing my issue.   Do we still need to get the OMIM database tables installed?

Are they essential for using the Browser and how?  I can get the registration to OMIM completed and get the downloads.

 

Dwayne

[McCully, Dwayne (NIH/NIAMS) [C]]

Hi Chris,

 

I downloaded the latest browerUpdate.sh script.  Here is the output from the run but mentions line 2070.

Are we finished?

 

----------------------------------------------------------------------------------------------------------------------------------

| Hiding some tracks by default and removing some tracks from searches

go

hg18

hg19

hg38

hgFixed

mm10

mm9

proteome

sacCer3

uniProt

| update finished

browserSetup.sh: line 2070: receiving: command not found

| Error: the UCSC Genome Browser installation script failed with an error

| You can run it again with "bash -x browserSetup.sh" to see what failed.

| You can then send us an email with the error message.

 

 

Dwayne

 

To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAC7r7-K0cHNg3HGvpQp7QzZ4GTWD61L_tErMJTgHEOdh5M_9dQ%40mail.gmail.com.

[McCully, Dwayne (NIH/NIAMS) [C]]

Hi Chris,

 

Will try with the -x option over the weekend and report back what I find.

Thanks for you and Max’s support in fixing my issue.   Do we still need to get the OMIM database tables installed?

Are they essential for using the Browser and how?  I can get the registration to OMIM completed and get the downloads.

 

Dwayne

 

[McCully, Dwayne (NIH/NIAMS) [C]]

Hi Chris,

 

Below is the error

 

----------------------------------------

 

| update finished

+ receiving incremental file list

browserSetup.sh: line 2070: receiving: command not found

++ errorHandler

++ startMysql

 

 

 

Dwayne

[Luis Nassar]

Hi, Dwayne.

It looks like something may be wrong with your browserSetup.sh script.

Could you send us the first 10 lines and the last 10 lines of your script?

Once we are able to have it update properly the issue should be fixed.

Lou Nassar
UCSC Genomics Institute

To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/DM8PR09MB6104BB8FB6CDC32F57810028E780A%40DM8PR09MB6104.namprd09.prod.outlook.com.

[McCully, Dwayne (NIH/NIAMS) [C]]

Hi Nassar,

 

See requested lines below.

 

Dwayne

 

 

# head -n 10 browserSetup.sh

#!/bin/bash

 

# script to install/setup dependencies for the UCSC genome browser CGIs

# call it like this as root from a command line: bash browserSetup.sh

 

# you can easily debug this script with 'bash -x browserSetup.sh', it

# will show all commands then

 

exec > >(tee -a "${HOME}/browserSetup.sh") 2>&1

 

# tail -n 10 browserSetup.sh

++ echo '|' You can run it again with '"bash -x browserSetup.sh"' to see what failed.

| You can run it again with "bash -x browserSetup.sh" to see what failed.

++ echo2 You can then send us an email with the error message.

++ command echo '|' You can then send us an email with the error message.

++ echo '|' You can then send us an email with the error message.

| You can then send us an email with the error message.

++ exit 0

+ exitHandler 0 1

+ '[' 0 == 100 -o 0 == 0 ']'

+ exit 0

#

[Luis Nassar]

Hi, Dwayne.

It looks like your browserSetup.sh script is in a strange state.

All of the "+" characters indicate perhaps it was overwritten when you ran it with the "-x" flag?

Could you rename or move your copy (just in case) and download a new one (https://raw.githubusercontent.com/ucscGenomeBrowser/kent/master/src/product/installer/browserSetup.sh) and run another update?

The original bug you wrote about should be fixed, where it complained about the missing table. That complaining track has now been set to hide.

I hope this is helpful. Please include gen...@soe.ucsc.edu in any replies to ensure visibility by the team. All messages sent to that address are archived on our public forum. If your question includes sensitive information, you may send it instead to genom...@soe.ucsc.edu.

Lou Nassar
UCSC Genomics Institute

[McCully, Dwayne (NIH/NIAMS) [C]]

Hi Luis,

 

After downloading the new version as you suggested I got the same error when running.

 

Dwayne

 

 

 

To view this discussion on the web visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAEa0ENpTnsTQU9%2Bo995ZTYYrtRaav7cXF4ZUG_ru-mO4JtwzoA%40mail.gmail.com.

[Maximilian Haeussler]

Hi Dwayne, 

uh oh, you and Lou were right, there is something odd there, a complete copy of the help text copied to the end of the file. This must have slipped into a recent edit. 

You're fine now, the script did complete, it just ran into this help text at the end and stopped with an error, other than this message, your mirror should have been installed fine. 

I've removed the copy of the help text now, if you redownload the script from Github again, the error should be gone.

Many thanks for this second email!

best

Max