Glitch with jksql.c

[Platts, Adrian]

Hi all,

Last year you helped debug an issue in which the mysql/mariaDb instance installed by GBIC in which it didn’t want to relocate the

datadir to a remote drive.  

I think as a result 95% of things have been working well.  But I suspect I still have an install issue somewhere as there are a few key utilities

that I have to either work around or that don’t work at all.

For example:

With the most current version of hgTrackDb I get Seg Faults in all databases

# /home/apps/hgTrackDb  . Potato trackDb /home/gbdb/trackDb.sql .

Loaded 21 track descriptions total

Segmentation fault (core dumped)

but the workaround here is that the August 2018 version works just fine

# ../../rumex/track_files/hgTrackDb . Potato trackDb /home/gbdb/trackDb.sql .

Loaded 21 track descriptions total

Loaded database Potato

More problematically (as I can’t find a workaround other than bedtools) featureBits also Seg Faults immediately:

# /home/apps/featureBits Potato AR3.gtp96.bed

Segmentation fault (core dumped)

I have checked that the readonly readwrite and ctdbuser accounts can all access the Potato database and select * from the chromInfo table.

Dropping into gbdb I see us dying in the mysql_options call in mysql.h:

(gdb) bt

#0  0x0000555555762600 in mysql_options ()

#1  0x00005555556379e1 in sqlConnRemoteFillIn (sc=0x555556482a40, sp=sp@entry=0x5555564818a0, database=database@entry=0x7fffffffe58b "Potato", abort=abort@entry=1, addAsOpen=addAsOpen@entry=1) at jksql.c:1144

#2  0x0000555555637d7f in sqlConnRemote (sp=sp@entry=0x5555564818a0, database=database@entry=0x7fffffffe58b "Potato", abort=abort@entry=1) at jksql.c:1231

#3  0x0000555555637f9d in sqlConnProfile (sp=0x5555564818a0, database=database@entry=0x7fffffffe58b "Potato", abort=abort@entry=1) at jksql.c:1311

#4  0x00005555556381b7 in sqlConnect (database=database@entry=0x7fffffffe58b "Potato") at jksql.c:1387

#5  0x0000555555605c55 in fbCreateChromInfoList (name=0x555555c04e36 "all", database=database@entry=0x7fffffffe58b "Potato") at featureBits.c:145

#6  0x0000555555607898 in featureBits (database=0x7fffffffe58b "Potato", tableCount=1, tables=tables@entry=0x7fffffffe2d8) at featureBits.c:910

#7  0x0000555555607d51 in main (argc=<optimized out>, argv=0x7fffffffe2c8) at featureBits.c:1037

but stepping through the code, it has no problems with the call to mysql options at line 1125.  Only when it calls the function again at line 1144 does it die

sqlConnRemoteFillIn (sc=0x555556482a40, sp=sp@entry=0x5555564818a0, database=database@entry=0x7fffffffe58b "Potato", abort=abort@entry=1, addAsOpen=addAsOpen@entry=1) at jksql.c:1121

1121 in jksql.c 

1125 in jksql.c. << also calls mysql_options here: mysql_options(conn, MYSQL_OPT_LOCAL_INFILE, NULL);

1130 in jksql.c

1144 in jksql.c 

Program received signal SIGSEGV, Segmentation fault.

0x0000555555762600 in mysql_options ()

The code here being:

else   // TURN VERIFICATION OFF

    {

    #if !defined(MARIADB_BASE_VERSION) && defined(MYSQL_VERSION_ID) && (MYSQL_VERSION_ID >= 80000) // OVER-RIDE DEFAULT COMPILED IN.

    mysql_options(conn, MYSQL_OPT_SSL_MODE, SSL_MODE_PREFERRED);   << line 1144, featureBits dies here I think

    #else

    my_bool flag = FALSE;

    mysql_options(conn, MYSQL_OPT_SSL_VERIFY_SERVER_CERT, &flag);

    #endif

    }

now if I compare this with hgLoadBed that also of course has to call sqlConnRemoteFillIn, but doesn’t die, the major difference is that it

doesn’t execute line 1144 but executes line 1146 instead (which is curious given that they’re accessing the same mysql instance so they should have the same version info)

Breakpoint 1, sqlConnRemoteFillIn (sc=0x137d860, sp=sp@entry=0x137c6c0, database=database@entry=0x7fffffffe573 "Potato", abort=abort@entry=1, addAsOpen=addAsOpen@entry=1) at jksql.c:1106

(gdb) bt

#0  sqlConnRemoteFillIn (sc=0x137d860, sp=sp@entry=0x137c6c0, database=database@entry=0x7fffffffe573 "Potato", abort=abort@entry=1, addAsOpen=addAsOpen@entry=1) at jksql.c:1106

#1  0x000000000043b833 in sqlConnRemote (sp=sp@entry=0x137c6c0, database=database@entry=0x7fffffffe573 "Potato", abort=abort@entry=1) at jksql.c:1231

#2  0x000000000043ba05 in sqlConnProfile (sp=0x137c6c0, database=database@entry=0x7fffffffe573 "Potato", abort=abort@entry=1) at jksql.c:1311

#3  0x000000000043bbf6 in sqlConnect (database=database@entry=0x7fffffffe573 "Potato") at jksql.c:1387

#4  0x0000000000411d4d in loadDatabase (database=database@entry=0x7fffffffe573 "Potato", track=track@entry=0x7fffffffe57a "test", bedSize=bedSize@entry=3, bedList=0x1381820) at hgLoadBed.c:542

#5  0x0000000000412814 in hgLoadBed (database=0x7fffffffe573 "Potato", track=0x7fffffffe57a "test", bedCount=<optimized out>, bedFiles=bedFiles@entry=0x7fffffffe2c0) at hgLoadBed.c:770

#6  0x0000000000412bcf in main (argc=<optimized out>, argv=0x7fffffffe2a8) at hgLoadBed.c:868

(gdb) b jksql.c:1125

Breakpoint 2 at 0x43b47d: file jksql.c, line 1125.

(gdb) c

Continuing.

Breakpoint 2, sqlConnRemoteFillIn (sc=0x137d860, sp=sp@entry=0x137c6c0, database=database@entry=0x7fffffffe573 "Potato", abort=abort@entry=1, addAsOpen=addAsOpen@entry=1) at jksql.c:1125

1125 in jksql.c

1130 in jksql.c

1146 in jksql.c

(gdb) q

So is it something about the way its calling SSL specifically in some but not all utilities??

else   // TURN VERIFICATION OFF

    {

    #if !defined(MARIADB_BASE_VERSION) && defined(MYSQL_VERSION_ID) && (MYSQL_VERSION_ID >= 80000) // OVER-RIDE DEFAULT COMPILED IN.

    mysql_options(conn, MYSQL_OPT_SSL_MODE, SSL_MODE_PREFERRED);   

    #else

    my_bool flag = FALSE;

    mysql_options(conn, MYSQL_OPT_SSL_VERIFY_SERVER_CERT, &flag);    << hgLoadBed skips to here instead and runs fine.

    #endif

    }

This is all kind of dependent on my version of featureBits/jksql matching the codebase in github, but it looks like it does.

Would be very glad to get ideas of where to go next. 

Adrian

Adrian Platts

MSU

[Platts, Adrian]

Please ignore my rambling question.  

The solution was simply to compile all the utils again from source rather than using the precompiled binaries.  

--

---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Mirror-Specific Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome-mirro...@soe.ucsc.edu.
To view this discussion visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/08FDE398-487A-41E1-B576-52610E608504%40msu.edu.

[Maximilian Haeussler]

Hi Adrian,

This is concerning, thanks for this detailed report and all the debugging. The browser binaries should not crash on local installs, they should either provide an error message or - ideally - just work. 

 

I have a few questions:

1) you are using mariadb in your local installation of the genome browser?

2) do you know which linux distro / mariadb version you're using?

On a side note, I see that you loaded your own genome into the genome browser. Can you tell us why you think that's necessary? You could have made your own assembly hub "Potato" and load such a hub into your own genome browser, which would have meant that you don't need to make changes to the mysql databases.

best

Max 

To view this discussion visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/C6713174-E2A1-4F8A-A685-CE31E3A975AC%40msu.edu.

[Platts, Adrian]

Hi Max

Thanks for the reply.  Per my follow-up note, I felt really dumb for not spotting an easy fix. 

So this only reflects negatively on me as the person who didn’t set up the browser terribly well, not on the compiled binaries.

1.   Maybe.  As we discussed last year the mysql (mariadb) instance I have absolutely would not start as a service when the data directory was not in /var/lib (even as a soft link) however it 

would start without any issues when I just started the daemon from the command line, and per the mysql-mariadb command line compatibility this did indeed indicate MariaDb was starting:

# mysqld --datadir=/home/mysql --user mysql &

2025-06-10 10:59:52 0 [Note] Starting MariaDB 10.11.13-MariaDB-0ubuntu0.24.04.1 source revision 8fb09426b98583916ccfd4f8c49741adc115bac3 server_uid ***************** as process 129305

2025-06-10 10:59:52 0 [Note] InnoDB: Compressed tables use zlib 1.3

2025-06-10 10:59:52 0 [Note] InnoDB: Number of transaction pools: 1

2025-06-10 10:59:52 0 [Note] InnoDB: Using AVX512 instructions

2025-06-10 10:59:52 0 [Note] InnoDB: Using liburing

2025-06-10 10:59:52 0 [Note] InnoDB: innodb_buffer_pool_size_max=128m, innodb_buffer_pool_size=128m

2025-06-10 10:59:52 0 [Note] InnoDB: Completed initialization of buffer pool

2025-06-10 10:59:52 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes)

2025-06-10 10:59:52 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=22647741685

2025-06-10 10:59:52 0 [Note] InnoDB: End of log at LSN=22647814085

2025-06-10 10:59:53 0 [Note] InnoDB: To recover: 89 pages

2025-06-10 10:59:53 0 [Note] InnoDB: 128 rollback segments are active.

2025-06-10 10:59:53 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"

2025-06-10 10:59:53 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...

2025-06-10 10:59:53 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.

2025-06-10 10:59:53 0 [Note] InnoDB: log sequence number 22647814085; transaction id 89843

2025-06-10 10:59:53 0 [Note] Plugin 'FEEDBACK' is disabled.

2025-06-10 10:59:53 0 [Note] InnoDB: Loading buffer pool(s) from /home/mysql/ib_buffer_pool

2025-06-10 10:59:53 0 [Warning] 'default-authentication-plugin' is MySQL 5.6 / 5.7 compatible option. To be implemented in later versions.

2025-06-10 10:59:53 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.

2025-06-10 10:59:53 0 [Note] Server socket created on IP: '127.0.0.1'.

2025-06-10 10:59:53 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1932: Table 'mysql.gtid_slave_pos' doesn't exist in engine

2025-06-10 10:59:53 0 [Note] mysqld: ready for connections.

Version: '10.11.13-MariaDB-0ubuntu0.24.04.1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Ubuntu 24.04

2025-06-10 10:59:55 0 [Note] InnoDB: Buffer pool(s) load completed at 250610 10:59:55

2. The Db is whatever was installed by the genome browser GBIC installer I think (see above for mariadb).

Linux ubuntu 6.8.0-60-generic #63-Ubuntu SMP PREEMPT_DYNAMIC

# cat /etc/os-release

PRETTY_NAME="Ubuntu 24.04.2 LTS"

NAME="Ubuntu"

VERSION_ID="24.04"

VERSION="24.04.2 LTS (Noble Numbat)"

VERSION_CODENAME=noble

ID=ubuntu

ID_LIKE=debian

HOME_URL="https://www.ubuntu.com/"

SUPPORT_URL="https://help.ubuntu.com/"

BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"

PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"

UBUNTU_CODENAME=noble

LOGO=ubuntu-logo

[3] I did use assembly hubs for a while, I think I stopped because there were things I knew how to do pretty easily in the original system that I was less sure about for assembly hubs.

For example, working with comparative genomics, I like to link locations between all our diploid and polyploid genome databases using chain files.  But I was perhaps 

less clear on how to link assembly hubs together through chain files.  Also if something is not working as I expect I can just go into the database and look to see what is happening in

the tables … even if I have to manually write a few rows as in the trackDb before I found the workaround.

All the best

Adrian

To view this discussion visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome-mirror/CAHm%3D4fwtEhfOpOpmLY4TAheo5Dn28XBq9dzJQPpFOO061uKtbg%40mail.gmail.com.