Derby 10.5 or newer

[Egon Willighagen]

Hi everyone,

BridgeDb 3.0 has been holding up pretty well. It comes with some API changes [0], initiated during the Open PHACTS projects (so, there has been plenty of time to catch up (plenty=5 years)). Some old method that resulted in undefined behaviour have been removed (which some of you already noticed).

However, there is a second hurdle: Derby.

Currently, we're using Derby 10.5 but that release is very outdated. The problem is that Derby files created with a newer Derby version do not read with read with 10.5 and the other way around. This has been biting me for the past 2-3 years already, but we need to face the fact: practically they are not compatible, so that we will have a transition phase where we have two versions of ID mapping databases: one compatible with 10.5 (11 years old) and one with 10.14 or 10.15.

Tools not affected or not strongly:

- the webservice: affected, but only the server side. so, this needs administration but doable

- webservice clients: they don't see anything of these complexities

Affected are basically any tool that needs to read the Derby files:

- PathVisio

- GPMLRDF

- BridgeDbR

- all scripts creating Derby files

All tools will need to communicate carefully with which Derby version they are compatible and/or are using.

If you wonder why this upgrade has not been made earlier, is compatibility with downstream tools and some of them hard to upgrade. For example, PathVisio and GPMLRDF are not full mavenized. And even if they are, they still need to hardcode which Derby version it needs, because otherwise it will pick a newer version which gives trouble.

My plans:

1. update https://bridgedb.github.io/data/gene_database/ to indicate the Derby version

2. make de Derby version part of the metadata

Now, one issue is that I am not sure if we can detect if a Derby file is for the wrong Derby version. That needs exploring.

The Derby version issues have been giving me headaches for some time now, so I expect a rough ride.

Egon

0.https://github.com/bridgedb/BridgeDb/blob/master/UPGRADE_NOTES.md

--

Have you heard about Wikidata already? "Use Scholia and Wikidata to find scientific literature" is a new tutorial from my colleague Lauren Dupuis. https://laurendupuis.github.io/Scholia_tutorial/

-----

E.L. Willighagen
Department of Bioinformatics - BiGCaT
Maastricht University (http://www.bigcat.unimaas.nl/)
Homepage: http://egonw.github.com/
Blog: http://chem-bla-ics.blogspot.com/
PubList: https://www.zotero.org/egonw
ORCID: 0000-0001-7542-0286
ImpactStory: https://impactstory.org/u/egonwillighagen

[Evelo, Chris (BIGCAT)]

Hi Egon,

 

I think I understand most of the problem and solutions you propose. And yes sounds like a good plan.

 

What you first want to fix is a way to figure out what version a derby file is actually for, so we can actually run multiple versions of databases with different versions of tools. Do I understand correctly that your problem there might be that a you might not even be able to get that info from the file?

 

If so other solutions I could think of:

1. Just let it break and catch the error. If you get an “open database error” just recover from that and give an error message “opening failed, possibly wrong Derby version”.
2. Use the file metadata systems on the OS to store this info. Two problems here: 1) I have no clue whether that is even possible. 2) It is probably very OS dependent.
3. Create a new naming system for the datafiles themselves so the filename actually shows the version. Both the user and the code could use that. Of course that is really primitive, but we do not need a solution that will work for even anyway. We do want to update all these places that use the old code.

 

Lucas, I am not sure you were actually on this email list (the bridgedb-discuss one) if not please add yourself.

 

                Best, Chris

[Egon Willighagen]

On Sun, Nov 22, 2020 at 5:39 PM Evelo, Chris (BIGCAT) <chris...@maastrichtuniversity.nl> wrote:

I think I understand most of the problem and solutions you propose. And yes sounds like a good plan.

 

What you first want to fix is a way to figure out what version a derby file is actually for, so we can actually run multiple versions of databases with different versions of tools.

Yes, that would be ideal, but practically really hard. You cannot have two versions of the library Derby on the classpath, and we would have to go double OSGi on all downstream tools. That is PathVisio would have to have *all* BridgeDb functionality as a plugin (is that even possible?) and also not sure R supports this. 

 

Do I understand correctly that your problem there might be that a you might not even be able to get that info from the file?

Correct, because you need the right Derby version to even figure that out. 

 

If so other solutions I could think of:

1. Just let it break and catch the error. If you get an “open database error” just recover from that and give an error message “opening failed, possibly wrong Derby version”.

Yes, I hope I can do something like that. But the error message is so generic, that 

> 2. Use the file metadata systems on the OS to store this info. Two problems here: 1) I have no clue whether that is even possible. 2) It is probably very OS dependent.

And it would need to get saved when people download the file...

> 3. Create a new naming system for the datafiles themselves so the filename actually shows the version. Both the user and the code could use that. Of course that is really

> primitive, but we do not need a solution that will work for even anyway.

Yes, I was thinking about something like that too. It could perhaps use that info when the file failed to load.

> We do want to update all these places that use the old code.

Yes. 

Egon