Crash & Data Recovery

[Nicolas Moya]

Hello,

I have been using Apollo for the past couple of months seamlessly, but today it stopped working.

I am running Apollo on an m4.large ECE2 AWS server, using the community AMI (Version 2.6.2.).

I am Java illiterate, so I'm unable to diagnose the error fully, but it seems there are memory issues occurring. I attached the most recent catalina.out and localhost logs. Needless to say, I am unable to log in, and back up my GFF3.

In the case the error is fatal, I was wondering if it was possible to extract the most recent GFF3 from the command line. I haven't backed up in a week, and we have curated over 1000 genes since then. It would be a shame if we had to re-do all that work. I am willing to open up a new instance to hopefully bypass the error, but I am very confused as to how I can migrate the current user created annotations.

Please let me know how to proceed.

Best,

--

Nicolas Moya, B.S.

Doctoral Student

Department of Molecular Biosciences

Interdisciplinary Biological Sciences Program

Northwestern University

[Nathan Dunn]

Its definitely a memory problem.  You can fix that here (and more resources online):

https://genomearchitect.readthedocs.io/en/latest/Troubleshooting.html?highlight=memory#tomcat-memory

You’re m4 large should be fine, even with a max of 2 GBs of memory.

The good news is that what is in the database should be fine. 

I would:

(1) stop tomcat

(2) Dump the SQL out (pg_dump database-name > dump.sql if using postgresql)

(3) restart tomcat and make sure everything you want is there

(4) dump out GFF3 (knock on wood)

(5) stop tomcat and apply the memory settings as alluded to above.

Cheers,

Nathan

--
To unsubscribe from this group and stop receiving emails from it, send an email to apollo+un...@lbl.gov.
<catalina.out><localhost.2021-05-05.log>

[Nicolas Moya]

Hi Nathan, 

Thanks for the prompt reply.

I have a few follow-up questions. I am a bit unfamiliar with the Apollo folder structure. I tried to search for the database based on the directions on the documentation(var/lib/tomcat8/apollo_data), but I was unable to find it. Additionally, I did not manually configure the database, I simply ran with the existing configuration in the community AMI. How can I find which database type am I currently using?

Sorry if these questions are basic. This is my first time running into any issues with Apollo.

Best,

Nic

[Nathan Dunn]

On May 6, 2021, at 2:56 PM, Nicolas Moya <nicolas...@u.northwestern.edu> wrote:

Hi Nathan, 

Thanks for the prompt reply.

I have a few follow-up questions. I am a bit unfamiliar with the Apollo folder structure. I tried to search for the database based on the directions on the documentation(var/lib/tomcat8/apollo_data), but I was unable to find it. Additionally, I did not manually configure the database, I simply ran with the existing configuration in the community AMI. How can I find which database type am I currently using?

How did you get it running and deployed? 

My guess is that there is a file called AnnotationDatabase:devDb.h2.db somewhere.

Also, what happens if you restart tomcat?  (Step 3 below)

Sorry if these questions are basic. This is my first time running into any issues with Apollo.

These are good questions.  We don’t want to lose data!

Nathan

[Nicolas Moya]

Hi Nathan,

Thanks again for the quick reply. I'll do a recursive search for any '.db' files. As for how I got it running, the moment I activated the instance with the AMI and accessed my ipv4, Apollo was already running and ready to use. Is this unexpected?

Additionally, how would I go about restarting tomcat? and should I backup my AWS drive prior to restarting it?

Best,

Nic

[Nathan Dunn]

On May 6, 2021, at 5:12 PM, Nicolas Moya <nicolas...@u.northwestern.edu> wrote:

Hi Nathan,

Thanks again for the quick reply. I'll do a recursive search for any '.db' files. As for how I got it running, the moment I activated the instance with the AMI and accessed my ipv4, Apollo was already running and ready to use. Is this unexpected?

Oh, if you launched from AWS (one of the community instances) its preconfigured with PostgreSQL so the data is probably safe. 

Its preloaded to run with enough memory.  Are you using an instance with enough memory (4 GBs)?   Typically a medium (I use t2.medium) should be sufficient. 

You can double-check:

cat /proc/meminfo 

MemTotal:        4044604 kB 

Mine is 4GBs, as you can see above. 

To backup the database, do as follows, below:

sudo su postgres

pg_dump apollo-release-production > /tmp/apollo-dump.sql

exit

cp /tmp/apollo-dump.sql . 

 

Additionally, how would I go about restarting tomcat?

 sudo service tomcat8 restart

and should I backup my AWS drive prior to restarting it?

It won’t hurt, but I don’t think its necessary.

Nathan

[Nicolas Moya]

Hi Nathan,

Thank you so much for all the advise.

I have successfully dumped the Apollo.sql. I have also restarted tomcat, and Apollo seems to be working once again. I dumped the GFF3 file successfully as well, and it seems all the information is intact!

I will migrate these annotations to a new instance with an updated memory configuration (we need to migrate because of an unrelated reason). I also checked that there is sufficient memory (MemTotal: 8173384 kB), so hopefully updating the configuration will do the trick.

Thanks again, you have saved us a week's worth of work.

Best,

Nic

[Nathan Dunn]

Awesome.  Glad it worked out. 

Nathan

[Nicolas Moya]

Hi Nathan,

I have one last follow up question. When trying to configure the memory limits for tomcat, I am unable to find setenv.sh. It is likely this file doesn't exist. I was confused about where the 'tomcat binaries reside'. Where should I create the setenv.sh file?

Additionally, I found a 'defaults.template' file in CATALINA_HOME (/usr/share/tomcat8/) that seems to set the defaults for JVM. Would modifying these parameters have the same effect?

# The home directory of the Java development kit (JDK). You need at least

# JDK version 7. If JAVA_HOME is not set, some common directories for

# OpenJDK and the Oracle JDK are tried.

#JAVA_HOME=/usr/lib/jvm/java-7-openjdk

# You may pass JVM startup parameters to Java here. If unset, the default

# options will be: -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC

#

# Use "-XX:+UseConcMarkSweepGC" to enable the CMS garbage collector (improved

# response time). If you use that option and you run Tomcat on a machine with

# exactly one CPU chip that contains one or two cores, you should also add

# the "-XX:+CMSIncrementalMode" option.

JAVA_OPTS="-Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC"

Please let me know how to proceed.

Thanks in advance,

Nic

[Nicolas Moya]

It seems that the binaries reside in:  /usr/share/tomcat8/bin

You can disregard the first question.

[Nicolas Moya]

After creating setenv.sh, restarting, and verifying with  'ps -ef | grep java',  JVM seems to be taking advantage of more memory. Should I set Xmx to my maximum memory(8GB)? or leave some free memory for other tasks?

Best,

Nic

[Nathan Dunn]

If your system memory is 8GB, you should use 4 GBs or less otherwise you risk crashing the entire machine.   

If its just a small group (less than 5 people), I would probably even do 2 GBs.  

In general, 500 MBs is the absolute minimum, with 1GB being the practical minimum and really shouldn’t need more than 8 GBs ever unless the diagnostics prove me wrong.  By default I use 2 GBs on most of my demo machines.

Glad it seems to be working. 

Nathan

[Nicolas Moya]

Hi Nathan.

Thanks for the info. Sorry to bother once again, but I have (hopefully) one last question.

I'm having trouble understanding the usage of the script to bulk load the GFF3. I am trying to load the recovered GFF3 into the new instance. I've uploaded the genome and tracks. Am I supposed to run the script from the AWS console using (localhost:8080) or from an external computer (using the AWS IP)? If the latter is correct, how do I handle the keys needed to connect to AWS?

Thanks in advance,

Nic

[Nathan Dunn]

I’m not sure if you are using the perl script or this (the arrow commands): https://pypi.org/project/apollo/  (with more instructions here: https://genomearchitect.readthedocs.io/en/latest/Web_services.html?highlight=arrow#python-client) 

Generally you’ll use the external IP (example http://17.34.88.19:8080/apollo) .    If you can access the website over http (generally 8080 or 80 for the stock AWS template), then you shouldn’t need any keys.   It emulates creating them from the web interface. 

For using the python library above I use:

arrow annotations load_gff3 organism_name test-mRNAOnly.gff

Nathan

[Nicolas Moya]

Hi Nathan,

Thanks for the details. I was able to successfully load the annotations into our new instance. However, it seems that the script throws several warnings when adding annotations with non-canonical splice sites.

"Ignoring unsupported sub-feature type: non_canonical_five_prime_splice_site"

We'd like to keep those annotations, yet I don't see a parameter for that in the add_features_from_gff3_to_annotations.pl script. Is there any way we might be able to keep them? or should we just re-annotate those few cases?

Best,

Nic

[Nicolas Moya]

Additionally, is there a way to remove all annotations in bulk? (without deleting the whole organism and its tracks)

Thanks,

Nic

[Nicolas Moya]

As a followup to the annotation removal. I tried using delete_annotations_from_organism.groovy and it seems to return an authentication error:

May 17, 2021 9:47:11 PM org.apache.http.impl.client.DefaultHttpClient handleResponse

WARNING: Authentication error: Unable to respond to any of these challenges: {}

Caught: groovyx.net.http.HttpResponseException: Unauthorized

groovyx.net.http.HttpResponseException: Unauthorized

        at groovyx.net.http.RESTClient.defaultFailureHandler(RESTClient.java:240)

        at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:508)

        at groovyx.net.http.RESTClient.post(RESTClient.java:140)

        at groovyx.net.http.RESTClient$post.call(Unknown Source)

        at delete_annotations_from_organism.run(delete_annotations_from_organism.groovy:74)

I'm ensured there were no mistakes in the admin username/password. Additionally, the users are now stuck loading in the Users menu (see below). Restarting tomcat doesn't seem to fix that issue.

How should I proceed?

Best,

Nic

[Nathan Dunn]

On May 17, 2021, at 1:51 PM, Nicolas Moya <nicolas...@u.northwestern.edu> wrote:

Hi Nathan,

Thanks for the details. I was able to successfully load the annotations into our new instance. However, it seems that the script throws several warnings when adding annotations with non-canonical splice sites.

"Ignoring unsupported sub-feature type: non_canonical_five_prime_splice_site"

We'd like to keep those annotations, yet I don't see a parameter for that in the add_features_from_gff3_to_annotations.pl script. Is there any way we might be able to keep them? or should we just re-annotate those few cases?

It should automatically re-add them on the fly as they aren’t features that you can explicitly annotate. 

There is this option, though not sure if that is what you want or not:

 

[--disable_cds_recalculation|-X]

[Nathan Dunn]

I would raise a GitHub issue for this with your usage clearly detailed. 

I’m also unsure if the python-apollo library might be a better fit as its been more recently maintained:

%arrow organisms 

Usage: arrow organisms [OPTIONS] COMMAND [ARGS]...

Options:

  -h, --help  Show this message and exit.

Commands:

  add_organism          Add an organism Output: a dictionary with...

  delete_features       Remove features of an organism Output: an empty...

  delete_organism       Delete an organism Output: A list of all remaining...

  get_organism_creator  Get the creator of an organism Output: a dictionary...

  get_organisms         Get all organisms Output: Organism information

  get_sequences         Get the sequences for an organism Output: The set

                        of...

  show_organism         Get information about a specific organism.

  update_metadata       Update the metadata for an existing organism.

  update_organism       Update an organism Output: a dictionary with...

%arrow organisms delete_features -h

Usage: arrow organisms delete_features [OPTIONS] ORGANISM_ID

  Remove features of an organism

  Output:

      an empty dictionary

Options:

  -h, --help  Show this message and exit.

https://github.com/galaxy-genome-annotation/python-apollo

Nathan

On May 17, 2021, at 2:53 PM, Nicolas Moya <nicolas...@u.northwestern.edu> wrote:

As a followup to the annotation removal. I tried using delete_annotations_from_organism.groovy and it seems to return an authentication error:

May 17, 2021 9:47:11 PM org.apache.http.impl.client.DefaultHttpClient handleResponse

WARNING: Authentication error: Unable to respond to any of these challenges: {}

Caught: groovyx.net.http.HttpResponseException: Unauthorized

groovyx.net.http.HttpResponseException: Unauthorized

        at groovyx.net.http.RESTClient.defaultFailureHandler(RESTClient.java:240)

        at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:508)

        at groovyx.net.http.RESTClient.post(RESTClient.java:140)

        at groovyx.net.http.RESTClient$post.call(Unknown Source)

        at delete_annotations_from_organism.run(delete_annotations_from_organism.groovy:74)

I'm ensured there were no mistakes in the admin username/password. Additionally, the users are now stuck loading in the Users menu (see below). Restarting tomcat doesn't seem to fix that issue.

<usersloading.png>

[Garrett Stevens]

Hi Nicolas,

I tried to reproduce your problem with the "Users" tab stuck loading, but was not able to. That Groovy script seemed to work for me, but even when I intentionally put in the wrong credentials and got the same error message as you, I didn't see the problem with the "Users" tab in the UI. If you're able to reproduce it, we can open a GitHub issue for it.

Like Nathan said, you might be better served using the apollo python package (installable with pip or bioconda), which comes with the arrow CLI tool. Then you can do something like:

$ # init lets you set the username, password, and URL

$ arrow init

$ arrow organisms delete_features MyOrganism

You also might have better luck using arrow to load non_canonical_five_prime_splice_site from your GFF3 (arrow annotations load_gff3) than the Perl script.

-Garrett