[maker-devel] MAKER Commercial User -- Install and run issues

[Hayley Mangelson]

Hi!

I am trying to get a fresh install of MAKER to work for me and am having a few issues. 

1. I got MAKER installed (I thought) and now the log is saying all contigs failed. See attached log.
2. Having tried several versions and methods of installation, I cannot get multithreading to work.

Could you help me work through these issues?

Thanks,

Hayley

--

Hayley Mangelson

Senior Bioinformatics Lead

Phase Genomics, Inc

www.phasegenomics.com

[Carson Holt]

Can you send me the STDERR from the maker run? If you don’t have it, run maker again with the output from the screen redirected to a file. Example —> maker &> all_output.txt

--Carson

Sent from my iPhone

On Jul 31, 2023, at 10:04 AM, Hayley Mangelson <hay...@phasegenomics.com> wrote:



<PGA_assembly_hap1_master_datastore_index.log>

_______________________________________________
maker-devel mailing list
maker...@yandell-lab.org
http://yandell-lab.org/mailman/listinfo/maker-devel_yandell-lab.org

[Carson Holt]

The error is coming from Augustus. There is a problem with the species file you selected. Or you put it in the wrong place.

This is the error —> 

Could not find the config file /home/ec2-user/MAKERTOOLS/maker/exe/augustus/config/species/BUSCO_evans_trout_hap1/BUSCO_evans_trout_hap1_parameters.cfg

The species info must exist under this base directory by default (it’s where you let MAKER install Augustus) —>  /home/ec2-user/MAKERTOOLS/maker/exe/augustus/config/species/

You can also place the species files somewhere else but you need to set the AUGUSTUS_CONFIG_PATH so Augustus knows where to look.

Example (in your ~/.bash_profile):

export AUGUSTUS_CONFIG_PATH=$HOME/muy_augustus_species

Thanks,

Carson

On Aug 11, 2023, at 7:20 AM, Hayley Mangelson <hay...@phasegenomics.com> wrote:

Hi Carson,

Sure thing! Sorry for the delay, I was on vacation. I reran with just a single scaffold. Here is the STDERR.

<all_output.txt>

[Hayley Mangelson]

Hi Carson,

Sure thing! Sorry for the delay, I was on vacation. I reran with just a single scaffold. Here is the STDERR.

On Mon, Jul 31, 2023 at 12:06 PM Carson Holt <cars...@gmail.com> wrote:

[Hayley Mangelson]

Hi Carson,

Excellent, thanks so much! I have that part working. The multithreading is still eluding me. Do you have a recommended method of installing a compatible version of MPI?

Thanks,

Hayley

[Carson Holt]

Once you install MPICH or OpenMPI you must launch MAKER through mpiexec and speciffy the CPU count to MPI. On a single instance you just give the CPU count off that instance, but you can also go across instances using Amazon’s Parallel Cluster. 

Here is an example —> https://efish.integrativebiology.msu.edu/2015/02/10/annotate.html

That example uses StarCluster which is now deprecated, but the MPI install and MAKER run still apply.

Here is another example of using MPi —> https://weatherby.genetics.utah.edu/MAKER/wiki/index.php/MAKER_Tutorial_for_WGS_Assembly_and_Annotation_Winter_School_2018#MPI_Support

Note that for OpenMPI you have to set the LD_PRELOAD environmental variable before launch.

export LD_PRELOAD=/usr/lib64/openmpi-1.10/lib/libmpi.so

—Carson

On Aug 18, 2023, at 2:53 PM, Hayley Mangelson <hay...@phasegenomics.com> wrote:

I am working on an AWS EC2 instance that runs Amazon Linux, and I have tried a couple of installs (MPICH and Open MPI), neither of which worked. I don't have any installed on my current instance.

On Tue, Aug 15, 2023 at 5:08 PM Carson Holt <cars...@gmail.com> wrote:

Do you already have MPI installed on your system, and if so which MPI and what version (i.e. MPICH, MVAPICH2, Intel MPI, Open MPI, etc)?

—Carson

[Carson Hinton Holt]

The seg fault is being thrown by MPI itself. That is why MAKER can run on its own.

There are several things you can check (or just do them all as a sort of nuclear option):

*First make sure you are not using conda or miniconda environments. This can create subtle conflicts between executables and libraries as condo/minicanda will try to override some paths.

*Next you may need to install OpenMPI or MPICH yourself (i.e. don’t use apt-get or other package install managers). https://www.mpich.org/downloads/.

*Lastly you can also install your own perl and override the system perl. Depends on your system configuration, i.e. if using CentOS 7, your perl is very out of date with other libraries that may have been updated on the system.

After compiling everything and installing, you will want to reinstall MAKER (the new MPI install must recompile it’s bindings to the application bering run).

Here is a step by step (comments marked with ‘#'):

#create a directory to prep install

mkdir $HOME/install_prep

#install MPICH

cd $HOME/install_prep

wget https://www.mpich.org/static/downloads/4.1.2/mpich-4.1.2.tar.gz

tar -zxvf mpich-4.1.2.tar.gz

cd mpich-4.1.2/

./configure --prefix=$HOME/opt/mpich

make

make install

echo 'export PATH=$HOME/opt/mpich/bin:$PATH' >> ~/.bash_profile

source ~/.bash_profile

#install a new Perl

cd $HOME/install_prep

wget https://www.cpan.org/src/5.0/perl-5.38.0.tar.gz --no-check-certificate

tar -zxvf perl-5.38.0.tar.gz

cd perl-5.38.0

./Configure -d -Dprefix=$HOME/opt/perl5

make

make install

echo 'export PATH=$HOME/opt/perl5/bin:$PATH' >> ~/.bash_profile

source ~/.bash_profile

#install MAKER with MPI support

cd $HOME/opt

wget http://weatherby.genetics.utah.edu/maker_downloads/AD6E/A1B0/C2B7/558563CFD35E343BA1B888148294/maker-3.01.04.tgz

tar -zxvf maker-3.01.04.tgz

rm -rf maker-3.01.04.tgz

cd maker/src

perl Build.PL ##say yes to MPI question

./Build installdeps ##just hit 'enter' to accept any defaults

### if you get errors mentioning '--no-check-certificate', run the command 'cpan' in your terminal.

### If it asks to set up everything automatically, say 'yes'.

### Next, whil einside cpan, run the command 'o conf init pushy_https'.

### Say 'no' to the question it asks, and then run 'o conf commit' to save changes and 'q' to exit.

### Then retry the './Build installdeps' command

./Build installexes

./Build install

echo 'export PATH=$HOME/opt/maker/bin:$PATH' >> ~/.bash_profile

source ~/.bash_profile

#test MAKER

cd $HOME/install_prep

mpiexec -n 10 maker --help ##should print a single help message

cp -R $HOME/opt/maker/data ./test

cd test

maker -CTL

mpiexec -n 10 maker --protein dpp_protein.fasta --genome dpp_contig.fasta --est dpp_est.fasta ##should run MAKER under MPI

—Carson

On Sep 12, 2023, at 2:22 PM, Hayley Mangelson <hay...@phasegenomics.com> wrote:

Hi Carson,

I have tried installing MPICH and OpenMPI various ways now and am still having issues. MAKER appears to be running when I am not using MPI, but I am attaching the maker output (with -debug flag) and here is the error I get when trying to multithread.

===================================================================================

=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

=   PID 35952 RUNNING AT ip-172-31-12-194

=   EXIT CODE: 9

=   CLEANING UP REMAINING PROCESSES

=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

===================================================================================

YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Segmentation fault (signal 11)

This typically refers to a problem with your application.

Please see the FAQ page for debugging suggestions

[Hayley Mangelson]

I am working on an AWS EC2 instance that runs Amazon Linux, and I have tried a couple of installs (MPICH and Open MPI), neither of which worked. I don't have any installed on my current instance.

On Tue, Aug 15, 2023 at 5:08 PM Carson Holt <cars...@gmail.com> wrote:

Do you already have MPI installed on your system, and if so which MPI and what version (i.e. MPICH, MVAPICH2, Intel MPI, Open MPI, etc)?

—Carson

[Hayley Mangelson]

Hi Carson,

Thought I should report back that this worked! Now I am just dealing with 4/32 of my contigs failing. It looks like they are failing on the repeat masking step. I am a bit confused as to why just a handful would fail here. Any suggestions on how to overcome this (hopefully final) obstacle?

Thanks,

Hayley

[Carson Hinton Holt]

What does the error say? One thing you can do is run on a single thread (i.e. no mpiexec, just call maker directly - increase try count before doing this). That will run up to where the error happens, and the last thing it prints out before the error is the Repeatmasker command line used. You can then cut and paste that command to test it outside of MAKER.

—Carson