LiftOver chain files for new version tree shrew genome and rabbit genome to human genome conversions
Lin Ou
Dear
UCSC staff:
I'm a Ph.D candidate from Tsinghua University, and
recently I'm trying to map genomic coordinates from the tree shrew genome(TS_3.0
genome annotation, http://www.treeshrewdb.org/download.html)
to human genome(hg38) and from rabbit genome(Genome assembly OryCun3.0, https://www.ncbi.nlm.nih.gov/datasets/genome/GCA_013371645.1/)
to human genome(hg38).
First
I used your program LiftOver (thank you for making it available for so
many species!), however the available version of tree shrew genome and rabbit genome are not what I am using now.
Hence, I'm attempting to build the necessary chain files following
the directions provided here :
http://genomewiki.ucsc.edu/index.php/DoBlastzChainNet.pl.
But then I faced the “timed out” problem. Even though I tried to install the Parasol job control system following the procedure and succeed:
http://genomewiki.ucsc.edu/index.php/Parasol_job_control_system
when I got back to build chain file, the problem still existed, like this:
29052 jobs in batch
0 jobs (including everybody's) in Parasol queue or running.
Checking finished jobs
updated job database on disk
Pushed Jobs: 29052
================
Checking job status 0 minutes after launch
29052 jobs in batch
0 jobs (including everybody's) in Parasol queue or running.
Sick Batch: consecutive crashes (45) >= sick batch threshold (25)
Checking finished jobs
updated job database on disk
total sick machines: 1 failures: 45
Sick batch! will sleep 10 minutes, clear sick nodes and retry
rudpSend timed out
pmSendString timed out!
pmSendString: will sleep 60 seconds and retry
Told hub to clear sick nodes
================
Checking job status 11 minutes after launch
29052 jobs in batch
0 jobs (including everybody's) in Parasol queue or running.
Checking finished jobs
updated job database on disk
Pushed Jobs: 29052
Retried jobs: 29052
Have you encountered this problem before, and do
you know how to solve this problem? perhaps there might be an update or some
other documentation for chain-file generation besides the link above? If
possible, would you please provide us the new version chain files?
I have tried to make this chain files for several weeks and I feel sorry because your team already provided detailed guidance. Besides, I was running these procedures in a single pod built from a cluster, and I don’t know if my configuration is suitable for this task. Any more suggestions from you would be very helpful.
Thanks very much again for providing this valuable
resource for the community. Wish you better and better!
best regards,
Lin Ou
Luis Nassar
Hello, Lin.
That documentation is the best we have; it's a technical pipeline originally designed for our system, so it is not uncommon for issues to come up when others run it.
We can generate the chain files you are looking for, GCA_013371645.1 to hg38 is not a problem, but we do require that the genome have an NCBI accession, e.g., GCA or GCF. Is TS_3.0 not in NCBI Genbank?
We see version 2 (https://www.ncbi.nlm.nih.gov/datasets/genome/GCA_033439345.1/), but that may not be suitable for you, let us know.
If you would like to try and debug the chain pipeline, one of our engineers shares the following:
This means that communication from node process to hub process is failing with a timeout,
so it is not connecting. This could be because the paraHub daemon is not running.Or perhaps a firewall interferes, although that is much less common on internal
machines that are not acting as web servers so they do not need as much security.What shell is the user running commands in.
Internally we use bash shell, I suggest the user might run it in their bash shell too.
--
---
You received this message because you are subscribed to the Google Groups "UCSC Genome Browser Public Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genome+un...@soe.ucsc.edu.
To view this discussion visit https://groups.google.com/a/soe.ucsc.edu/d/msgid/genome/77ac65e1-1951-4ee7-8c8a-eba28c920a33n%40soe.ucsc.edu.
Lou Nassar
UCSC Genomics Institute
Lin Ou
Thank you so much for your quick reply and help! You’re so kind for helping me build the chain files and deal with the bugs!
For the new tree shrew version you mentioned
( https://www.ncbi.nlm.nih.gov/datasets/genome/GCA_033439345.1/), it is suitable for us, the sequence are the same, but there is a little difference in the chromosome name, like this, the partial content in red box is from NCBI, and the content in green box is from ours:
For the chain pipeline, I also run it in our bash shell. So, the problem maybe the firewall interferes, which I don’t have the authority to check and change (so pity!).
But recently, when I browsed
and traced back the mailing lists of similar problems (thanks again for dealing
with our mails patiently and timely!), I found an
alternative method by using nf-lo(https://nf-lo.readthedocs.io/en/latest/index.html):
Then I tried according to the tutorial, and successfully generated many single chains. However, the run failed at “chainMergeSort: Argument list too long”. The end of the content in the “.nextflow.log” file is shown below:
Sep-18 15:37:41.695 [TaskFinalizer-5] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=LASTZ:chainMerge (chainmerge); work-dir=/home/oulin/software/chain_files/treeshrewTohuman4/work/b8/4cfd7ca26cfc3b83fdb6cee431f86b
error [nextflow.exception.ProcessFailedException]: Process `LASTZ:chainMerge (chainmerge)` terminated with an error exit status (1)
Sep-18 15:37:41.695 [TaskFinalizer-5] INFO nextflow.processor.TaskProcessor - [b8/4cfd7c] NOTE: Process `LASTZ:chainMerge (chainmerge)` terminated with an error exit status (1) -- Error is ignored
Sep-18 15:37:41.697 [main] DEBUG nextflow.Session - Session await > all processes finished
Sep-18 15:37:41.793 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Sep-18 15:37:41.793 [main] DEBUG nextflow.Session - Session await > all barriers passed
Sep-18 15:37:41.808 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Sep-18 15:37:41.809 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Sep-18 15:37:41.915 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1074826; failedCount=3; ignoredCount=1; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=2; abortedCount=0; succeedDuration=1d 14h 4m 59s; failedDuration=1h 10m; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=52; peakCpus=52; peakMemory=263 GB; ]
Sep-18 15:37:41.915 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Sep-18 15:37:41.934 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Sep-18 15:39:25.197 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Sep-18 15:39:37.400 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Sep-18 15:39:37.521 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Sep-18 15:39:37.521 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
It really upset me very much. Have you encountered this problem? Any more advice will be helpful, and I will do my best to try.
Thanks very much again! You are really nice! Wish you better and better!
I may bother you too much (so sorry! I will try my best to create chains myself), but it will be even helpful for us if these chain files can be also available:
human(hg38) to tree shrew (TS_3.0),
human(hg38) to rabbit (OryCun3.0),
tree shrew (TS_3.0) to mouse(mm10),
rabbit (OryCun3.0) to mouse(mm10),
mouse(mm10) to tree shrew (TS_3.0) and
mouse(mm10) to rabbit (OryCun3.0).
best regards,
Lin Ou