Nextflow stalled on workflow completion

[Steve]

For some reason, my workflow is completeing successfully but Nextflow fails to finish, it just hangs after all steps are completed. The logs all look something like this:

Feb-21 07:37:19.493 [Task submitter] INFO  nextflow.Session - [16/adffac] Submitted process > custom_analysis_report (180720_NB501073_0060_AHFMGJBGX7_test.report.html)
Feb-21 07:37:55.937 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- pending tasks are shown below
~> TaskHandler[id: 1287; name: custom_analysis_report (180720_NB501073_0060_AHFMGJBGX7_test.report.html); status: RUNNING; exit: -; error: -; workDir: /gpfs/data/molecpathlab/production/NGS580/180720_NB501073_0060_AHFMGJBGX7_test/work/16/adffac92d208b0fde59183386fbcea]
Feb-21 07:39:14.501 [Task monitor] DEBUG n.processor.TaskPollingMonitor - No more task to compute -- The following nodes are still active:
  [operator] tap
  [operator] splitCsv
  [operator] map
  [operator] tap
  [operator] map
  [operator] flatMap
  [operator] flatMap
  [operator] splitCsv
  [operator] map
  [operator] filter
  [operator] tap
  [operator] filter
  [operator] into
  [operator] splitCsv
  [operator] map
  [operator] unique
  [operator] tap
  [operator] combine
  [operator] combine
  [operator] collectFile
  [operator] collectFile
  [operator] collectFile
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] tap
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] tap
  [operator] combine
  [operator] tap
  [operator] combine
  [operator] combine
  [operator] tap
  [operator] concat
  [operator] collectFile
  [operator] combine
  [operator] collectFile
  [operator] mix
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] choice
  [operator] map
  [operator] map
  [operator] toList
  [operator] toList
  [operator] combine
  [operator] filter
  [operator] map
  [operator] combine
  [operator] filter
  [operator] map
  [operator] tap
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] tap
  [operator] combine
  [operator] tap
  [operator] splitCsv
  [operator] map
  [operator] unique
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] tap
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] concat
  [operator] choice
  [operator] map
  [operator] choice
  [operator] map
  [operator] combine
  [operator] combine
  [operator] concat
  [operator] collect
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] filter
  [operator] map
  [operator] combine
  [operator] filter
  [operator] map
  [operator] filter
  [operator] collectFile
  [operator] combine
  [operator] filter
  [operator] map
  [operator] combine
  [operator] filter
  [operator] map
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] combine
  [operator] tap
  [operator] filter
  [operator] filter
  [operator] filter
  [operator] concat
  [operator] tap
  [operator] concat
  [operator] collectFile
  [operator] concat
  [operator] collectFile
  [operator] collect
  [operator] collect
  [operator] map
  [operator] concat
  [operator] groupTuple
  [operator] map
  [operator] tap
  [operator] map
  [operator] concat
  [operator] groupTuple
  [operator] map
  [operator] tap
  [operator] combine
  [operator] filter
  [operator] map
  [operator] combine
  [operator] collect
  [operator] toList

Feb-21 07:39:20.532 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1287; name: custom_analysis_report (180720_NB501073_0060_AHFMGJBGX7_test.report.html); status: COMPLETED; exit: 0; error: -; workDir: /gpfs/data/molecpathlab/production/NGS580/180720_NB501073_0060_AHFMGJBGX7_test/work/16/adffac92d208b0fde59183386fbcea]
Feb-21 07:39:20.534 [Actor Thread 2111] DEBUG nextflow.Session - <<< barrier arrive (process: custom_analysis_report)
Feb-21 07:39:20.534 [main] DEBUG nextflow.Session - Session await > all process finished
Feb-21 07:39:20.534 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local)
Feb-21 07:39:24.521 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm)
Feb-21 07:39:24.521 [main] DEBUG nextflow.Session - Session await > all barriers passed

Any idea what could be causing this? It seems like some channels are failing to close after all tasks are completed. My workflow is pretty large with a lot of channels so the log message is not clear enough to determine where exactly the issue is coming from.

[Steve]

so I went through my workflow and refactored a bunch of dangling `.tap` operators into `.set` and `.into` operators, and the issue seems to have gone away, really strange considering that these have been in the pipeline for a long time and never caused an issue before.