High memory consumption mapping GMGC

[ullo...@googlemail.com]

Dear all, 

did anyone ever observe an unusually high memory consumption performing the mapping within ngless?

```

gmgc_mapped = map (input, reference='gmgc:no-rare',mode_all=True,block_size_megabases=100000)

gmgc_mapped_post = select(gmgc_mapped) using |mr|:

    mr = mr.filter(min_match_size=45, min_identity_pc=95, action={drop})

    if not mr.flag({mapped}):

        discard

gmgc_counts_new = count(gmgc_mapped_post,

                    features=['KEGG_Module'],

                    multiple={all1},

                    normalization={raw},

                    discard_zeros=True)

collect(gmgc_counts_new,

        current=current,

        allneeded=samples,

        ofile='rat'</>'metformin_rats_gmgc_KEGGmodules.all1.raw.txt')

```

I'm wondering why it fails even after allocating 2.5TB of RAM. 

```

[Tue 24-01-2023 06:46:06] Line 13: Will run process /bin/bwa mem -t 1 -K 100000000 -p -a /shared/NGLESSmodules/Modules/gmgc.ngm/1.0/cached/gmgc:no-rare.fna.splits_100000m.0-bwa-0.7.17.fna -

[Tue 24-01-2023 06:46:26] Line 13: fd:13: hClose: resource vanished (Broken pipe)

```

```

Exiting after fatal error:

fd:13: hClose: resource vanished (Broken pipe)

```

```

hard_resources           h_rt=345600,huge=true,m_mem_free=2500G,os=centos7                

granted_req.             8,0:h_rt=4:00:00:00,huge=1,m_mem_free=2.441T,os=centos7

```

Cheers,

Ulrike