plink2_text library

[choish...@gmail.com]

Hi Chris,

Thanks for the software and library. I am currently trying to update our software to use PGEN data and are having some trouble using the libraries.

For plink2_text, do we need to have a global memory pool like g_bigstack_base for us to use it? If not, what are the preferred way of memory allocation? I saw that SizeAndInitTextStream is used in psam.cc and TextStreamOpenEx is used in pmerge.cc. Which is the preferred function?

Sorry for the troubles.

Thanks

Sam

[Christopher Chang]

No, plink2_text does not require a global memory pool.  See LoadMinimalPvar() in pvar_ffi_support.cc for a straightforward TextStreamOpen() example.

[choish...@gmail.com]

Thanks! Will do.

[choish...@gmail.com]

Hi Chris, if you don't mind me asking, is multi-thread read / write implemented in the library? If so, is it InitBgzfCompressStream?

Thanks!

[Christopher Chang]

Yes.  TextStreamOpenEx() has a decompress_thread_ct parameter (which is set by TextStreamOpen() to the number of detected cores).

[Christopher Chang]

(oops, that's just the reader; yes, InitBgzfCompressStream can be used for writing and also has a thread_ct parameter.)

[choish...@gmail.com]

Thanks Chris, based on the usage in plink_merge.cc,  I guess the parameters should be:

1. File name

2. Maximum length of line allow? (plink2::kMaxLongLine)

3. The size of the buffer?  I guess that would be plink2::kMaxLongLine + plink2::kDecompressChunkSize  if we don't have the global memory pool size? Or should that preferrably be a different number?

4. Number of thread use for decompress

5. Not sure what this is, only know that it is a textFile pointer and can be a nullptr

6. The linebuf storage

7. The initialized plink2::TextStream

Is that correct? Sorry for the troubles and thank you for your library.

[Christopher Chang]

The simpler TextStreamOpen() wrapper is defined as:

HEADER_INLINE PglErr TextStreamOpen(const char* fname, TextStream* txs_ptr) {

  return TextStreamOpenEx(fname, kMaxLongLine, 0, NumCpu(nullptr), nullptr, nullptr, txs_ptr);

}

If you are not using a global memory pool, this form will allocate the necessary memory for you (and free it when you call CleanupTextStream on the returned plink2::TextStream).  Feel free to change the decompressor thread count.

[choish...@gmail.com]

That's exactly what I needed. Thanks Chris!