Reading data from a small number of files in parallel

[Ice man]

Hi everyone, I am building a system where we store data (text) in a file for a day and at the end of the day the file is rotated. For each data, one single file contains the data for that file. I was wondering what would be the best way to read from the file, opening the file for reading each time, seeking to the offset for the data and reading or maintaining a pool of frequently accessed files as *os.File and reading from them by seeking, if possible.

[Robert Engels]

You cannot use a pool of File if rotated means close, rename , reopen. A File is a user space representation of an inode so rotating means the File still points to the original file (beware on Windows). 

A file open / reopen is really cheap on recent files in a modern os. 

If the real issue you are having is the parallel reading - ie merging the days - you want something like an LVM structure.    

See github.com/robaho/key db and the parallel iterators it uses. 

On Feb 14, 2020, at 8:28 AM, Ice man <ice3m...@gmail.com> wrote:



Hi everyone, I am building a system where we store data (text) in a file for a day and at the end of the day the file is rotated. For each data, one single file contains the data for that file. I was wondering what would be the best way to read from the file, opening the file for reading each time, seeking to the offset for the data and reading or maintaining a pool of frequently accessed files as *os.File and reading from them by seeking, if possible.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f96c3435-73b0-4821-9374-10a9e357a8d2%40googlegroups.com.