Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Using memmapfile with a nested format structure

71 views
Skip to first unread message

AussieMike

unread,
Jul 14, 2016, 5:51:08 PM7/14/16
to
Hi everybody, I have been processing large binary files but have found the fread method to be far too slow. I have read up on the memmapfile method and it seems like it would solve my speed problems. However, I don't think my data structure allows for easy application without some way to 'nest' the format structure.

My files contain around 40 blocks of data in the following format: [16 byte header containing 3 values] [5 Data values totaling 12 bytes which are repeated around 300000 times].

I can see that if my blocks were just [Header] [Data] it would be easy to implement, but since it is [Header] [Data] [Data] [Data]...times 300000 then [Header] [Data] [Data] ... etc it is very hard to describe without having some way to nest the structure.

I hope I have included enough information for someone to offer some advice, I am happy to elaborate more. I don't see that memmapfile is compatible with what I want which is frustrating since I have seen this achieved in python but need to replicate it in matlab. Can anyone offer any advice?

Thanks very much.

AussieMike

unread,
Jul 14, 2016, 5:59:20 PM7/14/16
to
Actually I can see some possibilities treating the header and data as separate then using the offset in a loop over the length of the file. I will try this now but if anyone else has some advice that would be greatly appreciated. Thank you!
0 new messages