Does `roaring_bitmap_portable_serialize`(Croaring) really serialise in little endian encoding?

30 views
Skip to first unread message

Abhradeep Chakraborty

unread,
Jul 16, 2022, 9:04:17 AM7/16/22
to Roaring Bitmaps
I see the RoaringFormatSpec Readme[1] file saying - "All words are
written using little endian encoding.".

But when I look into the `ra_portable_serialize()`[2] function, it
seems to me like words are not written using little endian encoding.
Instead the endianness of these written words are dependent on the
machine's endianness.

For example,  in the below code (line no. 573 [3]), `cookie` variable
is not converted into little endian format explicitly. So, in a Big
endian system it would likely be stored in big endian.

  uint32_t cookie = SERIAL_COOKIE | ((ra->size - 1) << 16);
  memcpy(buf, &cookie, sizeof(cookie));

Am I missing something?

Daniel Lemire

unread,
Jul 17, 2022, 4:04:17 PM7/17/22
to Roaring Bitmaps
The question refers to the CRoaring library.

Please review the CRoaring requirements from the README, pay attention to the second point...

Requirements
  • Linux, macOS, FreeBSD, Windows (MSYS2 and Microsoft Visual studio).
  • We test the library with ARM, x64/x86 and POWER processors. We only support little endian systems (big endian systems are vanishingly rare).
  • Recent C compiler supporting the C11 standard (GCC 7 or better, LLVM 7.0 or better, Xcode 11 or better), there is also an optional C++ class that requires a C++ compiler supporting the C++11 standard.
  • CMake (to contribute to the project, users can rely on amalgamation/unity builds if they do not wish to use CMake).
  • Under x64 systems, the library provides runtime dispatch so that optimized functions are called based on the detected CPU features. It works with GCC, clang (version 9 and up) and Visual Studio (2017 and up). Other systems (e.g., ARM) do not need runtime dispatch.


If you need to support a big-endian system, it should not be very difficult to adapt the code. Pull requests are invited.

Reply all
Reply to author
Forward
0 new messages