RocksDB support on apple silicon M1

[Hu Zhixiang]

Hi,

Anybody tried to build rocksdb for Apple Silicon M1 so far?

Not sure if it's possible to build on an Intel x86_64 Mac, or we must build on Apple M1 Mac like mentioned here: https://github.com/facebook/rocksdb/issues/7710?

Or what would be the best/right setup for the build?

I'm also curious whether there is any plan to support Apple Silicon...

Thanks,

Jonathan

[Adam Retter]

Hi Hu,

I have been working on the Apple Silicon support. RocksDB should
compile just fine on M1. There are still some improvements in the
pipeline to offer Universal binaries for RocksJava etc, but at the
moment compilation works just fine.

> --
> You received this message because you are subscribed to the Google Groups "rocksdb" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/rocksdb/2128388f-8d2c-49ed-a61c-76d1d2d8bf3cn%40googlegroups.com.



--
Adam Retter

skype: adam.retter
tweet: adamretter
http://www.adamretter.org.uk

[Adam Retter]

Hi Jonathan,

As requested, you can find the start of my work on Universal Binaries
for RocksDB here -
https://github.com/adamretter/rocksdb/tree/macos-multi-arch

Running `make static_lib_ub` will give you a Universal Binary for both
x86_64 arm64.

Building the universal binaries can be done in macOS on either Intel
or Apple Silicon as long as you have macOS bigSur and the latest
XCode.

[胡志祥]

Hi Adam,

I tried and the build succeed on my x86_64 Mac machine. Awesome! Thanks.

best regards,

Jonathan

Adam Retter <adam....@googlemail.com> 于2021年1月25日周一 下午9:44写道：

[Adam Retter]

You are very welcome. Enjoy.

[胡志祥]

Hi Adam,

One quick update, few link errors jump out when trying to link.

I got rid of them by adding "ARMCRC_SOURCE=1 when making the arm64 build.

Not sure if this is the right solution though, not able to verify if it runs either.

Undefined symbols for architecture arm64:
 “crc32c_arm64(unsigned int, unsigned char const*, unsigned long)“, referenced from:
   rocksdb::crc32c::ExtendARMImpl(unsigned int, char const*, unsigned long) in librocksdb.a(crc32c.o)
 “crc32c_runtime_check()“, referenced from:
   rocksdb::crc32c::IsFastCrc32Supported() in librocksdb.a(crc32c.o)
   l002 in librocksdb.a(crc32c.o)
 “crc32c_pmull_runtime_check()“, referenced from:
   rocksdb::crc32c::IsFastCrc32Supported() in librocksdb.a(crc32c.o)
   l002 in librocksdb.a(crc32c.o)
ld: symbol(s) not found for architecture arm64

Best regards,

Jonathan

Adam Retter <adam....@googlemail.com> 于2021年1月26日周二 上午10:22写道：

[Adam Retter]

Okay that's strange, I didn't see those on my x86_64 mac. Are you
using HEAD? Also which exact make commands are you running?

Thanks Adam.

[胡志祥]

Hi Adam,

Here is what I did:

git clone https://github.com/adamretter/rocksdb.git

git checkout macos-multi-arch

Then I disabled all compression libs by export the following env variables for both x86_64 and arm64 build: 

export ROCKSDB_DISABLE_ZLIB=1 ROCKSDB_DISABLE_BZIP=1 ROCKSDB_DISABLE_ZSTD=1 ROCKSDB_DISABLE_SNAPPY=1 ROCKSDB_DISABLE_LZ4=1

Then I built with the following command

make static_lib_ub -j8

After that I took the lib file and link with my application, where the link error pops up.

So I could see from git log, that the last change is from you, and the previous one is from Levi Tamasi.

My MacOS version is 11.1 Big Sur, and Xcode version is 12.3

Thanks & best regards,

Jonathan

Adam Retter <adam....@googlemail.com> 于2021年1月26日周二 下午3:51写道：

[Adam Retter]

> Then I built with the following command
> make static_lib_ub -j8
>

So the linking issues arrive, when you reuse the library and try and
link it with your own application. I would guess then that you are
missing some options to the linker to include the crc arm libraries. I
am not sure what you need to here. Perhaps if you trace the
compilation in RocksDB you can work out what libraries it links with
to add the crc arm support, and then include those in your link
process also?

> After that I took the lib file and link with my application, where the link error pops up.
> So I could see from git log, that the last change is from you, and the previous one is from Levi Tamasi.
>

>> > Not sure if this is the right solution though, not able to verify if it runs either.
>> >
>> > Undefined symbols for architecture arm64:
>> > “crc32c_arm64(unsigned int, unsigned char const*, unsigned long)“, referenced from:
>> > rocksdb::crc32c::ExtendARMImpl(unsigned int, char const*, unsigned long) in librocksdb.a(crc32c.o)
>> > “crc32c_runtime_check()“, referenced from:
>> > rocksdb::crc32c::IsFastCrc32Supported() in librocksdb.a(crc32c.o)
>> > l002 in librocksdb.a(crc32c.o)
>> > “crc32c_pmull_runtime_check()“, referenced from:
>> > rocksdb::crc32c::IsFastCrc32Supported() in librocksdb.a(crc32c.o)
>> > l002 in librocksdb.a(crc32c.o)
>> > ld: symbol(s) not found for architecture arm64

[胡志祥]

Hi Adam,

Those functions are called from util/crc32c.cc, and one possible implementation is defined in util/crc32c_arm64.cc which is only built if 'ARMCRC_SOURCE' is defined.

//from src.mk

ifeq ($(ARMCRC_SOURCE),1)

LIB_SOURCES +=\

  util/crc32c_arm64.cc

endif

So that is why I added 'ARMCRC_SOURCE=1' in my compilation.

Don't you have the same error if you link the lib with your application?

For "crc arm libraries", do you mean any specific library? What's your view on the implementation in util/crc32c_arm64.cc?

Thanks,

Jonathan

Adam Retter <adam....@googlemail.com> 于2021年1月27日周三 上午10:28写道：