Problem compiling Chipyard (Firrtl) on M1 Mac

[sam...@gmail.com]

Hi all,

So on my new MacBook with M1 CPU, when I try to install some sub-projects of Chipyard locally (with sbt 1.5.5 from homebrew), say:

sbt "rocket-dsp-utils / publishLocal"

I would get this error:

protoc-jar: downloading: https://repo.maven.apache.org/maven2/com/github/os72/protoc/3.5.1-build2/protoc-3.5.1-build2-osx-aarch_64.exe
[error] java.lang.RuntimeException: error occurred while compiling protobuf files: Unsupported platform: protoc-3.5.1-osx-aarch_64.exe

which is probably due to the use of sbt-protobuf plugin in firrtl. I installed aarch64 version of protobuf via homebrew but it won’t pickup. So I looked into the place it’s called.
In chipyard/tools/firrtl/project/plugins.sbt

addSbtPlugin("com.github.gseitz" % "sbt-protobuf" % "0.6.5")

and in chipyard/tools/firrtl/build.sbt

lazy val protobufSettings = Seq(
  sourceDirectory in ProtobufConfig := baseDirectory.value / "src" / "main" / "proto",
  protobufRunProtoc in ProtobufConfig := (args =>
    com.github.os72.protocjar.Protoc.runProtoc("-v351" +: args.toArray)
  )
)

Then I tried switching the os72 line to version “-v3.17.3” coz that’s the version that has ARM binary, still the same error about unsupported platform.
Then I tried to change the plugin line to addSbtPlugin("com.github.gseitz" % "sbt-protobuf" % "0.6.5-osx-x86_64"), trying to use the x86_64 binary thru Rosetta2, gives error:

[error] sbt.librarymanagement.ResolveException: Error downloading com.github.gseitz:sbt-protobuf;sbtVersion=1.0;scalaVersion=2.12:0.6.5-osx-x86_64
[error]   Not found
[error]   Not found
[error]   not found: https://repo1.maven.org/maven2/com/github/gseitz/sbt-protobuf_2.12_1.0/0.6.5-osx-x86_64/sbt-protobuf-0.6.5-osx-x86_64.pom
[error]   not found: /Users/mhho/.ivy2/localcom.github.gseitz/sbt-protobuf/scala_2.12/sbt_1.0/0.6.5-osx-x86_64/ivys/ivy.xml
[error]   download error: Caught javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed (PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed) while downloading https://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases/com.github.gseitz/sbt-protobuf/scala_2.12/sbt_1.0/0.6.5-osx-x86_64/ivys/ivy.xml
[error]   not found: https://download.eclipse.org/jgit/maven/com/github/gseitz/sbt-protobuf_2.12_1.0/0.6.5-osx-x86_64/sbt-protobuf-0.6.5-osx-x86_64.pom
[error]   not found: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.github.gseitz/sbt-protobuf/scala_2.12/sbt_1.0/0.6.5-osx-x86_64/ivys/ivy.xml
[error]   not found: https://repo.typesafe.com/typesafe/ivy-releases/com.github.gseitz/sbt-protobuf/scala_2.12/sbt_1.0/0.6.5-osx-x86_64/ivys/ivy.xml
[error]     at lmcoursier.CoursierDependencyResolution.unresolvedWarningOrThrow(CoursierDependencyResolution.scala:258)

At last I tried to change the calling line to force the protoc path to be from my homebrew install:

lazy val protobufSettings = Seq(
  sourceDirectory in ProtobufConfig := baseDirectory.value / "src" / "main" / "proto",
  protobufProtoc in ProtobufConfig := "/opt/homebrew/bin/protoc"
  // protobufRunProtoc in ProtobufConfig := (args =>
  //   com.github.os72.protocjar.Protoc.runProtoc("-v351" +: args.toArray)
  // )
)

Still gives error, but seems to be the closest I get:

[error] /Users/mhho/repos/github.com/ucb-bar/chipyard/tools/firrtl/target/scala-2.12/src_managed/main/compiled_protobuf/firrtl/FirrtlProtos.java:1775:51: type InternalOneOfEnum is not a member of com.google.protobuf.AbstractMessage
[error]               com.google.protobuf.AbstractMessage.InternalOneOfEnum {

which seem to be a protobuf version issue.
So, anyone had any luck with firrtl on M1 or have any suggestions to try?
Best regards,
Sam

​

[Kamyar Mohajerani]

Hi Sam,

I've been building firrtl (master) on macOS/arm64 using homebrew native protoc and the same settings you have there with no issues. compile, test, and publishLocal work perfectly fine with no issues

I suspect the error could be coming from a previously generated proto Java files with an incompatible protoc version. Have you tried `git clean -fdx` before starting a fresh build?

My protoc version is 3.17.3. I'm also building for scala 2.13, but I really don't think that's making a difference.

Best,

Kamyar

[sam...@gmail.com]

Hi Kamyar,

Thanks for the reply, good news at least it’s working for someone. Still I can’t get it to work.
I did try cleaning firrtl with sbt clean before doing publishLocal on rocket-dsp-utils, plus it’s a newly cloned repo so probably it’s not cache problem.

Do you mind to clarify, by “same settings” do you mean commenting out the protobufRunProc com.github.os72.protocjar.Protoc.runProtoc("-v351" +: args.toArray) line, and setting
protobufProtoc in ProtobufConfig := "/opt/homebrew/bin/protoc" ?
I’m on Chipyard 1.5.0, therefore firrtl 7756f8f dated 18 Dec 2020.
Can you also confirm in (or just under firrtl/target)

chipyard/tools/firrtl/target/scala-2.12/src_managed/main/compiled_protobuf/firrtl/FirrtlProtos.java

that InternalOneOfEnum is referenced as member of AbstractMessage, or AbstractMessageLite? See below.

After a bit of searching, one possible cause seem to be mismatched version between protobuf and protoc:
https://stackoverflow.com/questions/58537310/grpc-maven-unable-to-use-oneof-symbol-internaloneofenum-missing
https://github.com/grpc/grpc-java/issues/6306
Yet there seem to be an open ticket in protobuf about “InternalOneOfEnum should reference AbstractMessageLite instead of AbstractMessage”:
https://github.com/protocolbuffers/protobuf/issues/7373
For scalaPB everything works just by setting protocVersion to 3.17.3:
https://github.com/scalapb/ScalaPB/issues/1024

Best regards,
Sam

​