Slow build times even for tiny examples

[Steve Burns]

I'm new to firesim and I have been trying to run the flow to generate simulators using VCS and Verilator.

It takes around 10 minutes to build up a simulator, even for some of the simple examples in the "midasexamples" directories. (I trying to make small modifications to the GCD example.)

Do you have any recommendations on how I can speed up this up? I'm mostly iterating the testbench .cpp and want a reasonably fast compile execute loop.

A lot of time is spent resolving key references (several times.)

(ash) bash-4.4$ make TARGET_PROJECT=midasexamples SCALA_TEST='firesim.midasexamples.GCDJsonF1Test' testOnly

cd /nfs/site/disks/scl.work.50/ash/users/smburns/firesim/sim && java -Xmx16G -Dsbt.supershell=false -jar /nfs/site/disks/scl.work.50/ash/users/smburns/firesim/target-design/chipyard/generators/rocket-chip/sbt-launch.jar "project firesim" "testOnly firesim.midasexamples.GCDJsonF1Test"

[info] Loading settings for project sim-build from plugins.sbt ...

[info] Loading project definition from /nfs/site/disks/scl.work.50/ash/users/smburns/firesim/sim/project

[info] Loading settings for project firesim from build.sbt ...

[info] Loading settings for project midas from build.sbt ...

[info] Loading settings for project targetutils from build.sbt ...

[info] Loading settings for project chipyard-build from plugins.sbt ...

[info] Loading project definition from /nfs/site/disks/scl.work.50/ash/users/smburns/firesim/target-design/chipyard/project

[info] Loading settings for project chipyardRoot from build.sbt ...

[info] Loading settings for project barstoolsMacros from build.sbt ...

[info] Loading settings for project mdf from build.sbt ...

[info] Loading settings for project gemmini from build.sbt ...

[info] Loading settings for project ariane from build.sbt ...

[info] Loading settings for project boom from build.sbt ...

[info] Loading settings for project hwacha from build.sbt ...

[info] Loading settings for project icenet from build.sbt ...

[info] Loading settings for project testchipip from build.sbt ...

[info] Loading settings for project rocketConfig from build.sbt ...

[info] Loading settings for project hardfloat from build.sbt ...

[info] Loading settings for project chisel_testers from build.sbt ...

[info] Loading settings for project treadle from build.sbt ...

[info] Loading settings for project firrtl_interpreter from build.sbt ...

[info] Loading settings for project chisel from build.sbt ...

[info] Resolving key references (34447 settings) ...

...

Has anyone tried to extract the "midasexamples" section completely out of the chipyard infrastructure? In the work I'm trying to do I won't need Rocket or BOOM or GEMMINI or ariane or hwacha. I want to build a simulator for new code written in Chisel with some Verilog black boxes.

[David Biancolin]

Is SBT key resolution the bulk of the aforementioned 10 minutes? Most of the midasexamples can proceed through elaboration -> gg compilation -> verilator -> simulation in under a minute using bloop on the dev branch (on a typical manager instance). Using SBT it should still be considerably quicker than what you're reporting. 

I'm not aware of anyone trying to remove BOOM/Ariane. Generally once these have compiled once you can forget about them unless you're hacking on chisel/firrtl/RC/midas.targetutils. If you want, you could define a new sbt project that doesn't depend on firechip, to avoid compiling all of the other chipyard IP.    Removing the Rocket Chip dependency is not possible since FireSim relies extensively on RC RTL libraries (e.g., diplomacy, util).

In principle it is possible to submodule FireSim into some other top-level project, but in practise this takes some hacking on the build system.  The high-level idea is that you'd have to provide your own target-specific make fragment, which would call your generator (as opposed to CY's). 

[David Biancolin]

Oh, if your NFS is slow that's a very typical culprit of bad SBT performance. 

[Steve Burns]

Thanks. I was able to create a shorter debug loop by just running make on Vverilator_top.mk and then running the ./VGCDJson executable using the command lines spat out by a complete top level make run.

I'll try copying the area to a temporary drive so that I can avoid the NFS delays and let you know how things improve. I'll also run it on a linux laptop with an SSD.

I'll also look into my own target-specific make fragment.

[Steve Burns]

Here are the results of testing on different machines.

5x slower on our servers if we run it over NFS.

Engineering Computing Servers

Temp drive

Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz

[success] Total time: 112 s (01:52), completed Oct 22, 2020 9:03:03 AM

real    2m10.109s
user    5m1.215s
sys     0m34.567s

Engineering Computing Servers

NFS drive

Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz

[success] Total time: 486 s (08:06), completed Oct 22, 2020 9:31:11 AM

real    11m3.167s
user    5m31.211s
sys     2m21.858s

Linux laptop with SSD
Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz

[success] Total time: 139 s (02:19), completed Oct 22, 2020, 9:14:41 AM

real	2m49.347s
user	9m50.667s
sys	0m31.721s

Just the testbench compile:

real    0m3.944s
user    0m3.280s
sys     0m0.469s

Just the simulator execution:

real    0m0.037s
user    0m0.021s
sys     0m0.007s

It would be nice if the Makefile system could be incremental enough so that if you just change the test bench, it could only do the 4 second compile instead of the 2 minute (or 11 minute) build from scratch.

--
You received this message because you are subscribed to a topic in the Google Groups "FireSim" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firesim/ewqYWLKylBo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firesim+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firesim/574f0840-52a3-4128-9845-18383a7feeb5n%40googlegroups.com.

[David Biancolin]

When you say testbench what are you referring to? C++ that constitutes the driver? 

The make system should support recompilation of only the simulator binary without having to go through elaboration. sbt test and sbt testOnly` by default clean out generated-src. The intermediate results are thus removed and you have to start from scratch. If you want to disable that you can comment out this line:
https://github.com/firesim/firesim/blob/master/sim/src/test/scala/midasexamples/TutorialSuite.scala#L90

If i'm iterating on C++ changes i generally rely on invoking the make run-* recipe directly.  The ScalaTest will print out the make commands it's running, you can just copy those. 

Concretely I'd suggest the following:
1) Invoke the make command for your designs directly. Do this from sim. You should not need to target a make fragment directly.
2) To speed up the flow through scalatest, do following.
  -  If you're on dev just use bloop.  It's blazing fast. Set ENABLE_BLOOP=1 in your env. 

  -  If you're on a release / master, put SBT caches on fast local storage. You'll need to the following to your JVM:

   -Dsbt.ivy.home=<your-local-fs>/.ivy2
   -Dsbt.global.base=<your-local-fs>/.sbt
   -Dsbt.boot.directory=<your-local-fs>/.sbt/boot/  

I'm lazy and do this using the _JAVA_OPTIONS env variable, but this could be dangerous if you're running the JVM for other scala applications that want to see the original locations for some reason. Additionally you'll want to tell coursier to use the same fast local store. On something sh like:
export COURSIER_CACHE=<your-local-fs>/.coursier-cache   

should do the job.