exceeds JVM code size limits

[Zhan Shi]

Hi there,

When I'm trying to make the generated resnet50opt, it throws an error below. I do find the x16826_inr_Foreach_kernel.scala file has a lot more code than others. Any idea how to work this around?

echo "$(date +%s)" > start.log
sbt compile
Picked up _JAVA_OPTIONS: -Xms1024m -Xss256m -Xmx16g -XX:MaxMetaspaceSize=30g
Copying runtime jar.
Picked up _JAVA_OPTIONS: -Xms1024m -Xss256m -Xmx16g -XX:MaxMetaspaceSize=30g
Exception in thread "main" java.nio.file.ProviderNotFoundException: Provider "jrt" not found
        at java.nio.file.FileSystems.getFileSystem(FileSystems.java:224)
        at io.github.retronym.java9rtexport.Export.main(Export.java:36)
Picked up _JAVA_OPTIONS: -Xms1024m -Xss256m -Xmx16g -XX:MaxMetaspaceSize=30g
[info] Loading project definition from /home/aleczshi/spatial-multiverse/spatial/gen/resnet50opt/project
[info] Loading settings from build.sbt ...
[info] Set current project to spatial-app (in build file:/home/aleczshi/spatial-multiverse/spatial/gen/resnet50opt/)
[info] Executing in batch mode. For better performance use sbt's shell
[info] Compiling 363 Scala sources to /home/aleczshi/spatial-multiverse/spatial/gen/resnet50opt/target/scala-2.12/classes ...
[error] Could not write class x16826_inr_Foreach_kernel$Block1Chunker29$2$ because it exceeds JVM code size limits. Method emul/FixFormat's code too large!
[error] one error found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 28 s, completed Jul 10, 2020 4:59:40 AM
Makefile:2: recipe for target 'all' failed
make: *** [all] Error 1

And I do find that the file x16826_inr_Foreach_kernel.scala has a lot more code than others. Any idea how to 

[Matt F]

Can you try compiling the spatial app with the flag --codeWindow=40 (the default is 50, but setting it to 20, 30, or 40 could fix this)?  It seems like something pathological could be happening here, so if this doesn't work, would you be able to email me the app or post it here so I can take a look?

[Zhan Shi]

Hi Matt,

I'm using the ResNet example provided in https://github.com/stanford-ppl/spatial-multiverse/tree/master/models/resnet

Are you suggesting that I should compile with 

bin/spatial resnet50opt --sim --codeWindow=40

Thanks!

[Matt F]

This issue should be independent of the _JAVA_OPTIONS and OS so I was trying to reproduce it but I haven't been able to yet.  Can you point out where I did something different from you?

git clone https://github.com/stanford-ppl/spatial-multiverse.git
cd spatial-multiverse
git submodule update --init
python dnn_to_spatial.py models/resnet/ResNet50_opt.pb
cp resnet50opt.scala spatial/test/spatial/tests/apps
cd spatial
bin/spatial resnet50opt --synth --forceFuseFMA --noBindParallels --fpga=AWS_F1 && cd gen/resnet50opt && make aws-F1

When I compile the Spatial app, there are only 293 Scala sources, but it looks like you have 363.

[Zhan Shi]

Hi Matt,

I really appreciate you trying to reproduce the problem. As I haven't set up AWS yet, I was trying to use the Scala simulator as the backend with the following commands:

git clone https://github.com/stanford-ppl/spatial-multiverse.git
cd spatial-multiverse
git submodule update --

init
python2 dnn_to_spatial.py models/resnet/ResNet50_opt.pb
cp resnet50opt.scala spatial/test/spatial/tests/apps
cd spatial
bin/spatial resnet50opt --sim
cd gen/resnet50opt && make

And I got the code size error at the last make step. 

If I followed your commands,

python dnn_to_spatial.py models/resnet/ResNet50_opt.pb
cp resnet50opt.scala spatial/test/spatial/tests/apps
cd spatial
bin/spatial resnet50opt --synth --forceFuseFMA --noBindParallels --fpga=AWS_F1 && cd gen/resnet50opt && make aws-F1

I got this error instead:

[ERROR] Failed to construct terminal; falling back to unsupported java.lang.NumberFormatException: For input string: "0x100" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:580) at java.lang.Integer.valueOf(Integer.java:766) at jline.internal.InfoCmp.parseInfoCmp(InfoCmp.java:59) at jline.UnixTerminal.parseInfoCmp(UnixTerminal.java:242) at jline.UnixTerminal.<init>(UnixTerminal.java:65) at jline.UnixTerminal.<init>(UnixTerminal.java:50) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at jline.TerminalFactory.getFlavor(TerminalFactory.java:211) at jline.TerminalFactory.create(TerminalFactory.java:102) at jline.TerminalFactory.get(TerminalFactory.java:186) at jline.TerminalFactory.get(TerminalFactory.java:192) at sbt.JLine$.sbt$JLine$$terminal(LineReader.scala:85) at sbt.JLine$.withTerminal(LineReader.scala:88) at sbt.JLine$.usingTerminal(LineReader.scala:96) at sbt.JLine$.createReader(LineReader.scala:102) at sbt.FullReader.<init>(LineReader.scala:132) at sbt.BasicCommands$$anonfun$shell$1.apply(BasicCommands.scala:184) at sbt.BasicCommands$$anonfun$shell$1.apply(BasicCommands.scala:181) at sbt.Command$$anonfun$command$1$$anonfun$apply$1.apply(Command.scala:30) at sbt.Command$$anonfun$command$1$$anonfun$apply$1.apply(Command.scala:30) at sbt.Command$.process(Command.scala:93) at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:96) at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:96) at sbt.State$$anon$1.doX$1(State.scala:183) at sbt.State$$anon$1.process(State.scala:189) at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:96) at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:96) at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17) at sbt.MainLoop$.next(MainLoop.scala:96) at sbt.MainLoop$.run(MainLoop.scala:89) at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:68) at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:63) at sbt.Using.apply(Using.scala:24) at sbt.MainLoop$.runWithNewLog(MainLoop.scala:63)

To take a step back, should I go ahead set up AWS or it's OK to stay with the Scala simulation?

Thanks!

[Matt Feldman]

Hi.  I played around a little with the scala codegen but I don't think there is going to be an easy way to get ResNet50_opt to work in scala sim because of quirks with the jvm and spatial IR. There is nothing fundamentally wrong with that network, it just happens that it needs to do so much book keeping with memory accesses that it blows up the jvm limit just by the way spatial codegen works right now.  Would it be possible to change the network to something a little smaller if you want to use scala sim?  I noticed that some of the newer experimental codegen flags are not on the commit of Spatial that multiverse points to, so you would have to get the latest develop branch of spatial if you wanted to pursue that further (I don't recommend it though since I can't guarantee the multiverse flow will work perfectly on this branch, although I think it should be ok).

If you want to debug ResNet50_opt, using aws to simulate the --synth backend may be the better route.  I don't recognize the error or any of these files in the stack trace you posted.  Can you show the entire printout from the command you ran onward?  I suspect this will be an easy fix but I'm not sure what is causing it yet.