Using Verilator on compute instances instead of F1 instances

[Theodore Jepsen]

Hi Everyone,

Before running large scale simulations with FireSim on F1 instances, we've been running scaled-down simulations on a single compute optimized (C5) instance. We have a script that sets up a topology with a single switch connected to some rocket cores running in Verilator:

https://github.com/l-nic/chipyard/tree/lnic-dev/software/local_firesim

We would like to scale-out our simulation, using thousands of rocket cores. However, this would require a lot of F1 instances! Since the workload in our simulation is very small, we don't get a huge speed-up from using the FPGAs on the F1 instances; it would be fine to run them through Verilator instead. This would be more cost effective and easier to reserve (it's difficult to find enough available F1 instances).

Do you know whether FireSim can run in "Verilator mode"? We would like to use the existing FireSim scripts, but instead of running the FPGA image, use our Verilator binary? Is this possible?

If you have done something like this, or have suggestions for changes necessary to FireSim to enable this, that would be great!

Theo

[Muhammad Akhtar]

My information is slightly outdated. so some things may have changed / updated in latest firesim release.

Firesim supports midas level (midas was later replaced by golden gate) verilator simulations. which means you have your target RTL, your widgets (bridges) and your widget drivers. 
However, one limitation was that you can not use ICE-NIC (and switch) in verilator simulations because ICE--NIC uses PCIS-DMA on F1 instances. You can modify the ICE--NIC widget / driver to avoid using PCIS-DMA and than you should be able to do verilator simualtions of complete firesim infrastructure. 

So the short answer:
Verilator simulations are possible with NO-NIC Configuration.
Some modifications are needed to do verilator simulations with NIC Configuration

[Theodore Jepsen]

Thank you, Muhammad, for the your response and suggestions.

Since my original post, I have been working on "Verilator mode". The main changes to FireSim are:

• Adding a new instance type (c5.4xlarge) for running Verilator
• Modifying classes to detect whether to use FPGA or Verilator (e.g. `self.use_verilator`)
• Copying the Verilator infrastructure (the sim binary, `VFireSim`) to the remote instances
• Running the sim binary, passing it the required arguments
  

There are still some hard coded paths and assumptions about the environment. It has to be made more generic. I'm not sure how to:

• Configure FireSim to use Verilator instead of the FPGA. Should this be in a parameter in `config.ini` or a topology class in `user_topology.py`?
• Specify the instance type to use for Verilator. Currently, FireSim knows that it should allocate FPGA instance slots to any available F1 instance types. FPGAs are only available some types of instances. Verilator, on the other hand, can run on any instance type. How should FireSim detect which instances should be used for Verilator?
• Specify path to Verilator sim binary. I have hard coded the path to `/home/centos/chipyard/sims/firesim/sim/generated-src/f1/FireSim-DDR3..._BaseF1Config/VFireSim`. Where would this be configured, in the `config.ini` possibly?
  

If you have any ideas about the above, that would be great. After I address these issues, I can tidy-up and share the code.

[Sagar Karandikar]

Hi Theo,

I'd really like this feature to get upstreamed! Glad to see you've already made a bunch of progress. Here are my thoughts on your questions:

--

> - Configure FireSim to use Verilator instead of the FPGA. Should this be in a parameter in `config.ini` or a topology class in `user_topology.py`?

I would say add some parameters in config_runtime.ini. E.g. add:

[software-metasimulation-mode]

# enable software metasimulation mode

software_metasimulation_only={no|yes}

# let user specify a host instance type

metasimulation_host_instance_type=c5.4xlarge

# launch N of these instances

metasimulation_host_instance_count=1

# map up to 10 simulations per instance

metasimulation_host_max_sims_per_host=10

# add other metasim-specific params here

When using metasimulation mode in the manager, defaulthwconfig should refer to an entry from config_build_recipes.ini rather than config_hwdb.ini.

> - Specify the instance type to use for Verilator. Currently, FireSim knows that it should allocate FPGA instance slots to any available F1 instance types. FPGAs are only available some types of instances. Verilator, on the other hand, can run on any instance type. How should FireSim detect which instances should be used for Verilator?

I think we can leave this open-ended for now (see above) and give a reasonable default, e.g. a c5.4xlarge.

> - Specify path to Verilator sim binary. I have hard coded the path to `/home/centos/chipyard/sims/firesim/sim/generated-src/f1/FireSim-DDR3..._BaseF1Config/VFireSim`. Where would this be configured, in the `config.ini` possibly?

We should be able to re-use the infrastructure the manager uses to figure out where the software driver comes from for real fpga-based firesim sims. The path can be determined using the config triplet you get from the config_build_recipe.ini that defaulthwconfig points to.

--

If you don't have the bandwidth to work on this now, I'd be happy to take care of finishing things up and upstreaming your code if you share it (privately with me is fine too). It'll also probably be easier to work out the exact user API once there's a public PR.

Thanks!

Sagar

[Theo Jepsen]

Hi Sagar,

In the end I was able to run a large-scale simulation with 16K verilator
servers using 264 c5.4xlarge instances! I'd love for this feature to be
upstreamed!

Here are the changes that made this possible:
https://github.com/l-nic/firesim/commit/91bf11500af2adc719661b1dde61b62f2fa51a67

This commit is on a branch based off an older version of FireSim (1.9.0
release, I think).

I had problems threading the config parameters through as you suggested.
The config has a flag to enable verilator mode:
https://github.com/l-nic/firesim/commit/91bf11500af2adc719661b1dde61b62f2fa51a67#diff-078ab4f0368b3f5f321fa1caf7084def9a34dce0f1cbfddc665dad099eaf0fb5R302

But the verilator binary is still hard-coded:
https://github.com/l-nic/firesim/commit/91bf11500af2adc719661b1dde61b62f2fa51a67#diff-c39a55a1bbe29bd68726f3e64bdc29f472c2f61c0b7cf91c361013926b375241R650

There are also some non-essential changes, like building the switch
binaries in parallel (this really speeds things up when it has to build
264 switches!):
https://github.com/l-nic/firesim/commit/91bf11500af2adc719661b1dde61b62f2fa51a67#diff-1b79ea65185acfd3d9642fd67d78f2e2e16a25af2bc673aff8c3db23475e7875R409

I don't have the bandwidth right now to clean this up for a PR. Also, I
may not be the best person to manage this, especially if there were
significant upstream changes.

If you have any questions about the code, just ask me.

Theo

> > - Adding a new instance type (c5.4xlarge) for running Verilator
> > - Modifying classes to detect whether to use FPGA or Verilator (e.g.
> > `self.use_verilator`)
> > - Copying the Verilator infrastructure (the sim binary, `VFireSim`) to
> > the remote instances
> > - Running the sim binary, passing it the required arguments

> >
> > There are still some hard coded paths and assumptions about the
> > environment. It has to be made more generic. I'm not sure how to:
> >

> > - Configure FireSim to use Verilator instead of the FPGA. Should this
> > be in a parameter in `config.ini` or a topology class in `user_topology.py`?

> > - Specify the instance type to use for Verilator. Currently, FireSim
> > knows that it should allocate FPGA instance slots to any available F1
> > instance types. FPGAs are only available some types of instances.
> > Verilator, on the other hand, can run on any instance type. How should
> > FireSim detect which instances should be used for Verilator?

> > - Specify path to Verilator sim binary. I have hard coded the path to
> > `/home/centos/chipyard/sims/firesim/sim/generated-src/f1/FireSim-DDR3..._BaseF1Config/VFireSim`.
> > Where would this be configured, in the `config.ini` possibly?
> >

> --
> You received this message because you are subscribed to a topic in the Google Groups "FireSim" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/firesim/jETA4sAc0pE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to firesim+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/firesim/51638a26-e834-47dc-9d97-b5812a0d7250n%40googlegroups.com.