Network Connection on RocketChip

[Jan T.]

Hi!

I have some questions about the networking capabilities of FireSim. There is not much information available so I am not sure if I am doing it right.

I am running FireSim on a local Alveo U250 FPGA. Is it possible to connect the simulated RocketChip a) to a virtual network with other cores simulated on the same FPGA and / or b) to the external network (i.e., ethernet of the host PC)? If it is possible to connect to ethernet, do I use the SFP ports of the FPGA or is the network traffic managed over the PCI bus?

I have tried to configure a RocketChip workload with a NIC. The target configuration looks like this:

class FireSimRocketConfigWithIceNIC extends Config(

  new WithNIC ++

  new FireSimRocketConfig

)

However, when I build the bitstream and then execute firesim infrasetup followed by firesim runworkload, the simulation seems to hang. This is the complete output of the simulation in the screen window:

Script started, output log file is 'uartlog'.
Using: 0000:01:00.0, BAR ID: 0, PCI Vendor ID: 0x10ee, PCI Device ID: 0x903f
Opening /sys/bus/pci/devices/0000:01:00.0/vendor
Opening /sys/bus/pci/devices/0000:01:00.0/device
examining xdma/.
examining xdma/..
examining xdma/xdma0_h2c_0
Using xdma write queue: /dev/xdma0_h2c_0
Using xdma read queue: /dev/xdma0_c2h_0
command line for program 0. argc=29:
+permissive +macaddr0=00:12:6D:00:00:02 +blkdev0=linux-uniform0-br-base.img +niclog0=niclog0 +blkdev-log0=blkdev-log0 +trace-select=1 +trace-start=0 +trace-end=-1 +trace-output-format=0 +dwarf-file-name=linux-uniform0-br-base-bin-dwarf +autocounter-readrate=0 +autocounter-filename-base=AUTOCOUNTERFILE +drj_dtb=linux-uniform0-br-base-bin.dtb +drj_bin=linux-uniform0-br-base-bin +drj_rom=linux-uniform0-br-base-bin.rom +print-start=0 +print-end=-1 +linklatency0=6405 +netbw0=200 +shmemportname0=default +domain=0x0000 +bus=0x01 +device=0x00 +function=0x0 +bar=0x0 +pci-vendor=0x10ee +pci-device=0x903f +permissive-off linux-uniform0-br-base-bin
UART0 is here (stdin/stdout).
using link latency: 6405 cycles
using netbw: 200
using netburst: 8
Using non-slot-id associated shmemportname:
opening/creating shmem region
/port_ntsdefault_0
Using non-slot-id associated shmemportname:
opening/creating shmem region
/port_stndefault_0
Using non-slot-id associated shmemportname:
opening/creating shmem region
/port_ntsdefault_1
Using non-slot-id associated shmemportname:
opening/creating shmem region
/port_stndefault_1
TraceRV 0: Tracing disabled, since +tracefile was not provided.
FireSim fingerprint: 0x46697265
TracerV: Trigger enabled from 0 to 18446744073709551615 cycles
Commencing simulation.

Has someone successfully executed a simulation with NIC and can help me out?

Best regards,

Jan

[Junho Yeom]

if you figure out how to fix, please share. I'm stuck like you

[Junho Yeom]

check below link
https://groups.google.com/g/firesim/c/qCPLK9xbs0U

[Amirmohammad Nazari]

Hi,

I am also facing the exact same issue with AU250 and haven't been able to solve it by changing the verilator path. My simulations stucks after showing 

Has anyone found another solution for this problem?

Here is the output of my uartlog: 

$ cat uartlog
Script started on 2025-06-30 17:09:12-04:00 [TERM="screen" TTY="/dev/pts/7" COLUMNS="80" LINES="24"]
+domain found: 0x0000
+bus found: 0xaf
+device found: 0x00
+function found: 0x0
+bar found: 0x0
Using: 0000:af:00.0, BAR ID: 0, PCI Vendor ID: 0x10ee, PCI Device ID: 0x903f
Opening /sys/bus/pci/devices/0000:af:00.0/vendor
Opening /sys/bus/pci/devices/0000:af:00.0/device

examining xdma/.
examining xdma/..
examining xdma/xdma0_h2c_0
Using xdma write queue: /dev/xdma0_h2c_0
Using xdma read queue: /dev/xdma0_c2h_0

widget_registry_t::add_widget(StreamEngine)
cpu2fpga: 1, fpga2cpu: 2

UART0 is here (stdin/stdout).
using link latency: 6405 cycles
using netbw: 200
using netburst: 8
Using non-slot-id associated shmemportname:
opening/creating shmem region
/port_ntsdefault_0
Using non-slot-id associated shmemportname:
opening/creating shmem region
/port_stndefault_0
Using non-slot-id associated shmemportname:
opening/creating shmem region
/port_ntsdefault_1
Using non-slot-id associated shmemportname:
opening/creating shmem region
/port_stndefault_1

BUFBYTES 58560

TraceRV 0: Tracing disabled, since +tracefile was not provided.

command line for program 0. argc=26:
+permissive +macaddr0=00:12:6D:00:00:02 +blkdev0=br-base0-br-base.img +niclog0=niclog0 +blkdev-log0=blkdev-log0 +trace-select=1 +trace-start=0 +trace-end=-1 +trace-output-format=0 +dwarf-file-name=br-base0-br-base-bin-dwarf +autocounter-readrate=0 +autocounter-filename-base=AUTOCOUNTERFILE +print-start=0 +print-end=-1 +linklatency0=6405 +netbw0=200 +shmemportname0=default +domain=0x0000 +bus=0xaf +device=0x00 +function=0x0 +bar=0x0 +pci-vendor=0x10ee +pci-device=0x903f +permissive-off br-base0-br-base-bin

FireSim fingerprint: 0x46697265
TracerV: Trigger enabled from 0 to 18446744073709551615 cycles
Commencing simulation.

Best,

Amirmohammad

[Muhammad Ali Akhtar]

I am running FireSim on a local Alveo U250 FPGA. Is it possible to connect the simulated RocketChip a) to a virtual network with other cores simulated on the same FPGA and / 

I think there is a SuperNode Configuration that can help you do that, but I am not sure. 

 b) to the external network (i.e., ethernet of the host PC)? If it is possible to connect to ethernet, do I use the SFP ports of the FPGA or is the network traffic managed over the PCI bus?

You can use the ethernet of the host PC. The simulated Switch in firesim is running on the Host. This can either be ToR Switch or Root Switch. If you have multiple Alveo Cards in the same Host, you won't need the Host ethernet interface to provide network connection between simulated RocketChips. The ToR switch (simulated switch) running on the Host will connect all the Rocket-Chips running on different Alveo FPGAs on the same host. In this case, the data communication is through PCIe bus and shared memory ports. 

If you have multiple hosts, each with multiple FPGAs (Alveos), the ToR (or Root) switch can use the host's ethernet interface. However, I am not sure if it works for On-Prem FPGAs. In AWS, all the hosts are accessible with their public / private IPs and Simulated Switch on Each host uses Host's ethernet connectivity to communicate. 

Muhammad Akhtar

--
You received this message because you are subscribed to the Google Groups "FireSim" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firesim+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/firesim/1f9c514b-4bc2-450b-8422-2f0b6d24878an%40googlegroups.com.

[Manchem Chandana Sai Sri]

Hi Amir,

        I observed this problem when the bitstream built is not compatible with the current code base you have in firesim directory, it might be a design file change or driver file change. I suggest you build the bitstream freshly, completely turn off the fpga, switch it on, reboot the farm machine and try again, In my case, doing this removed the above simulation stuck problem. I did the mentioned steps as it was mentioned in the firesim documentation that the hang in firesim simulation might be something to do with the drivers expecting something and not getting it. But making sure the codebase is compatible with the bitstream you are using for firesim simulation helped me. 
        The issue I had was I had 2 implementations of tracer bridge module and 2 tracerv driver versions that match one of the bridge implementation, when I dont make sure they are compatible ( the infrasetup recompiles your tracerv driver file, so you should actually make sure your bridge in your soc(emulated design in the fpga) is giving what is expected from driver) the simulation hangs.

        Please feel free to ask if you have any doubts in my above explanation

https://docs.fires.im/en/latest/Advanced-Usage/Debugging-and-Profiling-on-FPGA/Debugging-Hanging-Simulators.html#case-3-simulator-hang-due-to-driver-side-deadlock

Thanks & Regards
Chandana Manchem

--

[Manchem Chandana Sai Sri]

Hi Amir,
      You can do your sanity checks with your design using verilator simulator in chipyard, make sure all the basic riscv tests are passing. And also make sure the riscv tests are also working on firesim simulation. Can help with zeroing in on the problem, if the chipyard simulation itself is stuck then it might be the to host address is not written with the expected value, the fesvr keeps on reading the 'to host' address until it gets the expected value ig. 

Thanks & Regards
Chandana Manchem