Segmentation Fault when new Tilelink module is integrated to IceNIC

[Sabra Ossen]

Hi All,

I'm facing a weird problem in my design. I have two components that can be connected via tile link, namely a tcam (tile link manager) and a client querying said tcam.

This is the config I am using to test both of those components independently and works as expected - the client queries the tcam for a specific value. Here, I use the rocket core to populate the tcam.

class TCAMTLRocketConfig extends Config(
    new icenet.WithTCAMClient ++
    new icenet.WithTCAM ++
    new freechips.rocketchip.subsystem.WithNBigCores(1) ++
    new chipyard.config.AbstractConfig
)

I have integrated my tcam client to IceNIC, and therefore query the tcam for every incoming parsed packet. This logic works when I test the IceNICRecvPath unit test. However, I am getting a segmentation fault when I run this as a config below. To connect nic IOs I use the nic-loopback test logic. The config can be seen below.

class LoopbackNICRocketTCAMConfig extends Config (
  new chipyard.harness.WithLoopbackNIC ++ // drive NIC IOs with loopback
  new icenet.WithTCAM ++
  new icenet.WithIceNIC ++
  new freechips.rocketchip.subsystem.WithNBigCores(1) ++
  new chipyard.config.AbstractConfig
)

Here, the tcam client is integrated into the nic as shown by code snippets below, to provide a brief idea.

class IceNicRecvPath(val tapFuncs: Seq[EthernetHeader => Bool] = Nil)
    (implicit p: Parameters) extends LazyModule {

  val tcamClientConfig = new TCAMCLConfig(
      lookupAddress = 0x10030000,
      lookupMask = 0xfff)

  val writer = LazyModule(new IceNicWriter)
  val tcamReader = LazyModule(new TCAMClient(tcamClientConfig))
  val node = TLIdentityNode()
  val rdnode = TLIdentityNode()
  node := writer.node
  rdnode := tcamReader.node
  lazy val module = new IceNicRecvPathModule(this)
}

....

trait CanHavePeripheryIceNIC  { this: BaseSubsystem =>

....

    manager.coupleTo(portName) { icenic.mmionode := TLFragmenter(manager.beatBytes, manager.blockBytes) := _ }
    client.coupleFrom(portName) { _ :=* icenic.dmanode }
    ibus.fromSync := icenic.intnode
    client.coupleFrom("tcam_client") { _ := icenic.rdnode }

...
 }

And this is the segmentation fault that occurs.

(/home/sossen/Git/chipyard/.conda-env) sossen@test:~/Git/chipyard/sims/verilator$ /home/sossen/Git/chipyard/sims/verilator/simulator-chipyard.harness-LoopbackNICRocketTCAMConfig +permissive +dramsim +dramsim_ini_dir=/home/sossen/Git/chipyard/generators/testchipip/src/main/resources/dramsim2_ini +max-cycles=10000000     +verbose +permissive-off ../../tests/nic-loopback.riscv
%Warning: System has stack size 8192 kb which may be too small; suggest 'ulimit -c 448324' or larger
testing $random e23cbb39
Segmentation fault (core dumped)

I believe this is a seg fault occurring in verilator vs a seg fault occurring in the baremetal code running on the rocket core, because even when I comment out all the logic except the main function call it still seg faults. Valgrind also provides the trace below.

sossen@test$ valgrind --tool=memcheck --track-origins=yes /home/sossen/Git/PhD/PhD/chipyard/sims/verilator/simulator-chipyard.harness-LoopbackNICRocketTCAMConfig +permissive +dramsim +dramsim_ini_dir=/home/sossen/Git/PhD/PhD/chipyard/generators/testchipip/src/main/resources/dramsim2_ini +max-cycles=10000000     +verbose +permissive-off ../../tests/nic-loopback.riscv

==1073102== Process terminating with default action of signal 11 (SIGSEGV): dumping core

==1073102==  Access not within mapped region at address 0x1FFE7FD9C0
==1073102==    at 0x3B2711: VTestDriver___024root___eval_initial__TOP__197(VTestDriver___024root*) (in /home/sossen/Git/chipyard/sims/verilator/simulator-chipyard.harness-LoopbackNICRocketTCAMConfig)
==1073102==  If you believe this happened as a result of a stack
==1073102==  overflow in your program's main thread (unlikely but
==1073102==  possible), you can try to increase the size of the
==1073102==  main thread stack using the --main-stacksize= flag.
==1073102==  The main thread stack size used in this run was 8388608.

I realize this still requires more information for further debugging this problem, so please ask me and I can share my code if needed.

Thank you very much your help in advance to help debug this problem.

Regards,

Sabra

[Sabra Ossen]

I figured it out.

I was mistakenly connecting my tcam client (tile link client) to the FBUS, instead of the PBUS where the actual tcam (tile link manager) is connected to.

trait CanHavePeripheryIceNIC  { this: BaseSubsystem =>

....

    manager.coupleTo(portName) { icenic.mmionode := TLFragmenter(manager.beatBytes, manager.blockBytes) := _ }
    client.coupleFrom(portName) { _ :=* icenic.dmanode }
    ibus.fromSync := icenic.intnode
    client.coupleFrom("tcam_client") { _ := icenic.rdnode }

...
 }

should be,

trait CanHavePeripheryIceNIC  { this: BaseSubsystem =>

....

    manager.coupleTo(portName) { icenic.mmionode := TLFragmenter(manager.beatBytes, manager.blockBytes) := _ }
    client.coupleFrom(portName) { _ :=* icenic.dmanode }
    ibus.fromSync := icenic.intnode

    manager.coupleFrom("tcam_client") { _ := icenic.rdnode }

...
 }