Hammer fails on PAR using OpenROAD + sky130 tutorial

[heavy hussar]

I keep getting this error that's thrown by the OpenROAD + Sky130 PAR step every time I try running it in console. I don't know what the issue is or how to fix it. From what I can tell, it's some vector size check that fails during the PAR step somewhere in the process. This is probably happening due to user error. I don't know what exactly I'm doing with this.

Here is the detailed error log.

Using 2 tracks default min distance between IO pins.

/usr/include/c++/9/bits/stl_vector.h:1042: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) [with _Tp = int; _Alloc = std::allocator<int>; std::vector<_Tp, _Alloc>::reference = int&; std::vector<_Tp, _Alloc>::size_type = long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
Stack trace:
 0# 0x000055D5E3883D51 in openroad
 1# 0x00007EFD56C2A090 in /lib/x86_64-linux-gnu/libc.so.6
 2# gsignal in /lib/x86_64-linux-gnu/libc.so.6
 3# abort in /lib/x86_64-linux-gnu/libc.so.6
 4# 0x000055D5E3883E18 in openroad
 5# 0x000055D5E3945E63 in openroad
 6# 0x000055D5E3A09FE1 in openroad
 7# ppl::IOPlacer::run(bool) in openroad
 8# 0x000055D5E3A02DB7 in openroad
 9# TclNRRunCallbacks in /lib/x86_64-linux-gnu/libtcl8.6.so
10# 0x00007EFD58917924 in /lib/x86_64-linux-gnu/libtcl8.6.so
11# Tcl_EvalEx in /lib/x86_64-linux-gnu/libtcl8.6.so
12# Tcl_Eval in /lib/x86_64-linux-gnu/libtcl8.6.so
13# sta::sourceTclFile(char const*, bool, bool, Tcl_Interp*) in openroad

14# ord::tclAppInit(Tcl_Interp*) in openroad

15# Tcl_MainEx in /lib/x86_64-linux-gnu/libtcl8.6.so

16# main in openroad

17# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6

18# _start in openroad

[heavy hussar]

I'm using Ubuntu 20.04 and OpenROAD-flow-scripts with yosys and OpenROAD to run this tutorial.

[Nayiri Krzysztofowicz]

Hello,

We've never encountered this error before, I wonder if it's just an issue with your OpenROAD install.

If possible, could you send your par.tcl file (located under build/<long-name>/par-rundir/par.tcl), I can at least try to sanity check it.

Additionally, if you send the (zipped) hammer log file from this run I can take a look at that as well.

Thanks,

Nayiri

--
You received this message because you are subscribed to the Google Groups "Chipyard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chipyard+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/cc8785b3-f34f-4f73-b4be-29d94d53df7dn%40googlegroups.com.

[heavy hussar]

Alright. Here are the Hammer logs and par.tcl files from my attempted OpenROAD run. I hope they're of use. By the way, thank you for your help. I hope this doesn't turn into a major issue.

You received this message because you are subscribed to a topic in the Google Groups "Chipyard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/chipyard/pWpRn7mKv6k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chipyard+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/CABNXJp3e5sBRefjCzOE34pF3JFTrJahHq9WSmmpvYpHxtZyApQ%40mail.gmail.com.

[Philipp Schmitz]

Hi everyone,

I am running into exactly the same issue. Is there any update on this?

Best,

 Philipp

[Nayiri Krzysztofowicz]

Hi Philipp,

We weren't able to reproduce this issue on our end. One possible problem might be the OpenROAD version (did you use the conda install or install from source?).

Another possible fix is try setting the metal pin layers in the tutorial to one layer (e.g. layers: ["met4"]) here: https://github.com/ucb-bar/chipyard/blob/main/vlsi/example-sky130.yml#L95

Older OpenROAD versions have segfaulted from having multiple pin layers.

Thanks,

Nayiri

To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/294ffc3e-2050-4478-93fd-d2a48b7b77d8n%40googlegroups.com.

[Philipp Schmitz]

Hi Nayiri,

Thank you for your replay. The recommended conda install did not work on my system since it could not resolve some dependencies.

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package openroad-2.0_7070_g0264023b6-20230225_164303 requires _openmp_mutex >=5.1, but none of the providers can be installed
  - package openroad-2.0_7070_g0264023b6-20230225_164303 requires _openmp_mutex >=5.1, but none of the providers can be installed

Could not solve for environment specs
The following package could not be installed
└─ openroad 2.0_7070_g0264023b6**  is uninstallable because it requires
   └─ _openmp_mutex >=5.1 , which requires
      ├─ _libgcc_mutex 0.1 main, which can be installed;
      └─ libgomp >=7.5.0  but there are no viable options
         ├─ libgomp [10.3.0|10.4.0|...|9.5.0] would require
         │  └─ _libgcc_mutex 0.1 conda_forge, which conflicts with any installable versions previously reported;
         └─ libgomp [11.2.0|7.5.0|8.4.0|9.3.0] conflicts with any installable versions previously reported.

I tried resolving this by obtaining some packages from other channels but it did not work out.

So I installed it from source (v2.0-5288-g867deb724) and I also tried another install from conda (openroad -version spits out b848aaed251d09d5c855c1dbce9f189b52862707) but both result in the same error.

Setting OpenROAD to use one metal pin layer helped to get past this command (place_pins -random -hor_layers {met3} -ver_layers {met2 met4}) but then it fails aftwerwards.

[par] [Exec ...d/bin/openroad -no_init -log /...] [NesterovSolve] Iter: 320 overflow: 0.868718 HPWL: 4779802302
[par] [Exec ...d/bin/openroad -no_init -log /...] [NesterovSolve] Iter: 330 overflow: 0.842689 HPWL: 5245321551
[par] [Exec ...d/bin/openroad -no_init -log /...] [NesterovSolve] Iter: 340 overflow: 0.820655 HPWL: 5709032460
[par] [Exec ...d/bin/openroad -no_init -log /...] [NesterovSolve] Iter: 350 overflow: 0.795199 HPWL: 6085352724
[par] ERROR: OpenROAD returned with a nonzero exit code: -11.
[par] To create a tar archive of the issue, set:
            par.openroad.create_archive_mode: latest_run
        in your YAML configs and re-run your par command

Thank you!

[Nayiri Krzysztofowicz]

Hi Philipp,

I'm guessing this is still an issue from using an older version of openroad. Could you try running these, then install the openroad from the tutorial?

conda config --set channel_priority true
conda config --add channels defaults

If these work I'll update the tutorial docs. If this doesn't work, could you send your openroad log (the latest build/<long-name>/par-rundir/openroad-*.log).

Thanks,

Nayiri

To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/235fb634-796f-4c0c-a9a6-2c44797b4fa5n%40googlegroups.com.

[Philipp Schmitz]

Hello Nayiri,

thanks a lot! With these commands I was able to install OpenROAD and KLayout via conda and the par runs past the points where it previously crashed.

Unfortunately it still crashes later in the process due to memory utilization (>32GB)  so I can not report if all issues have been resolved but I am optimistic and will report when I tried the flow on a system with more memory available.

[par] [Exec ...d/bin/openroad -no_init -log /...] [INFO DRT-0195] Start 0th optimization iteration.
[par] [Exec ...d/bin/openroad -no_init -log /...]     Completing 10% with 0 violations.
[par] [Exec ...d/bin/openroad -no_init -log /...]     elapsed time = 00:01:08, memory = 18775.61 (MB).

[par] ERROR: OpenROAD returned with a nonzero exit code: -9.

Best,

Philipp

[Philipp Schmitz]

Hello together,

I set everything up on another machine and ran the flow.

However, the process has been stuck on

[par] [Exec ...d/bin/openroad -no_init -log /...] (hammer) report_check_types -max_slew -max_capacitance -max_fanout -violators -digits 3

for roughly two weeks. If I interpret the log correctly the PAR finished but is unable to create the report files. The process is still running at 100% CPU and ~40 GB of RAM.

Do I assume correctly that this is abnormal behavior and I can abort the run?

Best,

Philipp

[Nayiri Krzysztofowicz]

Hi Philipp,

Yes that is a bug with OpenROAD and you should abort the run, a typical run usually takes ~6hrs total. We haven't gotten around to reporting this issue to the OpenROAD team.

At this point there should be a database saved to vlsi/build-sky130-openroad/chipyard.harness.TestHarness.TinyRocketConfig-ChipTop/par-rundir/pre_extraction that Hammer can load and continue from.

Just run this command:

make redo-par HAMMER_EXTRA_ARGS="--start_before_step extraction"

In the future you can run these two commands to avoid openroad freezing. We noticed there were several commands that would freeze indefinitely after detailed_route was run, including the report commands and executing klayout to write out the final GDS. Let me know if it still freezes with these commands:

make par HAMMER_EXTRA_ARGS="--stop_after_step extraction"

make redo-par HAMMER_EXTRA_ARGS="--start_before_step extraction"

Thanks,

Nayiri

To view this discussion on the web visit https://groups.google.com/d/msgid/chipyard/4ae10a18-2bb2-48f3-a3e0-c303107ca166n%40googlegroups.com.