Sniper on RHEL 9 SDE using -s energystats

[Jake Merritt]

I recently went through a lengthy debugging process for an error that I saw posted here but not answered, and I was hoping that my work might save someone a fair bit of time. I built snipersim on a RHEL 9 virtual machine and I used the SDE frontend. When I tried to run this command:

./run-sniper \

-c /home/jmerri07/HotGauge/examples/skylake_14nm.cfg -n 8 \

-d output/test-fft \

-s scripts/energystats:40000 \

-s stop-by-icount:200000000 \

-- test/fft/fft 

I got this error:

[SNIPER] Start

[STOPBYICOUNT] Starting in ROI (detail)

[STOPBYICOUNT] Then stopping after simulating at least 200000000 instructions in detail

[SNIPER] --------------------------------------------------------------------------------

[SNIPER] Sniper using SIFT/trace-driven frontend

[SNIPER] Running full application in DETAILED mode

[SNIPER] --------------------------------------------------------------------------------

[SNIPER] End

[SNIPER] Elapsed time: 0.59 seconds

[RECORD-TRACE] Using the SDE frontend (sift/recorder)

[SIFT_RECORDER:0:0] Error:

Unable to open the output file /tmp/tmp3lh07_5k/run_benchmarks.app0.th0.sift

without -s energystats, sniper ran. 

this was my debugging trace:

1. Run-sniper does Popen starting lib/Sniper

2. This brings us to common/system/simulator.cc which sets up the sim

3. During setup (the start() function) we hit: m_stats_manager->recordStats("start");

4. recordStats func is in common/misc/stats.cc This calls: Sim()->getHooksManager()->callHooks(HookType::HOOK_PRE_STAT_WRITE, (UInt64)prefix.c_str());

5. callHooks (inside common/system/hooks_manager.cc) calls the function callPythonFunction with this line: SInt64 result = it->func(it->arg, arg);

6. This brings us to callPythonFunction in common/scripting/hooks_py.cc where I put a GIL check that returned 1

7. callPythonFunction calls PyObject_CallObject(pFunc, pArgs)

8. That calls the python function hook_pre_stat_write(self, prefix) which is in scripts/energystats.py

9. hook_pre_stat_write(self, prefix) sends us into update(), also inside energystats

10. Update() returns because sim.stats.time() == self.time_last_power = 0

11. We return back to callPythonFunction and a GIL check returns 1

12. We return back to callHooks and after this line SInt64 result = it->func(it->arg, arg); I inserted a GIL check that returns 0

13. At the end of the recordStats there are three nested for loops and inside those loops this is called: sqlite3_bind_int64(m_stmt_insert_value, 4, it3->second->recordMetric());

14. recordMetric (defined in common/misc/stats.h) calls statsCallback in common/scripting/py_stats.cc

15. We fail silently in statsCallback at the line PyObject *pResult = HooksPy::callPythonFunction(pFunc, Py_BuildValue("(sls)", objectName.c_str(), index, metricName.c_str()));

the issue was losing the GIL in step 12. I'm not sure why it gets lost, but the solution is these two edits:

1. In the functions callPythonFunction (in snipersim/common/scripting/hooks_py.cc) and statsCallback (in snipersim/common/scripting/py_stats.cc)

Add PyGILState_STATE gstate = PyGILState_Ensure(); at the very beginning

And PyGILState_Release(gstate); right before the return

2. I changed scripts/sim/__init__.py to (additions in bold)

import os, sqlite3, threading

stats.db = sqlite3.connect(

    os.path.join(config.output_dir, 'sim.stats.sqlite3'),

    check_same_thread=False,

    timeout=30.0,

)

stats.db_lock = threading.Lock()

3. I also edited this chunk of the db_delete function:
   
   This is in: scripts/sim/util.py

def db_delete(prefix, in_sim_end = False):

  global have_deleted_stats

  with sim.stats.db_lock:

    cursor = sim.stats.db.cursor()

    try:

      prefixid = sim.stats.db.execute('SELECT prefixid FROM prefixes WHERE prefixname = ?', (prefix,)).fetchall()

      if prefixid:

        cursor.execute('DELETE FROM prefixes WHERE prefixid = ?', (prefixid[0][0],))

        cursor.execute('DELETE FROM `values` WHERE prefixid = ?', (prefixid[0][0],))

      sim.stats.db.commit()

    

    except sqlite3.Error:

        sim.stats.db.rollback()

        Raise

I hope this helps! Good luck to everyone.