analysis/visualization of internal representations?

[Glen Ropella]

I apologize in advance if this question should go to the users' list.  But it seems more likely that the developers would use such tools.

Is there a tool used to parse/analyze/visualize the internal representations of models?  In particular, I'd like to do some static and dynamic analyses of models just a bit beyond the profiler.  Things like call graphs, complexity, etc. would be useful and interesting.  Are there already tools for such?

I'm also assuming that what comes out of org.nlogo.headless.Dump.main() -- or org.nlogo.headless.misc.Dump -- is the closest I can get to the internal representation without hacking.  Is that right?  (It's not yet clear to me what the difference is between that Dump and org.nlogo.api.Dump.)  I don't have much of a preference for working at the logo code level vs. the internal representation.  But it just seems appropriate to consider both layers.

Thanks.

[Seth Tisue]

>>>>> "Glen" == Glen Ropella <gepr...@gmail.com> writes:

Glen> Is there a tool used to parse/analyze/visualize the internal
Glen> representations of models? In particular, I'd like to do some
Glen> static and dynamic analyses of models just a bit beyond the
Glen> profiler. Things like call graphs, complexity, etc. would be
Glen> useful and interesting. Are there already tools for such?

That would be very cool. I'm pretty sure nothing like that exists.
(Unless it's lurking on someone's hard drive and they just haven't been
telling us about it.)

Glen> I'm also assuming that what comes out of
Glen> org.nlogo.headless.Dump.main() -- or org.nlogo.headless.misc.Dump
Glen> -- is the closest I can get to the internal representation
Glen> without hacking. Is that right?

Yes, and note you can call headless.Dump in the command center using
`__dump` and `__dump1` (as mentioned at
https://github.com/NetLogo/NetLogo/wiki/Unofficial-features).

The code dumper includes the output of the bytecode generator -- you
probably don't want that level of detail. If you start NetLogo with
-Dorg.nlogo.noGenerator=true (`nogen` for short in sbt) you'll get
something more manageable, so e.g.:

% sbt nogen run

observer> ask turtles with [color = red ] [ fd 1 ] print __dump1
Command Center:[]{O---}:
[0]_ask:+3
_with
_turtles
_equal
_turtleorlinkvariable:COLOR
_constdouble:15.0
[1]_fd1
[2]_done
[...]

Here what we're seeing is what the NetLogo compiler back end
outputs, namely nvm.Procedure objects, where each Procedure
contains an array of nvm.Command objects, and each nvm.Command
is the root of a tree of nvm.Reporter objects.

If you want to do static analysis of NetLogo code, even that
representation might be too low-level for you; you might want to
work directly with the output of the front end of the compiler,
not the back end. The AST (abstract syntax tree) classes
output by the front end are defined in
https://github.com/NetLogo/NetLogo/blob/5.0.x/src/main/org/nlogo/compiler/AstNode.scala
For examples of how different kinds of code end up being
represented using these classes, scroll to the bottom of
https://github.com/NetLogo/NetLogo/blob/5.0.x/src/test/org/nlogo/compiler/ExpressionParserTests.scala
and look for `testStartAndEndPositions10` which uses the
same code as the example above. The major difference is
that the AST classes preserve the tree structure of the
original code, whereas after the back end has run, the
structure has been flattened so that commands are no longer
nested inside each other. (Reporters are left unflattened.)

Glen> (It's not yet clear to me what
Glen> the difference is between that Dump and org.nlogo.api.Dump.)

api.Dump produces string representations of Logo values. It's what
powers primitives like `word`, `print`, and `file-print`:

% sbt console

[...]
scala> import org.nlogo._
import org.nlogo._

scala> val ws = headless.HeadlessWorkspace.newInstance
ws: org.nlogo.headless.HeadlessWorkspace = [...]

scala> ws.openString(util.Utils.url2String(api.ModelReader.emptyModelPath))

scala> ws.command("crt 100")

scala> org.nlogo.api.Dump.logoObject(ws.report("one-of turtles"))
res2: String = (turtle 37)

--
Seth Tisue | http://tisue.net

[glen e. p. ropella]

On 03/23/2014 11:04 AM, Seth Tisue wrote:
> If you want to do static analysis of NetLogo code, even that
> representation might be too low-level for you; you might want to work
> directly with the output of the front end of the compiler, not the
> back end. The AST (abstract syntax tree) classes output by the front
> end are defined in
>
> https://github.com/NetLogo/NetLogo/blob/5.0.x/src/main/org/nlogo/compiler/AstNode.scala
>
> For examples of how different kinds of code end up being represented
> using these classes, scroll to the bottom of
>
> https://github.com/NetLogo/NetLogo/blob/5.0.x/src/test/org/nlogo/compiler/ExpressionParserTests.scala
>
> and look for `testStartAndEndPositions10` which uses the same code
> as the example above. The major difference is that the AST classes
> preserve the tree structure of the original code, whereas after the
> back end has run, the structure has been flattened so that commands
> are no longer nested inside each other. (Reporters are left
> unflattened.)

Just to follow up on this in case anyone cares. Here's what I've done.
I'm sure it would be cleaner to write my own class like
ExpressionParserTests. Or perhaps there's already a built-in way to get
this output. But I just wanted to, quick and dirty, see what a whole
model would look like. With this change in the headless branch:

diff --git a/src/main/compile/Compiler.scala
b/src/main/compile/Compiler.scala
index 3d056bd..ef56e45 100644
--- a/src/main/compile/Compiler.scala
+++ b/src/main/compile/Compiler.scala
@@ -36,6 +36,10 @@ object Compiler extends nvm.CompilerInterface {
flags: nvm.CompilerFlags): nvm.CompilerResults = {
val (defs, structureResults) =
frontEnd.frontEndHelper(source, displayName, program,
subprogram, oldProcedures, extensionManager)
+ val (booga) = frontEnd.frontEnd(source, oldProcedures, program) match {
+ case (procs, _) => procs.map(_.statements)
+ }
+ print(booga)
middleEnd.middleEnd(defs, flags)
backEnd.backEnd(defs, structureResults.program, source,
extensionManager.profilingEnabled, flags)
}


Recompiled. Then:

./sbt |tee model.ast
run --model model.nlogo --experiment experiment.xml

Which produced stuff like this (except without the formatting, of course):

Vector(
_nodisplay[]
_clearall[]
_ask:+0[_patches[],
[_set
[_patchvariable:2[], _constdouble:9.9[]]
]
]
_set[_observervariable:17[], _mult[_observervariable:16[],
_constdouble:10.0[]]]
_set[_observervariable:18[], _constdouble:5.0[]]
_set[_observervariable:19[], _constdouble:1.0[]]
_set[_observervariable:20[], _mult[_constdouble:10.0[],
_observervariable:18[]]]

_set[_observervariable:21[],
_div[_mult
[_floor [_observervariable:2[]], _observervariable:15[]],
_constdouble:100.0[]
]
]

_set[
_observervariable:22[],
_div[_mult
[
_ceil
[_observervariable:2[]],
_minus[
_constdouble:100.0[],
_observervariable:15[]
]
],
_constdouble:100.0[]
]
]

_call:INITIALISE-BUGS[]

_ask:+0[_breed:BUGS[],
[_call:SETUP-BUGS[]]
]

_call:SHOW-PLOTS[],

_createturtles:BUGS,
+0[_observervariable:4[],
[_hideturtle[]
_set[_breedvariable:ENERGY[], _observervariable:16[]]
]
]

... etc. ...


Thanks again, Seth!
--
glen ep ropella -- 971-255-2847